Improving Gloss-free Sign Language Translation by Reducing Representation Density
Authors: Jinhui Ye
Xing Wang
Wenxiang Jiao
Junwei Liang
Hui Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: Gloss-free Sign Language Translation; Representation Density; Performance Drop; Contrastive Learning
Primary area: natural_language_processing

Abstract: Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify **a representation density problem** that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCLachieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39\% and 46\%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCLachieves better performance with only 35\% of its parameters. We will release our code and model to facilitate further research.

Improving Gloss-free Sign Language Translation by Reducing Representation Density
Jinhui Ye!
Xing Wang'
Wenxiang Jiao Junwei Liang 22 1.2.
Hui Xiong 1.23.f
Artificial Intelligence Thrust, HKUST (Guangzhou), Guangzhou, China Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China
Guangzhou HKUST Fok Ying Tung Research Institute *Tencent AI L.a
ye624@connecct hkust-gz. edu en; joalwwjaaooteeeeettttcm @tencent.com xionghui uust.hk; (brightxwang, jellxjiaol
edw..ddu... cn;
Abstract
Gloss-free sign language translation (SLT) aims develop well well-performing SLT systems with no requirement for the costly gloss amnotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a represen- tation density problem that could be bottleneck in restricting the performance of gloss-free SL.T. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free free models to learn more discriminative feature representation in a suff-supe manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCI. achieves significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model param- eters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at https://githinn Com/Jinnuii BachuuiEE/Siie YE/SignCL
Introduction
Sign languages are the primary form of communication for millions of deaf individuals. Sign language translation (SLT) aims convert sign language into fluent spoken language sentences, which is challenging task as it needs to extract information from continuous video and translate it into discrete text tokens. Most prior studies promoted the SLT by utilizing intermediate representations,
49 46 41). Gloss annotations are beneficial as they provide a simplified representation and sequential ordering of each gesture within continuous sign videos, which aids in representation learning for visual encoders. However, the creation of sign language translation datasets with gloss annotations is both resource-intensive and consuinn..
namely gloss annotations, either directly or indirectly 3 48 53
Recently, there has been shift towards gloss-free sign language translation methods, which do not rely on gloss annotations to train SLT models. These methods usually rely on general datasets [7
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MICHSSOT
CUTT
LUPD
Wib
LEFT
RECIPROCATE
REVENGE
Figure 1:
An example of the representation density problem in sign language translation. The two images show the sign gestures for "RECIPROCATE" (blue dot) and "REVENGE" (orange dot). Although the two have opposite meanings, their visual representations are densely clustered together, as shown the t-SNE visualization. The various colors in the visualization indicate sign gestures with different meanings.
general pretraning strategy [52], or general large-scale foundation models 442 to promote gloss- free SLT. However, there is substantial gap between the sign language domain and the general Models trained with general strategies or datasets often fail to capture the subtle differences in semantically distinct gestures, which are crucial for accurately understanding specific sign language. Therefore, the performance gloss-free methods still significantly lags behind that of gloss-based approaches.
domain 45 24
In this paper, identify representation density problem in sign language translation: the visual representations of sign gestures with distinct semantics are likely to be close in representation space. This problem is attributed to the nature of sign language, form of visual language that utilizes intricate hand gestures, facial expressions, and body movements to convey the signer's 's message B35 40 meanings, "RECIPROCATE" and "REVENGE" with similar visual information (i.e., only subtle differences in facial movements). The visual encoder in SLT models will encode similar visual information to visual representations in adjacent representation space, even though they have distinct semantics. Without explicit gloss annotations, SLT models struggle learn semantic boundaries in continuous sign videos and capture distinguishing visual representations for different sign gestures. As result, the representation density problem poses a significant challenge for the SL.T models in distinguishing between various sign gestures, leading to sharp performance drops. (Section (.2).
For example, in Figure ! the signer performs sign gestures for opposite
Further, we investigate various popular sign feature extraction methods, including gloss-based and gloss-free [47 [2] to systematically study the representation density problem. As shown in Figure our investigation reveals that the representation density problem is prevalent across sign feature extraction methods. Specifically, due to the lack of gloss annotations, the representation density problem appears to more serious in gloss-free methods. Then, we conduct extensive SLT experiments and observe that SLT models using gloss-free free sign features as input consistently suffer a drop in performance both sign language recognition and translation tasks compared to those using gloss-based sign features (Section 3.3) Therefore, we demonstrate that the representation density problem can be a bottleneck in restricting the improvement of gloss-free free sign language translation.
More importantly, we propose a simple but effective contrastive learning strategy named SignCL to address the representation density problem. Specifically, SignCL draws the visual representations of sign gestures with identical semantics closer together and pushes those with different semantics farther apart. Experimental results show that SignCL can learn more distinctive feature representations and lead to significant improvements in terms of BLEU score various well-known SLT frameworks (Section 55.. To summarize, the main contributions of this work are as follows:
To the best of our knowledge, our work identifies the representation density problem in sign language translation for the first time. This problem consistent across various sign feature extraction methods for SLT. including gloss-based and gloss-free methods.
Experimental results empirically reveal that an increase in representation density leads to significant performance drop in the accuracy of sign language recognition and translation. We find that the representation density problem poses significant challenge for the gloss-free SLT.

---

We propose simple but effective contrastive learning strategy, namely SignCL, to address the representation density problem. Our experiments demonstrate that SignCL can significantly enhance various well-known SLT frameworks. Specifically, SignCI. yields a 39% BLEU score improvement for the Sign Language Transformer and a 46% BLEU increase for GFSLT- VIP 53 on the CSL-Daily dataset.
2 Related Works
2.1 Sign Language Translation
Sign Language Translation (SLT) methods can be broadly categorized into gloss-based and gloss-free approaches. For gloss-based methods, essential factor is to directly indirectly employ sign gloss annotations to improve sign video encoder performance [3, 48
53
44 6]. These methods often 8.
employ Connectionist Temporal Classification [16] (CTC) loss perform sign language recognition 4. Joint-SLT [4) firstly introduces a multitask encoder-decoder framework with a CTCloss to soft- match sign representations and gloss sequences. STMC-T [54] introducing intra-cue and inter-cue CTC loss to model multi-cue sequence information. Despite their effectiveness, creating SLT datasets with gloss annotations is ressurce-intensiin as a promising alternative, as they do not rely on gloss annotations during training, making them more generalizable. And recently, growing body of literature has promoted the gloss-free SLT, such as GASLT 47 proposed local gloss attention to mimic gloss assistant, GFSLT [22 adapted CLIP do visual-language pretraining. and Sign2GPT [42] promoted performance by making use of surge-scale pre-trained vision and language models. Nonetheless, the performance of gloss-free methods still significantly lags behind that of gloss-based approaches.
and time-consuming. Gloss-frre methods have emerged
2.2 Contrastive Learning
Contrastive Learning [21] [55 30), popular unsupervised learning algorithm, aims to learn effective representations by pulling positive pairs closer together and pushing negative pairs farther apart. This approach has been widely utilized in both Natural Language Processing and Computer Vision (13. In Sign Language Translation (SLT), Jin and Zhao (22 utilize Contrastive Learning to create a Signer-Independent SLT model, using videos demonstrating signs from different signers as positive samples. Additionally, Gan et al. [15] proposes a visual-level contrastive learning method with various image augmentation strategies. ConSLT [14] do contrastive learning for effective token representation learning in text decoder. Zhou et al. 52 and Cheng et al. [9 employ contrastive learning techniques align video and text representations SLT. In this paper, we are the first one to address the representation density problem, focusing particularly on visual gesture duration as central aspect.
2.3 Representation Density
Representation Density is often a focal point in classification tasks, also known category den- sity 11 (4. representations across different categories. In the context of sign language, various methods have been developed to address the subtle nuances of sign actions. TSPNet (28) proposes a temporal hierarchical attention network to learn segmented representations. HST-GNN 24 utilizes hierarchi-
37
31
32
51
29 12). This concept pertains to the compactness and clarity of feature
cal spatio-temporal graph neural network to learn graph representations from multiple perspectives. GLE-Net [20] employs global contextual relationships and fine-graine cues to distinguish non- manual-aware features in isolated Sign Language Recognition. These methods are beneficial for addressing the subtleties of sign language movements. However, integrating them into existing state-of-the-an ahe-art frameworks presents significant challenges. often resulting in performance disparities when compared to the SOTA. This paper the first to propose the concept of representation density within this field and introduces SignCL, which enhances the current mainstream transformer- -ased frameworks.
3 Representation Density Problem
This section investigates and identifies the representation density problem within existing sign feature extraction techniques, and examines whether representation density bottlenecks sign language recognition and translation performance.
3

---

3.1 Preliminaries
Existing Sign Feature Extraction Techniques Existing sign feature extraction methods can be divided into two categories: 1) gloss- based (e.g., Sign Recognition Pretrained [3] and Self-Mutual Knowledge Distillation (33] and 2) gloss-free (e.g.. I3D Pretraining [47] and Visual-Laan Lasgua-- Pretraining ) These methods were chosen for their representativeness in SLT and their well- documented open-source sign features.
Sign Recognition Pretrained (SRP) 3: This approach leverages the sign language recognition datasets to train sign language recognition models and uses it as the feature extractor for the SLT task. Notably, the features released by Camgoz et al. [3] have been widely adopted as input
features in range of works 53 22
46
Self-Mutual Knowledge Distillation (SMKD) 18] This approach enhances SRP by enforcing
the visual and contextual modules focus short-term and long-term information [8]
SMKD feature extraction has been shown to substantially enhance SLT translation performance
compared SRP [50 46
I3D Pretraining (L3D) 47]: This method employs I3D models as the backbone to pre-train the
feature extractor, initially trained on the Kinetics dataset [25] and subsequently fine-tuned on extensive web SLR datasets, such as WSLR (27].
Visual-Language Pretraining (VLP) [52] This method entirely forgoes gloss annotations and leverages general visual-language pretraining strategy to align sign video representation with text. Embodied by GFSL.TVVLP [52), this approach offers more general solution that utilizes broader range of sign language resources without the constraints of gloss annotations.
Drawing inspiration from Fisher's Discriminant Ratio (FDR) 23 19). a typical measure used to evaluate the discriminative power of features in the classification, we combine the average Inter-Gloss Distance and Intra-Gloss Distance into Sign Density Ratio (SDR, see Eqn. D. which reflects the degree of representation density for each gloss G, This is given by the formula:
Representation Density Metrics
Dintra
D(G.)
SDR(G.)
avg. avg.DD.
Dinter Meanj (D(G.,
Here, D(G,, Gj) represents the Inter-Gloss Distance between two glosses G, and Gj. and avg. avg.Dir. Dinter reflects the average distance of G; to other glosses. The Intra-Gloss Distance Dintre evaluates the average distance within a single gloss G, These distances are given by the following formulas:
D(G,,
Σ DEGI,NEG,
d(z,y):
(2)
GG
D(G.) G (
Σ
d(z,y):
Z.
Where, |G:| and |Gy| denote the number of instances in glosses G; and Gj, respectively, and d(x; y) represents the distance measure between the embeddings of instances and y, i... euclidean distance.
The average Sign Density Ratio (SDR) of all glosses, denoted as SDR = Mean SDR(G:)), is calculated to evaluate the overall representation density of the dataset comprchensively.
Sign-Gloss Alignment To calculate the Sign Density Ratio (SDR), we need determine the mapping relationship between input frames and gloss categories. Following previous works 26 46), we employ the CTC classifier as a fign-gloss forced aligner to establish the mapping between each gloss and its corresponding sign frames. The aligner provides the start position l, and end position re within the video frame sequence for each corresponding gloss g.- To optimize alignment performance on the test set, we merge the training and test datasets for comprehensive training and engage two volunteers to select the best frame fa from the range [I,r,,) to align with each gloss fo. Extensive details on the training procedure and the aligner's 'ss performance metrics documented in Appendix 9

---

3.2 Demonstrating Representation Density Problem
Experiment Setups We primarily use the SHOENIX-20144 benchmark B to investigate the representation density problem in existing sign feature extraction techniques. This benchmark was selected due to its rich collection of open-source sign features contributed by various research efforts. We obtained the sign features by either downloading the officially released versions reproducing the feature extraction process. Then, we employed t-SNE [39] to visualize the feature distribution of these semantically distinct sign gestures to investigate representation density.
(a,b,c) Gloss-free Feature Extraction
(d,e) Gloss-based Feature Extraction
SDR=92.59%
33R=83.33%
SDR=81.309
SDR=74.07%
SDR=66.233
(a) VL.
(b) I3D
(c) SSignC.(OOr))
(d) SRP
(c) SMKD)
Figure 2: The t-SNE visualization of sign features across existing extraction techniques. SRP, SMKD, and I3D are downloaded from their official websites, while VLP reproduced with official code. The addition of +SignCL denotes our proposed method that integrates a contrastive learning strategy into the VLP method (see Section 4 Different colors represent sign gestures with distinct semantics. Points in gray represent other sign categories not listed. Better viewed by zooming in.
Results and Findings Through empirical analysis of various visualized open-source sign features, we have identified widespread representation density problem across different sign feature extraction methods. As depicted in Figure E all evaluated methods display Sign Density Ratio exceeding 50%, with inevitable overlap of feature representation. Notably, gloss-free fres--fre methods that do not utilize gloss annotations as additional supervision (e.g., I3D and VLP) exhibit even more severe representation density compared to gloss-based methods. This is evident sign gestures representing different semantics, indicated by different colors, significantly overlap, resulting in translation ambiguity during inference. Specifically, the Sign Density Ratio (SDR) of VIP is 92.59% which is significantly higher than the SDR of SMKD at 66.23%.
(a) Impact on SL.
(b) Impact on SLT
Jucdd Genooret 540
Garhhe Spri.
Counfee VLP
24
Soar fooam
Sint-SIT
Jont-SLi
Closeee
sur nit nad Nan
Pe Performan arcce Werife Bipd
NSk SigaeL
NS.T
Oct corwinttnnen
Aecrge SDR
Sun Accurasy
NSLT
JoRt
Jain-au UST-Sigii
X5.7
Neir
NSIT
Karaaatasee Deary (-201g)
Sign Rejensentaiion ion Damsily Rannking (Axcanding)
Sign Resresentation Density Ratio SDR/S) i
Figure 3: Comparative analysis of representation density and its impact on sign language recognition (SLR) and translation (SLT). The left panel (a) shows the correlation between representation density and SLR accuracy across different sign feature types and sign gesture groups. Binning in this context is based on sorting by gloss density within group, where higher bins indicate higher density, The right panel (b) illustrates the performance drops in SLT caused by the representation density problem. This figure assesses both the recognition and translation accuracies, reflecting how denser representations impact these metrics.
5

---

3.3 Demonstrating Performance Drop
This section investigates the impact of representation density on sign language recognition (SLR) and translation (SLT) systems.
General Setups This part employs the widely utilized Sign Language Transformer (NSLT) as the foundational model for our evaluations. The choice is because its capability to perform both SLT and SLR tasks, as well as take SLR and SLT at the same time (joint-SLT). Additionally, the NSLT framework is well-established within sign language research and benefits from comprehensive documentation and support in open-source sign feature sets and baseline results. The NSLT relies on sign features derived using pretrained sign feature extractor. This section studies all types of sign features introduced in Section 3.2 to investigate the representation density problem. We use the Sign Density Ratio (SDR, see Eqn. I to measure the representation density within each type of input feature. We measure SLR and SLT performance by the recognition accuracy and the BLEU-4 score (B @ 4), respectively.
Task Setups We set up tasks to examine whether representation density bottlenecks sign language recognition and translation performance.
To evaluate the ability of the extracted sign features to distinguish between different semantic gestures, we the NSLT 4 perform sign language recognition (SLR) tasks with various types of sign features 3 as model input. Due to the limited number of samples for each gesture in the dev set, we rank the sign glosses based on their density using SDR(G,) under SMKD features (see Eqn. [T. These glosses are then divided into nine groups (bins), each containing approximately 60 glosses. The average SDR(G; (G,) and recognition accuracy for each bin represents the overall density and mean accuracy of the glosses within that bin, respectively.
Sign Language Recognition:
Sign Language Translation: This evaluation aims to demonstrate the impact of representation density on translation tasks. We evaluate various sign features as inputs to the Sign Language Transformer, including SRP, SMKD, I3D, VLP, and VLP+SignCL. These inputs are tested across different translation frameworks, such as NSLT [3, Joint-SLT [4, and NSLT+SignCL. NSLT means use NSLT to perform SLT without CTC loss (gloss-free fross-free) and the NSLT+SignCL configuration integrates the proposed contrastive learning strategy into the encoder of NSLT B models, as detailed Section
Results and Findings
As depicted in Figure 3, the following observations were made regarding the impact representation density on both recognition (SLR) and translation (SLT):
Performance suffers from representation density. We consistently observed negative relationship between representation density and performance across all feature types and tasks. Higher representation density leads to worse accuracy in SLR and lower BLEU scores in SLT. Specifically, an increase in the representation density ratio by 26% can result in a 39% performance drop in NSLT.
Gloss-free methods suffer from worse representation density. Gloss-free based feature extractions, which do not use any gloss annotations for assistance (c.g.. VLP), typically exhibit
higher representation density scores than gloss-based approach (c.g., SDR(VLP)=92.599 SDR(SMKD)=66.22 23%... Using gloss-free features results in worse recognition and translation performance compared to gloss-based feature extractions (e.g., VLP vs. SMKD).
Contrastive learning boosts performance by reducing representation density. When con- trastive learning is applied to augment gloss-free based feature representation learning, i.c., VLP+SignCL for feature extraction or NSLT+SignCL for downstream finetuning, there is consistent reduction in feature representation density accompanied by a significant improvement in both of the SLR accuracy and the SLT performance (see detail can be found in Section 4
4 Contrastive Learning for Gloss-free Sign Langauge Translation
Contrastive Learning 21
a popular self-supervised learning algorithm, aims to learn effective representations by pulling positive pairs closer together and pushing negative pairs farther apart. In

---

this section, we introduce simple but efficient sign contrastive learning strategy, namelySignCL. which addresses the challenge of the representation density problem in gloss-free sign language translation.
4.1 Sign Contrastive Learning
The key factor in contrastive learning is how to sample positive and negative training pairs. As illustrated in the framework shown in Figure the sampling strategy of SignCL is as follows: two frames close enough (e.g., adjacent), they are considered to belong the same sign gesture and are treated as positive samples. Conversely, if two frames are far apart by double the margin (e.g-, |fed fael 20 frames), they are considered be associated with different semanties and are treated as negative samples. Statistically, the average duration of each gesture in sign video is nine frames [ [ and according to the speech-to-gesture Zipf's Law [2], each gloss represents approximately 2.3 spoken words. Therefore, set the margin max(10,
lentrrames)
2.3).
Jen(text)
positive pair (fsts fad): lfed - fatl S negative pair (fat, fed):
(4)
Ifed fael 2 margin
N
LsignCL = N
Σ [d(fst: + max(0, m d(fst, fea))];
(5)
Where d is the distance function, i.e., Euclidean distance for frame features (fst= Fea). and N the total number of frames in one sign video, N = len(frames). The margin parameter is used to prevent the features of the negative pair from being too far away. We empirically set m = 64 based on the average Inter-Gloss Distance (see Eqn. [ of gloss-based sign features (e.g.. SMKD
Contrast Loss
Contrast Loss
Sign Sentence
Positive Loss
Negative Loss
VLP
SLT
Visual Encoder
Text Encoder
Visual Encoder
Text Decoder
"Sign Language"
"He"
"Know"
Sign Video
Sign Sentences
Sign Video
(a) Sign Contrastive Learning
(b) SignCL in Pretraining
(c) SignCL in Finetuning
Figure Overview of the SignCL in gloss-free sign language translation: (a) Sign contrastive learning sampling strategy, (b) Showcases the integration of SignCI. in the pretraining stage, and (c) Displays the application of SignCI. during the finetuning stage.
4.2 Integrating Contrastive Learning into Sign Language Translation Tranining
As illustrated in Figures 4b and 4c SignCL be integrated into both the sign feature extraction
pretraining stage (e.g., Visual-Languaal Pretraining (52)) and the downstream task finetuning stage
(e.g., GFSLT-VLP (52) The optimization objective for these approaches the weighted sum of
LSigeC1. and the original objective loss (e.g., VLP Loss for pretraining and SLT loss for finetuning 3
52
defined as:
L A Lsigutt. + LMLES
(6)
Where EMLE is the original objective loss in the pertaining or finetuning.
5 Experiments
In this Section, we conduct experiments to demonstrate the efficiency of proposed SignCL in reducing representation density and boosting floss-free sign language translation performance. Specifically, we apply SignCL to the Sign Language Transformer 4 to facilitate a direct comparison with prior empirical analyses of the representation density problem in Section 3.3 Additionally, integrate SignCL into the GFSLT-VLP (52) framework, a robust new gloss-free baseline that improves SLT through pretraining and finetuning.
7

---

5.1 Experiments on Sign Language Transformer
In Section 3 we investigate the representation density problem using the Sign Language Transformer (SLT) A and the PHOENIX- [3] and CSL-Daliy [53] Dataset. These benchmarks are chosen for their established relevance sign language translation research, including gloss- and gloss- free based. Here, we first conduct experiments on the same framework and dataset to facilitate direct comparison with the prior empirical analyses.
Experiment Settings: In this experiment, we introduce SignCI. as additional supervision information in the encoder of SLT under gloss-free settings. This enhanced model is referred to +SignCL.
Results and Findings: The integration of SignCL into the SLT has significantly improved translation performance across all test conditions by reducing the representation density, shown in Table 1 Notably, SignCL encourages SLT to learn a more distinct feature distribution, reducing the Sign Density Ratio (SDR) significantly. e.g., 66.23 to 62.18 and 92.59 to 81.30.
Figures 3a and 3b show experiments on SLR and SLT tasks using features with varying SDRs as inputs to SL.. The representation density reduction leads to observable improvements in both recognition accuracy (red line vs. purple line in Figure 3a) and translation BLEU score (purple point vs. red point in Figure 3b). Further details and additional experiment results on the CSL-Daily dataset are provided Appendix A.4.3
Table T presents a comparative analysis of representation density and performance on the PHOENIX- 2014T dataset. The inclusion of SignCL during VLP feature extraction or SLT training processes significantly enhances performance metrics. WERs (Word Error Rates) in the gloss-free set, derived from an independent SLR task, are specifically used to probe the quality of sign features and do not participate in the SLT training process. This analysis underscores the significant enhancements brought by SignCI. in terms of both efficiency and effectiveness in SLT frameworks.
PHOENIX-2014T
CSL-DDall WER4
Feature Type
SDR4
WER
B@4f
SDR
B@41
Gloss based
Joint-SIT [5] / Self-Mutual KD
3R
66.23
25.38
22.79
48.34
29.52
11.61
+ SignCL into Feature Extraction
62.18
24.76
23.23
+ SignCL into Finetuning
66.23
25.12
22.92
+ SignCL into both
62.18
24.58
23.46
Gloss free
SLT 3 VLP Pretrained [52
92.59
69.72
10.73
76.55
85.78
1.82
+ SignCL into Feature Extraction
81.30
63.33
12.04
68.39
80.71
2.15
+ SignCL into Finetuning
92.59
12.14
76.55
2.19
+ SignCL into both
81.30
13.51
68.39
2.53
Table 1: Comparative analysis of representation density and performance the PHOENIX-20147
dataset. "+SignC1." indicates the inclusion of the proposed contrastive learning strategy during VIP (Video Language Processing) feature extraction SLT (Sign Language Translation) training processes. WERs (Word Error Rates) in the gloss-free set are derived from an independent SLR (Sign Language Recognition) task, used specifically for probing the quality of sign features. These WERs do not participate in the SLT training process.
5.2 Experiments on Gloss-free Sign Language Translation
Gloss-free sign language translation, which does not rely gloss annotations, has become trend as it makes the approach more generalizable. In the realm of gloss-free sign language translation, GFSLT-VLP [22 stands out strong baseline. It incorporates CLIP 36) and MBART (10) for model pretraining and finetuning. In this set of experiments, we use GESLT-VLP as the baseline model and integrate the proposed SignCL into the framework to demonstrate the effectiveness of our method in both pretraining and finetuning settings.
Experiment Settings: This set of experiments is conducted using the PHOENIX-20141 3 and CSL-Daily (53 datasets. We reproduce GFSLT-VIPP using the official code and integrate SignCI. into both the pretraining and finetuning stages. All models and training details are consistent with

---

Density SDR
Performance
Model
R@Lt
B@ t
B@2 f B@3 1 B@41
NSLT 3
30.07
29.86
17.52
11.96
9.00
GASLT
39.86
39.07
26.74
21.86
15.74
GFSLT 122
40.93
41.39
31.00
24.20
19.66
GFSLT-VLP
42.49
43.71
33.18
26.11
21.44
Sign2GPT(w/PGP) 42
48.90
49.54
35.96
28.83
22.52
GFSLT-VLP [22
68.53
42.97
42.13
32.04
25.62
21.25
+ SignCL into Pretraining
62.68
49.25
49.99
36.73
29.76
22.69
SignCI, into Finetuning
62.73
48.17
48.56
35.04
27.73
22.16
+ SignCL into Two State
62.32
49.04
49.76
36.85
29.97
22.74
Improvement
-6.21
+6.07
+7.63
+4.81
+4.35
+1.49
Table 2:
Improvement in the GFSLT-VLP framework by reducing representation density PHOENIX-200 -014T- test set. "+SignCL into Pretraining" indicates applying the proposed contrastive learning strategy during the pretraining stage, while "+SignCL into Finetuning" indicates the inclu- sion of the SignCL during the finetuning stage. "+SignCL into Two State" means plus SignCL both in pertaining and finetuning states.
Density SDR
Performance
Model
R@Lt
B@1
B@21 1
B@3
B@4
GASLT
47
20.35
19.90
9.94
5.98
4.07
NSLT
34.54
34.16
19.57
7.56
7.56
GFSLT
35.16
37.69
23.28
14.93
9.88
GFSLT-VLP
152
36.44
39.37
24.93
16.26
11.00
42.36
41.75
28.73
20.60
15.40
GFSLT-VLP [52
58.20
39.08
36.37
23.32
15.45
11.10
SignCL into Pretraining
55.24
47.38
46.20
32.33
22.35
15.85
SignCL into Finetuning
55.03
48.26
46.53
32.41
22.42
15.98
SignCL, into Two States
54.61
48.92
47.47
32.53
22.62
16.16
Improvement
-3.59
+9.84
+11.10
+9.21
+7.17
+5.06
Table 3: Enhancing GFSLT-VLP by reducing representation density on CSL-Daily test set.
those used in GFSLT-VLP [2]. with the sole exception being the incorporation of SignCL, weighted by A 0.01, illustrated in Figure and Equation Further details are provided in Appendix A.I
compare our proposed methods with existing gloss-free sign language translation approaches. The results demonstrate that integrating the proposed SignCI. strategy into the GFSL.T-VI.P framework consistently reduces representation density and significantly boosts translation performance, whether SignCI. is applied during pretraining, finetuning, or both stages. Specifically, compared to the baseline model GFSLT-VLP [52]. our approach achieves substantial improvement of 45.58% (+5.06) in the BLEU-4 score on the CSL-Daily dataset, without any increase in the number parameters. Additionally, despite having significantly fewer parameters (~600M vs. ~ 1.7B), our approach achieves better performance than Sign2GPT [4], which leverages large-scale pretrained vision and language models for sign language translation.
Results and Findings: Tables
5.3 Qualitative Analysis
To understand our SignCL approach in scenarios of addressing representation density, we present case from the CSL-Daily dataset Figure As shown, the way display sign gestures for " " (laptop) and "4 (piano) differ subtly. As indicated by the t-SNE results, the representations of these two semantically different gestures are closely packed together in the feature space, causing the baseline GFSLT-VLP model to incorrectly translate "$9 3 (piano) as " """ "aa""(" (laptop). In contrast, our proposed SignCI. effectively separates the representations of d f (laptop) and "$ (piano) in the feature space, enabling the accurate translation of 44 $5 (piano).
A

---

GFSLT-VLP
SignCL
LAPTOP
PIANO
PIANO
LAPTOP
Reference
3 $
$4
(Mom bought me piano ))
GFSLT-VLP
*
i
(Mom bought me laptop
GFSLT-VLP+ + SignCL # पm
(Mom bought me piano
Figure 5: Qualitative comparison of translation results on CSL-Daily test set. The red background denotes model misinterpretations about the sign gestures, while green one means accurate recognition. Content (... is English translation for non-Chinese readers.
6 Conclusion
In this work, we identify a crucial representation density problem in gloss-free sign language trans- lation. Our systematic investigation reveals that this problem persists across various existing sign feature extraction methods and causes sharp performance drops in both sign language recognition and translation, particularly gloss-free methods. To address this problem, propose simple but effective contrastive learning strategy, termed SignCL. Our experiments demonstrate that SignCL encourages gloss-free free models to learn more discriminative features and significantly reduces repre- sentation density. Furthermore, our experiments show that SignCL improves translation performance across various frameworks and datasets by significant margin, achieving a new state-of-the-art in gloss-free sign language translation. We illustrate the effectiveness of SignCI. through detailed exam- ples our qualitative analysis. Finally, we provide several ablation studies for a better understanding of SignCL and discuss the limitations and potential societal impacts of this work in the Appendix A
7 Limitations
Our work, while promising, has several limitations that should be considered:
Boundary Cases:
We assume that adjacent frames output the same sign gestures, while distant frames belong to different sign gestures. This assumption might not hold in special sign language videos with extensive repetitive gestures. In extreme cases, SignCI. might affect feature convergence.
Semantic Similarity: SignCL. does not account for the semantic similarity between sign gestures, which can result in increased feature distances between semantically similar gestures. This could potentially affect the learning linguistic features.
Despite acknowledging these limitations, our experiments demonstrate that our approach works effectively in most cases. We will address these issues in future work further enhance the robustness and applicability our method.
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (Grant No. 92370204), in part by the National Key R&D Program of China (Grant No. 2023YFF0725001,, in part by the Muangzhou-HKUSS HKUST (GZ) Joint Funding Program (Grant No. 2023A03J0008), and in part by the Education Bureau of Guangzhou Municipality. This work was also supported by the Guangzhou Municipal Science and Technology Project (No. 2024A04J4390).
10

---

References
[1] Bengio. Y., Courville, A., and Vincent, P. (2013). Representation learning: review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822
[2] Börstell, C., Hörberg, T., and Östling, R. (2016). Distribution and duration of signs and parts of speech in swedish sign language. Sign Language & Linguistics, 19(2):4331196
[3] Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., and Bowden, R. (2018). Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784-7793.
[4] Camgoz. N. C., Koller, O., Hadfield, S., and Bowden, R. (2020a). Multi-channel transformers for multi- articulatory sign language translation. Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 301-319. Springer.
[5] Camgoz, N. C., Koller, O., Hadfield, S., and Bowden, R. (2020b). Sign language transformers: Joint end-to- end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023 10033.
[6] Chen, C., Guo, W., Ma. C., Yang. Y., Wang. Z., and Lin, C. (2021a). semg-based continuous estimation of finger kinematics via large-scale temporal convolutional network. Applied Sciences, 11(10):4678.
[7] Chen, R., Chen, Y., Guo, W., Chen, C., Wang. Z., and Yang. Y. (2021b). Semg-based gesture recognition using with strong robustness against forearm posture. In 2021 IEEE International Conference on Real-time Computing and Robotics (RCAR), pages 275-280. IEEE.
[8] Chen, Y., Wei, F., Sun, X., Wu, Z., and Lin, S. (2022). A simple multi-modaligg transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition. pages 5120- 5130.
[9] Cheng, Y.. Wei, F., Bao, Chen, D., and Zhang, W. (2023). Cico: Domain-aware sign language retrieval via cross-lingual contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9906-19926.
[10] Chipman, H. A.. George, E. I., McCulloch, R. E., and Shively, S. (2022). mbart: multidimensional monotone bart. Bayesian Analyxis, 17(2)511-544.
[11] Cohen, I., Huang. Y., Chen, Benesty, J., Benesty, J., Chen, J., Huang, Y., and Cohen, (2009). Pearson correlation coefficient. Noise reduction in speech processing, pages 1 1-4.
[12] Dai, L.. Liu, H., and Xiong, H. (2024). Improve dense passage retrieval with entailment tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.
[13] Dai, L., Ma, L.,, Qian, S., Liu, H., Liu, Z., Xiong, H. (2023). Cloth2body: Generating 3d human body mesh from 2d clothing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15007-15017.
[14] Fu, B., Ye, P., Zhang, L., Yu, P., Hu, C., Shi, X., and Chen, Y. (2023). A token-level contrastive framework for sign language translation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE,
[15] Gan. S. Yin, Y.. Jiang. Z., Xia, K.. Xie, L.. and Lu, S. (2023). Contrastive learning for sign language recognition and translation. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 763-772.
[16] Graves, A., Fernández, S., Gomez, F., and Schmidhuber. J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, pages 369-376.
[17] Guo, W. Li, Z., Yang, Y., Wang. Z., Taylor, R. H., Unberath, M., Yuille, A., and Li, Y. (2022). Context- enhanced stereo transformer. In European Conference on Computer Vision, pages 263-279. Springer.
[18] Hao. A., Min. Y., and Chen. X. (2021). Self-mutual distillation learning for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11303_ 11312.
[19] Harish, B. and Manju, N. (2018). Hybrid feature selection method using fisher's discriminate ratio to classify internet traffic data. In Proceedings of the 4th International Conference on Frontiers of Educational Technologies, pages 75-79.

---

[20] Hu. H., Zhou, W., Pu. J., and Li, H. (2021). Global-local enhancement network for nmf-aware sign language recognition. ACM transactions on multimedia computing. communications, and applications (TOMM), 170):1-19.
[21] Jaiswal, A., Babu, R., Zadeh, M. Z., Banerjee, D., and Makedon, F. (2020). survey on contrastive self-supervised learning. Technologies, 9(1):2
[22] Jin, T. and Zhao, Z. (2021). Contrastive disentangled meta-lcarning for signer-indepemdent sign language translation. In Proceedings of the 29th ACM International Conference Multimedia, pages 5065-5073.
[23] Kalinkov, K., Ganchev, T., and Markova, V. (2019). Adaptive feature selection through fisher discriminant ratio, In 2019 International Conference on Biomedical Innovations and Applications (BIA), pages 1-4.
[24] Kan, J., Hu, K., Hagenbuchner, M., Tsoi, A. C., Bennamoun, M., and Wang, (2022). Sign language translation with hierarchical spatio-temporal graph neural network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3367-3376.
[25] Kay, W., Carreira, J. Simonyan, K., Zhang, B., Hillier, C., Vijyanarasimhan, S., Viola, F. Green, T., Back,
T., Natsev, P., al. (2017). The kineties human action video dataset. arXiv preprint (tt.::/70500500
[26] Kürzinger, I.., Winkelbauer, D., Li, L.., Watzel, T., and Rigoll, (2020). Cte-segmentation of large corpora for german end-to-end speech recognition. In International Conference on Speech and Computer, pages 267-278. Springer.
[27] Li, D., Rodriguez, C., Yu, X., and Li, H. (2020a). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1459-1469.
[28] Li, D., Xu, C., Yu, X., Zhang, K., Swift, B., Suominen, H., and Li, H. (2020b). Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. Advances Neural Information Processing Systems, 33:1 13:12033-2
[29] Lin, C., Chen, X., Guo, W., Jiang, N., Farina, D., and Su, J. (2022). A bert based method for continuous estimation of cross-subject hand kinematics from surface electromyographic signals. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 31:87-96.
[30] Lin, K.-Y,, Ding, H., Zhou, J., Peng, Y.-X., Zhao, Z., Loy, C., and Zheng, W.S. (2024). Rethinking clip- based video learners in cross-domain open-vocabulary action recognition. arXiv preprint arXiv:2403.01556)
[3I] Liu, W., Wen, Y., Yu, Z., and Yang. M. (2016). Large-margin softmax loss for convolutional neural networks. arXiv preprint
[32] Ma, C., Guo, W., Zhang, H., Samuel, O. W., Ji. X., Xu, L., and Li, G. (2021). A novel and efficient feature extraction method for deep learning based continuous estimation. IEEE Robotics and Automation Letters, 6(4).7341-7348.
[33] Min. Y.. Hao, A., Chai, X., and Chen, X. (2021). Visual alignment constraint continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11542 11551.
[34] Papineni, K., Roukos, S., Ward, T., and 7hu, W.-J (2002). Bleu: method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting the Association for Computational Linguistics, pages 311-318.
[35] Pizzuto, E. (2003). Review "the hands are the head the mouth the mouth as articulator in sign languages" by penny boyes braem and rachel sutton-spence (eds.). Sign Language & Linguistics, 6(2):284-289.
[36] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,
(2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR.
[37] Schroff, F., Kalenichenko, D., and Philbin, J. (2015). Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823.
[38] Sedgwick, P. (2014). Spearman's rank correlation coefficient. Bmj. 349.
12

---

[39] Van der Maaten, L. and Hinton, G. (2008). Visualizing data using L-sne. Journal of Machine Learning Research, 9(11).
[40] Vinson, D. P., Thompson, R. I.., Skinner, R., Fox, N., and Vigliocco, G. (2010). The hands and mouth do not always slip together in british sign language: Dissociating articulatory channels in the lexicon. Psychological Science, (8))11558-114
[41] Wang. Y., Liu, S., Wang. M., Liang, S., and Yin, N. (2024). Degree distribution based spiking graph networks for domain adaptation. arXiv preprint arXiv:2410.006
[42] Wong, R., Camgoz, N. C., and Bowden, R. (2024). Sign2GPT: Leveraging large language models for gloss-free sign language translation. In The Twelfth International Conference on Learning Representations.
[43] Xie, J., Girshick, R., and Farhadi, A. (2016). Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478 487. PMLR.
[44] Yao, H., Zhou, W., Feng, H. Hu, H. Zhou, II, and Li, H. (2023). Sign language translation with iterative prototype. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 15592-15601.
[45] Ye. J., Jiao. W. Wang. X., and Tu, Z. (2023a). Scaling back-translation with domain text generation for sign language gloss translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguisticy, pages 463 463-476, Dubrovnik, Croatia. Association for Computational Linguistics.
[46] Ye. J.. Jiao, W., Wang. X., Tu, Z.. and Xiong. H. (2023b). Cross-modality data augmentation for end-to-end sign language translation. In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023. pages 13558 13571, Singapore. Association Computational Linguistics.
[47] Yin, A., Zhong, T., Tang, L., Jin, W. Jin, T. and Zhao, (2023). Gloss attention for gloss gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 2551-2562.
[48] Yin, K. and Read, J. (2020). Better sign language translation with tranCerraefoomrer. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5975-5989, Barcelona, Spain (Online). International Committee Computational Linguistics.
[49] Zhang, B., Müller, M., and Sennrich, R. (2023a). Slunet:: A simple unified model for sign language translation. In International Conference Learning Representations.
[50] Zhang, B., Müller, M., and Semnrich. R. (2023b). Sltunet: A simple unified model for sign language translation. arXiv preprint X3xx:3305001778
[51] Zhang, H., Sun, Y., Guo, W., Liu, Y., Lu, H., Lin, X., and Xiong, H. (2023c). Interactive interior design recommendation via coarsc-to-fine multimodal reinforeement learning. arXiv preprint arXiv:2310.0728)
[52] Zhou, B., Chen, Z., Clapés, A., Wan, J., Liang, Y., Escalera, S., Lei, Z., and Zhang, D. (2023). Gloss-free sign language translation: Improving from tisual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20871-20881.
[53] Zhou, H., Zhou, W., Qi, W., Pu, J., and Li, H. (2021a), Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1316-1325.
[54] Zhou, H., Zhou, W., Zhou, Y., and Li, H. (2021b). Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedio, 24:768-779.
[55] Zhou, J., Ma. Lin, K. Y., Qiu, R., Wang. Z., and Liang. (2024). Mitigating the human-robot domain discrepancy visual pre-training for robotic manipulation. arXiv preprint arXiv:2406. 14235.
13

---

A Appendix
A.1 Hyper-parameters of Baselines
Sign Language Transformers Baseline: Table 4 presents the hyper-parameters of Sign Language Transformers used in this work.
Parameter
PHOENIX-2014T CSL-Daily
encoder-layers
3
decoder-layers
3
attention heads
8
8
ctc-layers
1
hidden size activation function
512
512
gelu
gelu
learning rate
1 -003
1- 1-10-3
Adam B
(0.9, 0.98)
(0.9, 0.98)
label-smoothing
0.1
0.1
max output length
30
50
dropout batch-size
0.3
0.3
128
128
Table 4: Hyperparameters of Sign Language Transformer models. models...
Gloss-Free Sign Language Translation Baseline: The Gloss-Free Sign Language Translation
(GFSLT) model incorporates various modules designed for processing sign language input without the of glosses. Below is the detailed architecture used in this work:
Module
Stride
Kernel
Output Size
Sign Input Resnet w/o fc
B x T X 224 x 224 x 3
T x 512
Conv ID-BNID-RELU
1
5
B X T X 1024
MaxPooling
2
2
B X T X 1024
T X 1024
ConvID--BNID-RELL
7
MaxPooling ID
4 X 1024
Linear-BNID-RELU
B T X 1024
Transformer Encoder
B T X
Text Input
B U
Word Embedding
Bx 1024
Transformer Decoder
Bx x 1024
FC
BxU C
Table 5: Detailed Gloss-Free SLT (GFSLT) Framework. represents batch size, 7 denotes
length of the longest input sign video in the batch, and U is the length of the longest input text in the batch. It is copied from GFSLT-VLP (52)
A.2
Parameter Sensitivity Analysis of the SignCI.
A.2.1 Sensitivity Analysis on Dynamically Estimated Margin
The margin for negative sampling dynamically depends the estimated average margin of each gloss, calculated as (iframmess/eet(txex)) x speech-to-gestee to-gturrr Zipf's factor, with a minimum threshold set 10. The Zipf's factor, set 2.3, refers the speech-tt-gesture application of Zipf's Law.
calculated the distribution of the dynamically estimated margin, with the results displayed in the table below. A more detailed distribution can be seen in Figure 6
14

---

leder ofittt
Jath
Madge
Figure 6: The distribution of the estimated margin during training the PHOENIX-201 dataset. The green distribution represents our current paper's s method (factor = 2.3), while the orange distribu- tion shows the ground truth calculated based on gloss annotations.
Experiment Setup: To conduct principled analysis, evaluated the threshold values [0, 10, 20, 30, 40, 50]. Here, a threshold of O indicates that the margin is dominated by the dynamically estimated margin, while threshold of 50 suggests dominance by the fixed threshold.
dataset due to resource limitations. The results, as shown in the table below, indicate that SignCL is not sensitive to threshold parameter, with variance of 0.062.
Experiment Results: We uniformly trained for 80 epochs on the PHOENIX-200
Threshold
0
10
20
30
40
50
B@4
17.24 17.63
17.55
17.63
17.13
17.11
Table 6: Threshold sensitivity analysis results.
Zipf's factor
GT
2.3
3
4
B B@4
17.45
17.89
17.63
17.29
16.26
Table Sensitivity Zipf's factor.
A.2.2 Sensitivity Analysis on integrating SignCL into the SLT framework.
As shown in Eqn. 6 we vary the hyperparameter over the range [10-3, 10-2, 10-1, 100, 100] and conduct repeated experiments on the PHOENIX-20144 dataset with GFSLT-VLP.
As shown in Figure 7 excessively incorporating SignCL into the model can negatively impact the SLT task. Empirically, we find that À = 10-2 achieves balance between reducing representation density and improving translation performance.
A.3 Ablation Studies
We conduct ablation studies to investigate the impact of different loss components in the +SignCL approach during both the pretraining and fine-tuning stages. It is copied from Tabel 2 but with an ablation perspective.
A.4
Correlation between Representation Density and Recognition Performance
A.4.1
The efficiency of sign-gloss mapping building up
To calculate SDR, we need to establish the mapping relationship between input frames and gloss categories. This section presents performance our trained gloss-sign aligner. The experimental methodology follows the approach outlined in XmDA [46).
15

---

GFSLT-VLP Baseline
22.5
+SignCL
BLEU Score
22.0
21.5
21.0
20.5
10-3
10
10
10
10
X
Figure 7: The effect of the hyperparameter À on BLEU scores. the grey dashed line indicates the baseline performance GFSLT-VLP, while the red solid line represents the performance with SignCI. integrated.
Pretraining Stage
Finetuning Stage
Density SDR
Performance B B@41
VLP Loss SignCL Loss
SLT Loss
SignCL Loss
R@Lt
72.83
38.67
18.53
63.23
39.12
18.71
X
X
68.53
42.97
21.25
62.68
49.25
22.69
X
69.54
41.78
19.81
X
63.67
44.52
20.03
62.73
48.17
22.16
62.32
49.23
22.74
Table 8: Ablation study on the impact of different loss components in the +SignCL approach.
A.4.2 Correlation Coefficient
We analyze the relationship between the representation density of individual glosses and their recog- nition accuracy using the Sign Language Transformer on the PHOENIX-20147 dataset, leveraging the Self-Mutual Knowledge Distillation (SMKD) method for feature extraction. We compute the following correlation coefficients:
Pearson Correlation Coefficient [II]: This measures the linear relationship between recognition accuracy Acc(G,) for gloss G2 and the density metric SDR(G.), calculated as:
Σ(Aco(G.)
((Aco(..) - Acc)3 ((SDR( - SDR)
Spearman's 's Rank Correlation Coefficient [38]: This assesses the monotonic relationship between two datasets by considering the rank order of values.
WER
Dataset
Train Test Dev
PHOENIX-20147 8.68 8.03 25.28 CSL-Daily
9.23 8.39 29.32
Table 9: Evaluation of the gloss-sign aligner effectiveness and generalizability with WER (%) (the lower the better).
16

---

As shown in Table TO both correlation coefficients indicate medium negative correlation between the Sign Density Ratio (SDR) and sign recognition performance (Acc). This suggests that higher representation density correlates with poorer recognition performance. Specifically, Inter-Gloss Distance (D shows strong positive correlation, meaning that greater distances between different glosses correlate with better recognition performance. All Spearman P-values are lower than 0.01, confirming the high confidence in the non-randomness of these correlations.
PHOENIX-201
Cai- Dintro
Correlation Acc
Ditter Distro
SDR 4
G. t
SDR
Pearson
0.36
-0.22
-0.35
0.30
-0.14
-0.20
Spearman p
0.43
-0.24
-0.34
0.45
-0.16
-0.22
P-value
2.6E-17
5.5E-6
4.7E-11
6.6E-19
3.0E-3
-2.7E-5
Table 10: Correlation analysis between sign recognition performance and density metrics.
A.4.3 More Experiment Results on Sign Language Transformer
In this section, we present additional experimental results using the Sign Language Transformer (NSLT) the CSL-Daily dataset to further validate the effectiveness of the proposed SignCL strategy. We compare various feature extraction methods to assess their representation density and translation performance.
Density SDR
Performance
Feature Type
SLR(WER L)
Joint-SLT
NSLT
+SignCL(ours)
Gloss-based Feature Extraction
Sign Recognition 5
74.07
29.59
21.32
17.68
19.02
Self-Mutual KD
33
66.23
25.38
22.79
19.35
20.23
Gloss-free Feature Extraction
I3D Pretrained
47
83.33
61.74
14.17
11.70
12.81
VLP Pretrained 52
92.59
69.72
12.73
10.73
12.14
+ SignCL (ours)
81.30
63.33
14.76
12.04
13.51
Table 11: Comparative analysis of representation density and performance on the PHOENIX-2014T dataset. "+SignCL (ours)" indicates the inclusion of the proposed contrastive learning strategy during
VIP feature extraction or NSLT training processing.
Density SDR
Performance
Feature Type
SLR(WER L)
Joint-SLT NSLT
+SignCL(ours)
Gloss-based Feature Extraction
Self-Mutual KD
33
48.34
29.52
11.61
8.97
10.35
Gloss-free Feature Extraction
VLP Pretrained [22]
76.55
85.78
2.93
1.82
2.19
+ SignCL (ours)
68.39
80.71
3.18
2.29
2.53
Table 12: Comparative analysis of representation density and performance on the CSL-Daily dataset. The Self-Mutual KD features are provided by XmDA (46) and the VLP feature is reproduced with official code. Due to the incomplete open source of the CSL-Daily dataset, we were unable to obtain features for Sign Recognition and I3D Pretraining.
A.5 Broader Impacts
This paper focuses on research in sign language translation, which has the potential to significantly benefit individuals who are deaf or hard hearing. By improving the accuracy and efficiency of sign language translation, our work can facilitate better communication between individuals with hearing impairments and the broader community. This can help break down communication barriers, promoting inclusivity and equal opportunities in various social, educational, and professional settings.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the main contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Appendix, we discuss the limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results the Methodology. Experiment, and Appendix sections. We will release the code upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We commit to releasing the code and models upon acceptance of the paper. All the data and baselines are based on open-source benchmarks.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
Poolic//guides/Codeuubbbsss wtt.:eeewuumiisioooooooooooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplggguuiieeeeee
Stleee//CoSSSSbmsssS.S.. www.ssssiicggoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details necessary understand the results in the Methodology, Experiment, and Appendix sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because performing multiple runs for each experi- ment would too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments are conducted using PyTorch on 8*NVIDIA A800 GPUs for about 12 hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:[Yes)
Justification: We adhere the NeurIPS Code of Ethies, since the paper does not include any content or practices that violate ethical guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts of the paper in the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve data models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits the creators original owners of all used assets, and properly respects the license and terms of use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
No new assets are introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects, thus there are no study participants, no risks to disclose to subjects, and no need for Institutional Review Board (IRB) approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "67tdK56jkL", "reviewer": "Reviewer_rTJg", "summary": "The main content of this paper is about improving the performance of Gloss-free sign language translation. The author discovered the representation density problem in sign language translation, that is, in the feature space, the semantic visual representations of different sign language gestures tend to be closely clustered together, making it difficult for the gloss-free method to distinguish different gestures, thus significantly affecting the translation performance. To solve this problem, the paper proposes a simple and effective contrastive learning strategy SignCL, which encourages term-free models to learn more discriminative feature representations in a self-supervised manner. Experimental results show that SignCL is able to significantly reduce representation density and improve performance in different translation frameworks.", "questions": "see Weaknesses", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors discovered the representation density problem for the first time in the field of sign language and conducted a detailed analysis to show that this problem does affect the performance of sign language translation. These findings will help advances in the field of sign language processing.\n  \n- Based on this finding, the authors proposed a contrastive learning method to improve the representation density problem. Experiments show that this method works well in the gloss-free setting.\n  \n- The author promises to open source the code and model", "weaknesses": "- The core of the contrastive learning method in this paper is the selection of corresponding positive and negative samples. The analysis and supplement of strategy selection can make this work more perfect. For example, what is the impact of choosing different distance parameters in formula 4? In addition, the selection of negative examples does not seem to be limited to a single video. For example, randomly selecting other frames with the same signer may be a better choice.\n  \n- The contrastive learning method proposed by the author can be considered as a method to enhance the representation of sign language videos. This method should also have a certain effect on the gloss-base method, although the benefit may not be as large as that in the gloss-free setting. I would be happy to see the author add some relevant data, which can expand the scope of application of this paper's method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed comments and insightful suggestions. We provide point-wise responses to your concerns below.\n\n**W1: The analysis and supplement of strategy selection can make this work more perfect**\n\nThank you for your suggestions. We have added a systematic and sensitivity analysis of the sampling strategy.\n\n1. Before we proceed, let's briefly revisit some details from Formula 4: margin =  max(10, len(frames)/len(text)* 2.3).\n* The margin for negative sampling **dynamically** depends on the estimated average margin of each gloss (i.e., len(frames)/len(text) * speech-to-gesture Zipf’s factor) and a minimum threshold (i.e., 10). The Zipf’s factor, set as 2.3, refers to the speech-to-gesture Zipf’s Law. \n* We calculated the distribution of the dynamically estimated margin, with the results shown in the table below. A more detailed distribution can be seen in the [attached PDF](https://openreview.net/attachment?id=LU1t0zCyyb&name=pdf).\n\n| **Margin**|[0, 10)|[10, 20)|[20, 30)|[30, 40)|[40, 50)|[50, ∞)|\n|----|:--:|:--:|:--:|:--:|:--:|:--:|\n| **Count** | 113 | 3460 | 3264 | 230 | 23 | 6 |\n* Only 1.6% fall into the [0, 10) range, which means that the margin is primarily determined by the estimated average frames of each gloss in our paper (i.e., len(frames)/len(text) * 2.3).\n\n\n2. Sensitivity analysis on the threshold: We designed a minimum threshold to ensure the margin is not too small.\n* Experiment setup: To make the analysis more principled, we evaluated the threshold values at [0, 10, 20, 30, 40, 50]. Note that threshold=0 and threshold=50 indicate that the margin is dominated by the estimated margin or the fixed threshold, respectively.\n* Experiment results: We uniformly trained for 80 epochs on PHOENIX-2014T due to resource limitations. The results in the table below indicate that SignCL is not sensitive to the threshold parameter, with a variance of 0.062.\n\n|  |  | |  |  | ||  \n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| **Threshold** | 0 | 10  | 20  | 30 | 40  | 50 |\n| **B@4**  | 17.24| 17.63| 17.55| 17.63| 17.13| 17.11|\n\n3. Sensitivity analysis on dynamically estimated margin: We use text length and Zipf’s factor to estimate the average frames of each gloss (gloss-free setting).\n* Experiment setup: To make the analysis more principled, we first use gloss labels to calculate the ground truth margin distribution for PHOENIX-2014T (green in Figure 8), with a specific Zipf’s factor of 2.1. We then evaluated the threshold values at [1, GT, 2.3, 3, 4]. Note that Zipf’s factor = 1 means we use len(frames)/len(text) directly to estimate the margin, while GT represents using the ideal len(frames)/len(gloss) to determine the margin (Zipf’s factor = 2.1).\n* Experiment results: The results show that using Zipf’s factors between 1 and 4 does not lead to significant differences. When Zipf’s factor is set to 8, there is a noticeable drop in performance due to the margin being too large, which leads to a high probability of sampling from the same negatives.\n\n|  |  | |  |  | ||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| **Zipf’s factor** |1 | GT | 2.3 | 3| 4 | 8 |\n| **B@4**  |17.45|17.89|17.63|17.29| 17.10| 16.26  |\n\n4. Conclusion for sensitivity analysis.\n* We noticed that SignCL is not sensitive to the threshold and Zipf’s factor.\n* We believe this insensitivity is because the contrastive function inherently accommodates variability by focusing on relative differences rather than the absolute correctness of positive and negative pairs. The size of the margin boundary does not significantly affect the overall performance of contrastive learning, as long as it is not too large or too small.\n* To systematically determine the optimal margin, we suggest setting it approximately equal to the mean ± standard deviation of the estimated average frames of each gloss based on a given dataset, e.g., len(frames)/len(text)* 2. The threshold can be mean - standard deviation.\n\n**Q1: Why not randomly select from other videos with the same signer?**\n* This is an insightful question that explores alternative sampling methods. We did consider this approach. However, sampling from other videos with the same signer would require additional labels to identify the signer, which limits the applicability of SignCL. We believe maintaining a gloss-free setting is important as it represents a significant trend in the field.\n* As a supplementary experiment, we attempted to randomly select other frames within the same batch as negative samples (in-batch sampling). Unfortunately, this approach resulted in worse performance on PHOENIX-2014T and CSL-Daily datasets.  This decline is because the model may use non-sign language features for contrastive learning, such as signer characteristics and background elements.\n* We have found that sampling within the same video has distinct advantages. It natively ensures consistency of non-sign features, allowing the contrastive learning process to focus on sign-specific features. This is our best practice.\n\n**W2: Looking forward to more results on gloss-based settings**\n\nThank you for your insightful question. We appreciate your interest in applying our contrastive learning method to the gloss-based approach. We have shown some results in Figure 3 and Appendix A.3.3, which indicate the proposed SignCL method also benefits the gloss-based method. To further validate this, we have applied SignCL to gloss-based feature extraction (e.g., Self-Mutual KD[25]) and translation methods (e.g., Joint-SLT[5]. The results indicate that SignCL can indeed enhance fully gloss-based SLT).\n\n| **Methods / Feature Extraction**  | **PHOENIX-2014T** | |\n|--|--|--|\n|  | **WER ↓**  | **B@4 ↑**   |\n| Joint-SLT / Self-Mutual KD| 25.38 | 22.79 |\n| &nbsp;&nbsp;&nbsp;&nbsp;+ SignCL into Feature Extraction | 24.76  | 23.23 |\n| &nbsp;&nbsp;&nbsp;&nbsp;+ SignCL into Downstream Training | 25.12  | 22.92  |\n| &nbsp;&nbsp;&nbsp;&nbsp;+ SignCL into Two States  | **24.58** | **23.46** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjvqX5de9U", "reviewer": "Reviewer_bAiC", "summary": "This paper focuses on gloss-free sign language translation (SLT) and is largely motivated by the large cost to annotating glosses. The authors discussed a so-called \"representation density problem\" in gloss-free SLT where semantically dissimilar signs appear in a similar part of the latent space. \n\nThe first technical contribution relates to a metric they introduce to quantify the representation problem, which uses Fisher’s Discriminant Ratio (FDR) to assess the difference between the inter- and intra-gloss features. To improve the separability, as their second contribution, they propose SignCL, a contrastive learning strategy that encourages the model to learn more discriminative feature representations in a self-supervised manner. \n\nThe results show significant improvements in BLEU scores on the CSL-Daily dataset compared to state-of-the-art methods.", "questions": "There are two types of issues that are highlighted: (1) challenges where signs have the same hand motion but different facial expression and (2) challenges where the same hand shape is used but with different motions (e.g., piano vs typing). Can you talk specifically about how the proposed approaches help with face and motion errors? \n\nI noticed some odd results in A.3.1. The WER results for train/test/dev are hugely different: 8.68/8.03/25.28. How is it the case that 'dev' has a 25% WER but train and test have a hugely improved 8%?", "limitations": "Their responses seem reasonable.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Intuitively the motivation for the paper makes sense, the methods are relatively straightforward (in a good way), and their solution seems to work well. The authors provide a good amount of detail (especially in the appendix) in case someone wanted to reproduce.", "weaknesses": "A primary motivating example in the paper is that “RECIPROCATE” and “REVENGE” live in similar parts of the latent space because the hand motions are similar (even though the facial expressions are different). The proposed contrastive approach takes samples from different parts of the same sentence, where each sign should contrast with dissimilar signs in that sentence. Perhaps this is just an issue with the motivation, but it doesn't seem like the approach would help with the motivating example. Similarly, it doesn't seem like the contrastive approach should be any more helpful with gloss-free SLT vs gloss-based SLT. \n\nI am wary about some of the t-SNE visualizations. The Sign Density Ratios don't change that much between approaches, and given that t-SNE is known to be deceptive at times, I wonder if these visualizations are representative or overstated. \n\nSec 4.3 on qualitative analysis is interesting but is too anecdotal. It would be useful to have a much deeper set of analyses here covering a much larger set of examples. \n\nOverall, I think this paper is certainly not bad, but I'm on the fence about whether the novelty or depth is sufficient for this venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed comments and insightful suggestions. We provide point-wise responses to your concerns below.\n\n**Q1: How do the proposed approaches help with face and motion errors?**\n* Thank you for your question. Our SignCL approach samples positive and negative examples from a single video, which means that non-sign features such as the signer’s background and camera angle remain consistent. Consequently, when the contrastive learning framework distinguishes between positive and negative samples, the model naturally focuses on sign-specific features, such as the differences in facial expressions and hand movements, rather than on non-sign features like the signer or background. \n* To sum up, although SignCL is not explicitly designed to address face and motion errors (due to they are not the primary focus of this paper), the strategy of sampling from the same video inherently directs the model's attention to more sign-specific features, such as subtle differences in facial expressions and motions.\n* We are happy to see that future work will design face and motion-aware sign embedding backbones and apply SignCL to them. To the best of our knowledge, frame-based sign embedding models using CNN and ViT architectures are currently the best practice for sign language translation.\n\n**Q2: Why are there odd results in A.3.1?**\n\n* Sorry for the confusion. To briefly recap, the purpose of the gloss-sign aligner is only to derive labels to calculate SDR, not for the model itself. We want to ensure the SDR is as accurate as possible, so we mix the test set into the training set to train the gloss-sign aligner. The dev set is reserved for evaluating the training process of the gloss-sign aligner and is not used in the SDR calculation. Therefore, the results for the dev set may appear less optimal.\n\n**W1: The proposal SignCL approach seems not  aligned with the primary motivating example** :\n\nWe respectfully disagree.  Here is our explanation:\n* The examples in Figure 1 or Figure 5 are intended to aid in illustrating the representation density. We do not explicitly tackle these examples in our method. We apologize for any confusion, and we will emphasize these points more clearly in future versions.\n* The key contribution of this paper is identifying the representation density problem for gloss-free SLT for the first time. We discuss the overall discriminability of feature representation of sign gestures, measured by the Sign Density Ratio (SDR). \n* The proposed SignCL serves as a relative and straightforward attempt to improve the discriminability of representations in the gloss-free setting.\n\n**W2:  It doesn't seem like the contrastive approach should be any more helpful with gloss-free SLT vs gloss-based SLT**:\n* Of course, SignCL can also work for gloss-based methods, and we have results in Section 4.1 and Appendix A3.3 that show this. However, the benefit may not be as large as that in the gloss-free setting because gloss-based methods use costly gloss annotations and CTC loss as a discriminative optimization objective.\n* The motivation of SignCL is that gloss-free methods suffer from worse visual representation due to the lack of costly gloss annotations. SignCL is designed in a self-supervised manner and does not rely on any gloss information to fit the gloss-free setting.\n\n**W3: t-SNE visualizations might be misleading**:\n\nWe understand the t-SNE visualizations can be deceptive at times. We want to emphasize that t-SNE is not the primary basis for our conclusions; it serves as a visual aid to better illustrate the representation density problem.\n* The conclusions about representation density and performance drop are derived from the quantitative metrics and experiment results, especially when comparing gloss-free and gloss-based approaches.\n* In Section 2, we comprehensively investigate the representation density problem by using the Sign Density Ratio to measure feature discriminability within existing sign feature extraction methods. We also use sign recognition and translation tasks to analyze how different densities of representations affect performance.\n* We believe the representation density problem is not overstated. As shown by the experiment results in Figure 3 and Appendix A.3.3, gloss-free features indeed exhibit higher SDR and significantly poorer sign language recognition and translation performance.  It highlights a potential bottleneck in restricting the performance of gloss-free SLT.\n\n**W4: Qualitative analysis in Section 4.3 is anecdotal and needs more depth**:\n* Thank you for your insight. We are open to adding more cases and analyses. Unfortunately, the field of sign language research currently lacks an appropriate benchmark that is annotated with a large number of gestures that are similar in motion but distinct in meaning.\n* However, this paper focuses on the overall discriminability of feature representation of sign gestures. We would like to emphasize that, in addition to the visual analysis, we have provided overall sign recognition accuracy as quantitative evidence.\n* In Section 2 and Appendix A.3.3, we examine sign recognition accuracy and translation performance when applying SignCL to GFSLT-VLP pretraining. Figure 3 and Tables 7/8 demonstrate that the representation has better discriminability (lower SDR) and higher performance in sign language recognition after applying SignCL to the GFSLT-VLP baseline. We will make this qualitative analysis a separate section and make it clearer in future versions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urKYA7eW6r", "reviewer": "Reviewer_UuUf", "summary": "This paper addresses the challenge of gloss-free sign language translation (SLT) by identifying and tackling the \"representation density problem\". The authors observe that visual representations of semantically distinct sign gestures tend to be closely packed in feature space, making it difficult for gloss-free methods to distinguish between different gestures. To address this, they propose a contrastive learning strategy called SignCL, which encourages models to learn more discriminative feature representations in a self-supervised manner. The paper demonstrates significant improvements in BLEU scores across various translation frameworks on the CSL-Daily dataset.", "questions": "1 How does the performance of SignCL vary with different choices of the margin parameter in the sampling strategy? Is there a systematic way to determine the optimal margin for a given dataset?", "limitations": "The authors discuss their limitations in the Appendix。", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper identifies a novel problem (representation density) in gloss-free SLT and proposes an innovative solution (SignCL) to address it. The proposed SignCL method shows substantial improvements in SLT performance without requiring additional model parameters, potentially advancing the field of gloss-free SLT.\n\n2 The authors conduct thorough experiments to demonstrate the existence of the representation density problem and the effectiveness of their proposed solution. They evaluate their method across multiple datasets and SLT frameworks.\n\n3 The paper is well-structured and clearly written. The problem statement, methodology, and results are presented in a logical and easy-to-follow manner.", "weaknesses": "1 The sampling strategy for SignCL makes assumptions that may not always hold true in real-world scenarios. Specifically, it assumes that adjacent frames always belong to the same sign gesture and that frames far apart always represent different semantics. This approach overlooks the possibility of rapid transitions between gestures or the repetition of gestures over time, which could lead to incorrect positive or negative samples. \n\n2 The SignCL method relies on several manually defined parameters and thresholds, such as the margin for determining positive and negative samples. The paper lacks a rigorous justification for these choices or an analysis of how sensitive the method is to these parameters. A more principled approach to parameter selection or a comprehensive sensitivity analysis would strengthen the scientific basis of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed comments and insightful suggestions. We provide point-wise responses to your concerns below.\n\n**W1: Lack of comprehensive sensitivity analysis on sampling strategy**\n\nThank you for your insightful suggestions. We conducted an additional comprehensive sensitivity analysis on the sampling strategy.\n\n**1. Before we proceed, let's briefly revisit some details from Formula 4**: margin =  max(10, len(frames)/len(text)* 2.3).\n* The margin for negative sampling **dynamically** depends on the estimated average margin of each gloss (i.e., len(frames)/len(text) * speech-to-gesture Zipf’s factor) and a minimum threshold (i.e., 10). The Zipf’s factor, set as 2.3, refers to the speech-to-gesture Zipf’s Law. \n* We calculated the distribution of the dynamically estimated margin, with the results shown in the table below. A more detailed distribution can be seen in the [attached PDF](https://openreview.net/attachment?id=LU1t0zCyyb&name=pdf) (green background).\n\n| **Margin**|[0, 10)|[10, 20)|[20, 30)|[30, 40)|[40, 50)|[50, ∞)|\n|----|:--:|:--:|:--:|:--:|:--:|:--:|\n| **Count** | 113 | 3460 | 3264 | 230 | 23 | 6 |\n* Only 1.6% fall into the [0, 10) range, which means that the margin is primarily determined by the estimated average frames of each gloss in our paper (i.e., len(frames)/len(text) * 2.3).\n\n\n**2. Sensitivity analysis on the threshold:** We designed a minimum threshold to ensure the margin is not too small.\n* Experiment setup: To make the analysis more principled, we evaluated the threshold values at [0, 10, 20, 30, 40, 50]. Note that threshold=0 and threshold=50 indicate that the margin is dominated by the dynamically estimated margin and the fixed threshold, respectively.\n* Experiment results: We uniformly trained for 80 epochs on PHOENIX-2014T due to resource limitations. The results in the table below indicate that SignCL is not sensitive to the threshold parameter, with a variance of 0.062.\n\n|  |  | |  |  | ||  \n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| **Threshold** | 0 | 10  | 20  | 30 | 40  | 50 |\n| **B@4**  | 17.24| 17.63| 17.55| 17.63| 17.13| 17.11|\n\n**3. Sensitivity analysis on dynamically estimated margin:** We use text length and Zipf’s factor to estimate the average frames of each gloss (gloss-free setting).\n* Experiment setup: To make the analysis more principled, we first use gloss labels to calculate the ground truth margin distribution for PHOENIX-2014T (green in Figure 8), with a specific Zipf’s factor of 2.1. We then evaluated the threshold values at [1, GT, 2.3, 3, 4]. Note that Zipf’s factor = 1 means we use len(frames)/len(text) directly to estimate the margin, while GT represents using the ideal len(frames)/len(gloss) to determine the margin (Zipf’s factor = 2.1).\n* Experiment results: The results show that using Zipf’s factors between 1, 2.3 and the GT margin does not lead to significant differences. When Zipf’s factor is set to 4, there is a noticeable drop in performance due to the margin being too large, which reduces the negative sampling interval (high probability of sampling from the same negatives).\n\n|  |  | |  |  | |\n|---|:---:|:---:|:---:|:---:|:---:|\n| **Zipf’s factor** | 1  | GT | 2.3   | 3   | 4   |\n| **B@4**  | 17.45| 17.89| 17.63| 17.29| 16.26|\n\n----\n**Q1:  How to systematically determine the optimal margin?**\n\nThank you for your insightful question. Based on our comprehensive sensitivity analysis, we found that SignCL is not sensitive to the threshold and Zipf’s factor. Therefore, we suggest setting the optimal margin approximately equal to the mean ± standard deviation of the estimated average frames of each gloss based on a given dataset, e.g., len(frames)/len(text) * 2. The threshold can be set to mean - standard deviation.\n\n----\n**W2: The sampling strategy could lead to incorrect positive or negative samples**\n\n* We agree that the sampling strategy might indeed produce errors in certain special cases. However, we would like to emphasize that a range of contrastive learning frameworks demonstrate that contrastive learning can still perform well even when there is noise in sampling strategies[a], like SimCLR[b] and MoCo[c]. \n\n* This robustness is because the contrastive function inherently accommodates variability by focusing on relative differences rather than the absolute correctness of positive and negative pairs. Special cases of incorrect positive or negative samples do not significantly impact overall performance. Our sensitivity analysis also indicates that variations in the margin have minimal effect.\n\n---\n[a] Contrastive Representation Learning: A Framework and Review\n\n[b] A Simple Framework for Contrastive Learning of Visual Representations\n\n[c] Momentum Contrast for Unsupervised Visual Representation Learning", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8C914dEUZm", "reviewer": "Reviewer_rKGZ", "summary": "This paper identifies a critical issue of representation density in gloss-free sign language translation. A series of models were employed to verify the existence of this problem. To address this, the author proposed a straightforward and effective solution, SignCL loss. This objective improves the discriminative representations of input videos, achieving promising performance in PhoenixT and CSL-Daily datasets.", "questions": "* The writing in the article is overly colloquial, and the presentation of Formula 4 may be confusing.  Additionally, there is an extraneous character \")\" in the second line of the title in Figure 4. \n* Line 236 claims that GFSLT \"incorporates CLIP and MBART for model pretraining and finetuning.\" However, to my knowledge, GFSLT does not utilize CLIP's pre-trained weights. \n* The performance of CSL-Daily in Table 2 is notably inconsistent. In Table 1, when Sign2GPT achieves R@L of 48.90, its B@4 is 22.52. However, in SignCL's CSL-Daily, despite achieving a similar R@L of 48.92, its B@4 drops to 16.16. Please verify the accuracy of these performance metrics.\n* The t-SNE visualization method is unclear. How should the SLT task be correctly labeled with gloss?\n*  The datasets PHOENIX-2014T and CSL-Daily are relatively small in size. Can the effectiveness of the method be validated on larger datasets like How2Sign and Open-ASL?", "limitations": "Yes, the limitations has been discussed in the main paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "* The paper is straightforward and comprehensible, and the proposed method is somewhat effective.\n* The adequate visualizations validate the effectiveness of the proposed objective. \n* The extensive experiments on both public benchmarks show plausible performance.", "weaknesses": "* The technical novelty and contribution are somewhat limited. The entire paper mainly introduces a contrastive loss (SignCL), which has been widely utilized in other various domains.\n* The proposed objective is only applied in the single model, which seems to lack persuasiveness. Experiments on more SLT models are essential to validate its effectiveness.\n* On line 118, why is there a need to optimize the test set? Why must we manually determine the best frame from $l_{v}$ and $r_{v}$ instead of utilizing CTC's gradient? For VLP, how are $l_{v}$ and $r_{v}$ generated?\n* In section 2.2, why are VLP, I3D, and SMKD methods grouped together? These methods should differ in terms of their training objectives. SDR of SMKD will naturally be low, because CTC loss is a discriminative loss. Achieving low SDR is essential for successful completion of the CSLR task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed comments and suggestions. We provide point-wise responses to your concerns below.\n\n**W1: The technical novelty and contribution are somewhat limited:**\nWe respectfully disagree with the statement that \"The entire paper mainly introduces a contrastive loss\". We emphasize our contributions below:\n* Introducing a contrastive loss is not the only contribution of this work. We have an entire Section 2 and extensive analyses dedicated to investigating the representation density problem in SLT.\n* The key contribution of this paper is identifying the representation density problem for gloss-free SLT for the first time. It highlights a potential bottleneck in restricting the performance of gloss-free SLT, which advances the SLT field.\n* Introducing SignCL as the second contribution. Even though it is based on a well-known contrastive learning method, it is a simple but effective solution to address the representation density problem.  We are the first to validate this, providing a direction for future gloss-free SLT to learn more discriminative representations.\n\n**W2: The proposed objective is only applied in the single model:**\nWe respectfully disagree with this conclusion. As described in line 13 and experiments listed in Section 4.1, we evaluate SignCL across multiple datasets and SLT frameworks, including both gloss-based and gloss-free settings. Reviewers rTJg and UuUf have recognized this contribution and even considered it as a strength.\n\n**Q1: how are $l_v$ and $r_v$ generated?**\u000bWe noticed that you have several questions about the sign-gloss forced aligner. We provide a clearer introduction to address your concerns.\n* To start, we want to emphasize that the purpose of the sign-gloss forced aligner is only to derive labels to measure the discriminability of the feature representations, not for the model itself. So for VLP, it does not need to produce $l_v$ and $r_v$; the entire Sign-Gloss Alignment consistently uses pre-trained models.\n* We mix the test set into the sign-gloss forced aligner training process and employ volunteers to manually determine the best frame to ensure accurate SDR evaluation when assessing different SLT approaches.\n* The generation of $l_v$ and $r_v$ has been covered in several previous works [a, b]. For specific details, you may refer to these studies. We highlight that, as shown in Appendix A.3, the sign-gloss forced aligner yields a Word Error Rate (WER) of 8.68, which is on par with human performance [c]. Even with this accuracy, we strive to ensure the most precise results on the test set.\n\n\n**Q2: Why are VLP, I3D, and SMKD methods grouped together?**\n* Section 2 aims to investigate the representation density problem within existing sign feature extraction methods, including both gloss-free and gloss-based methods.\n* We acknowledge your insightful observation that the SDR of gloss-based methods, such as SMKD, will naturally be lower because the CTC loss is a discriminative loss. We grouped them together to highlight that gloss-free SLT suffers from worse representation density compared to gloss-based methods. \n\n\n**Q3: Please verify the accuracy of these performance metrics in table 1 and table 2.**\n* We have re-calculated our metrics, and they are accurate. We believe the differences in R@L and B@4 metrics are due to the different aspects they measure. R@L focuses on the retrieval accuracy, while B@4 emphasizes the fluency and relevance of the generated translations. \n* Additionally, the characteristics of Chinese and German languages are quite different, which can impact the consistency of R@L and B@4 across different languages.\n* It is also worth noting that similar discrepancies have been observed in past methods. For example, Sign2GPT achieves an R@L of 42.36 and a B@4 of 15.40, whereas GFSLT-VLP achieves a similar R@L of 42.49 but a B@4 of 21.44 on PHOENIX-2014T.\n\n\n**Q4: Can the effectiveness of the method be validated on larger datasets like How2Sign and Open-ASL**\n* We believe that SignCL can still be effective on larger datasets, as larger datasets often lack costly gloss annotations to construct a discriminative loss. SignCL can serve as an effective optimization objective in this process, encouraging the learning of more discriminative representations of sign gestures.\n\n* We train on the How2Sign dataset for 40 epochs from scratch. In this limited experiment, the GFSLT baseline performance was 0.79 B@4, while applying SignCL as an additional optimization objective improved the performance to 1.58 B@4. These experiments are comparable to the results from [d], where they also trained from scratch.\n\n||  | |  | |\n|--|--|--|--|--|\n|  **Methods**  | **B@1** | **B@2**   |  **B@3**   |  **B@4**   | \n| H2S(no pretraining)[d]| 13.92 | 4.69 | 1.82| 0.86|\n| H2S(pretraining)[d]| 14.96 | 5.11 | 2.26  | 1.22|\n| GFSLT| 12.31 | 3.85 | 1.53  | 0.79|\n| &nbsp;&nbsp;&nbsp;&nbsp;+ SignCL into Finetuning | **17.36**  | **6.27** |**2.87**|**1.58**|\n| |  | |  | |\n\n\n\n**Suggestions And Typos:**\n* We apologize for the lack of clarity in the text and the typographical errors. We have revised our paper according to your suggestions to improve its readability and accuracy.\n\n* To clarify, Line 236 states that GFSLT-VLP \"incorporates CLIP and MBART for model pretraining and finetuning.\" What we mean is that GFSLT-VLP uses the pretraining strategies of CLIP and MBART to retrain the GFSLT model, not talking about using their pre-trained weights. We will make this point clearer in future versions of the paper.\n\n\nReferences:\n\n[a] CTC-segmentation of large corpora for German end-to-end speech recognition.\n\n[b] Cross-modality Data Augmentation for End-to-End Sign Language Translation\n\n[c] Achieving human parity in conversational speech recognition Microsoft Research Technical Report 2017\n\n[d] YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZHIsvNJEoT", "reviewer": "Reviewer_9u2m", "summary": "This work uses contrastive learning of sign language gestures to improve the discrimination power of learned representations of gestures.   This is motivated by showing, via TSNE projections, that gloss-free representations (that do not benefit from an intermediate representation), are less well represented that gloss-based representations.  After incorporating contrastive learning, either into the pre-training, fine-tuning, or both, performance improvements can be achieved for both sign-language recognition and for sign-language translation over the presented baseline.", "questions": "Figure 2 — is the reported SDR just for the highlighted signs in the figure or is it computed across all sign gestures?\n\nIs the SDR issue artificially inflated because of the dataset used?  For example, using a corpus of signing from weather forecasting with a limited vocabulary and many signs around a small number of topics?\n\nA missing limitation of the work is that a fixed margin is used to define the threshold for frame selection for the contrastive learning.  Whilst 20 frames might be sufficient,  as well as being impacted by closely repeating signs, could frame-selection also be impacted by signing rate? \n\nMy understanding is that two annotators select a frame representative for each gesture, and the main text said that details are in the appendix.  However, I did not see how you select the final frame from the two (potentially different) annotations?  Do you average the time stamps to pick the one on the middle?\n\n\nSome other observations:\n- Remove subjective qualifiers from your writing.\n- Your equations should be punctuated and flow with the text.\n- For Figure 2, I would highlight gloss-based vs. gloss-free approaches as at first glance approaches like SMKD look significantly better, and then later realized that this is expected since it is not a gloss-free approach.\n- For Figure 3, the binning needs to be explained in the caption.  Also is the axis label for the vertical axis correct?  Is this figure not showing sign recognition accuracy and sign translation accuracy? \n- It seems somewhat redundant including both Figure 1 and Figure 5.\n- I think it would be worth reinforcing in Section 2.1 when you talk about mixing the training and testing tests that the purpose of this is only to derive labels, not for the model itself.\n- Has the contrastive learning been used with the other baselines?  It seems that since the performance there is less good, there might be more room for improvement?", "limitations": "Yes the limitations are addressed adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is tackling an important problem as accurate sign-language recognition and translation will enable accessibility for the hard of hearing and those that need to interact with the hard of hearing.\n\nThe paper is well motivated from the perspective of demonstrating that the particular measure used to determine representation quality (SDR) is significantly lower for gloss-free representations, which is the focus of interest here.  However, the approach itself will have limited impact in the broader NeurIPS community.  The paper is using known methods applied to the representation of sign language. \n\nThe work is based on open data and the authors will release their code and model with the paper.  This will ensure reproducibility.", "weaknesses": "The word “significance” has specific meaning, and you should not claim that results are significant without properly showing that they are so.\n\nResults for only a single run are provided.  This is acknowledged in the check sheet with the justification that increasing the number of runs is computationally expensive.  However, the resources required are ~12 hours on 8x NVIDIA A100s, which does not seem excessive by current standards.  One solution might be to just use the best values for the hyper parameters, and show these are representative across runs.\n\nSome of the findings appear obvious -- of course representations of dissimilar gestures that project closely in embedding space will be difficult for a model to tease apart for downstream tasks.\n\nThere is a lot of emphasis and conclusions drawn from the visualizations, which are TSNE projections of the embeddings.  TSNE visualizations should be considered with a degree of caution, and there are better solutions if you are trying to consider the global structure of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed comments and insightful suggestions. We provide point-wise responses to your concerns below.\n\n**W1: Lack of proper statistical validation**\n\nThank you for your suggestion. We reran the representative experiments of Table 2 five times, using different seeds (23,77,256,1024,2056). The table below shows the average results (with a standard deviation). The results demonstrate the statistical significance of our experiments.\n\n| **Model**   | **R@L ↑**  | **B@1 ↑*| **B@2 ↑**  | **B@3 ↑**  | **B@4 ↑**   |\n|---|----|--|----|----|----|\n| GFSLT-VLP  | 38.61 (&plusmn;1.37) | 35.98 (&plusmn;1.00)    | 23.52 (&plusmn;0.88)    | 15.80 (&plusmn;0.79)    | 11.21 (&plusmn;0.54)   |\n| + SignCL into Finetuning  | 48.54 (&plusmn;1.14)    | 46.41 (&plusmn;0.79)   | 32.65 (&plusmn;0.51)    | 22.68 (&plusmn;0.42)    | 16.05 (&plusmn;0.16)   |\n\n**W2: Limited contributions**\n\nWe respectfully disagree with this. We provide our explanation below:\n* We are not merely \"using known contrastive learning methods applied to the representation of sign language.\" The key contribution of this paper is identifying the representation density problem for gloss-free SLT for the first time.\n* This discovery is neither obvious nor trivial. It is well-known that gloss-free methods in SLT lag significantly behind gloss-based approaches, but the reasons are still under investigation. We highlight that the representation density problem could be a bottleneck in restricting the performance of gloss-free SLT.\n* The insights provided in this paper for SLT have been recognized by Reviewers rTJg and UuUf, even though they expressed concerns about the sensitivity analysis on sampling parameters. \n* The simple but effective SignCL is the second contribution. Even though it is a straightforward solution to the representation density problem, we are the first to validate this and improve performance in two different frameworks by 39% and 46%, respectively. It will provide a direction for future gloss-free SLT.  Reviewer rTJg has noted that this approach is \"relatively straightforward in a good way.\"\n\n**W3: Some findings appear obvious and are heavily reliant on t-SNE visualizations**\n\nWe respectfully disagree with these statements. Here is our explanation:\n* Our findings are not obvious and limited: While it is evident that poor representation hinders a model's ability, whether gloss-free SLT faces a significant representation density problem needs to be researched.\n* Our emphasis and conclusions are not based on visualizations alone. We first used SDR to quantify the discriminative capability of various existing sign feature extraction techniques. Our conclusions are derived from these quantitative metrics and results, particularly when comparing gloss-free and gloss-based approaches. Visualizations only serve as a visual aid to better illustrate the representation density problem.\n\n**Q1: Is the reported SDR just for the highlighted signs in the figure or is it computed across all sign gestures?**\n\nAll reported SDRs in Figure 2 are computed across all sign gestures in the training set.\n\n **Q2: Is the SDR issue artificially inflated due to the dataset used?**\n\nWe believe the SDR issue is not artificially inflated. Our related conclusions have been consistently validated on the CLS-Daily Dataset, which is a multi-topic dataset. Moreover, many of our conclusions are derived from comparisons between gloss-free and gloss-based methods on the same dataset.\n\n**Q3: Could frame selection be impacted by the signing rate given the fixed margin?**\n\nThank you for your insightful question. We address this concern from the following aspects:\n* The margin is not fixed; it dynamically adapts based on the estimated average frames of each gloss. Therefore, it actually adapts to the signing rate. \n* Our sensitivity analysis shows that SignCL is not sensitive to the margin parameter, as evidenced by a variance of 0.062 when using different thresholds in [0, 10, 20, 30, 40, 50].\n\n**Q4: How to select the final frame from the two (potentially different) annotations?**\n\nWe apologize for the missing details. By default, we select the middle frame as the representative, and the two annotators first check if this default frame is a good representative. For interval [$l_v$,$r_v$], we ensure the current frame clearly represents a gesture and is different from the previously selected one. If the default frame does not meet these criteria, the annotators review the frames within the interval [$l_v$,$r_v$] proposed by the sign-gloss aligner. If a suitable frame cannot be found within this interval, the entire video is marked for discard. Only data that both annotators consistently agree upon as good will be used.\n**Q5: Is the axis label for the vertical axis correct?**\n\nWe apologize for the confusion. To clarify:\nIn Figure 3(a), the bar chart represents the SDR of different gloss bins (left axis), while the line represents the corresponding sign recognition accuracy (right axis).\nIn Figure 3(b), the left axis indicates the translation performance B@4.\n\n**Q6: Could there be more room for improvement?**\n\n*  We agree that achieving performance in gloss-free SLT similar to gloss-based approaches still has a long way to go. However, gloss-free methods have the advantage of not requiring costly gloss annotations, making it easier to scale up training datasets and perform large-scale pretraining. Then, the pretrained model can serve as a strong starting point for us to finetune on a small amount of high-quality annotated data.\n* We believe that addressing the representation density issue is crucial for effective pretraining. SignCL can serve as an effective optimization objective, encouraging the learning of more discriminative sign gesture representations.\n\n**Other suggestions:**\n\nWe appreciate your insightful suggestions. It will certainly help improve our paper. We will incorporate your feedback to refine our paper in the upcoming version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their detailed comments and insightful suggestions. We are encouraged that they find our paper identifies a **novel problem** (representation density) in gloss-free SLT [Reviewer rTJg and UuUf], and introduces a relatively straightforward SignCL to address this representation problem (**simple and effective method**) [Reviewer  rTJg, UuUf, bAiC and rKGZ].  

We would like to address some key concerns below:

1. We respectfully disagree that our contributions are limited.
>    * **We are not merely "using known contrastive learning methods applied to the representation of sign language."** The key contribution of this paper is identifying the representation density problem for gloss-free SLT for the first time.
>    * **This discovery is neither obvious nor trivial.** It is well-known that gloss-free methods in SLT lag significantly behind gloss-based approaches, but the reasons are still under investigation. We are the first to take a closer look at the representation of sign gestures and demonstrate that gloss-free methods suffer from worse representation density. We highlight that the representation density problem could be a bottleneck in restricting the performance of gloss-free SLT, providing a direction for future gloss-free SLT to learn more discriminative representations.
>    * Even though contrastive loss has been widely utilized in other domains, we believe SignCL is a very straightforward and effective solution to the representation density problem in the gloss-free setting. Our experiments show that it can improve performance in two different frameworks by 39% and 46%, respectively. This also highlights that representation density is a critical issue in gloss-free SLT.

2. Findings and conclusions are not merely based on t-SNE visualizations.
>    * In Section 2, we comprehensively investigate the representation density problem within existing sign feature extraction methods by using SDR to measure feature discriminability. We also use sign recognition and translation tasks to analyze how different densities of representations affect performance.
>    * Our conclusions are derived from these quantitative metrics and experiment results, especially when comparing gloss-free and gloss-based approaches. The visualizations are used to aid in illustrating the findings about the representation density.

3. Another main concern is the lack of analysis and supplement for the straightforward sampling strategy, which may lead to incorrect positive or negative samples in some certain special cases. We provide point-wise responses to this concern below:
>    * There is a misunderstanding that the margin in SignCL is fixed or heavily influenced by the threshold. In fact, the margin is dynamically evaluated based on the text length and a validated speech-to-gesture Zipf’s factor (i.e., len(frames)/len(text) * 2.3). The distribution of margins during training is shown in the [global PDF](https://openreview.net/attachment?id=LU1t0zCyyb&name=pdf).
>    * We want to emphasize that the contrastive function inherently accommodates variability by focusing on relative differences rather than the absolute correctness of positive and negative pairs.  Therefore, partially incorrect positive or negative samples do not affect performance. This has been verified in our supplemental sensitivity analysis.
 >   * What's more, a range of related works, like SimCLR and MoCo, demonstrate that contrastive learning can still perform well even when there is noise in sampling strategies.

Meta-review: The submission proposed a contrastive method for Gloss-free sign language translation. It focuses on the representation density problem in sign language translation. 4/5 reviewers are positive to this submission while reviewer  rKGZ has some concerns, especially on limited novelty. After read the submission, comments, and rebuttal, the AC recognize the submission still has some merits and agree with most of the reviewers, and suggest accepting this submission.

Related work is suggested move from session 5 to session 2 or merged into session 1.
Decision: Accept (poster)