SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognition
Authors: Ariel Basso Madjoukeng
Jérôme Fink
Pierre Poitier
Edith Belise Kenmogne
Benoit Frenay
Venue: Accepted by TMLR ()

Abstract: Sign language recognition (SLR) is a machine learning task aiming to identify signs in videos. Due to the scarcity of annotated data, unsupervised methods like contrastive learning have become promising in this field. They learn meaningful representations by pulling positive pairs (two augmented versions of the same instance) closer and pushing negative pairs (different from the positive pairs) apart. In SLR, only certain parts of the sign videos provide information that is truly useful for their recognition. Applying contrastive methods to SLR raises two issues: (i) contrastive learning methods treat all parts of a video in the same way, without taking into account the relevance of certain parts over others; (ii) shared movements between different signs make negative pairs highly similar, complicating sign discrimination. These issues lead to learning  non-discriminative features for sign recognition and poor results in downstream tasks. In response, this paper proposes a self-supervised learning framework designed to learn meaningful representations for SLR. This framework consists of two key components designed to work together: (i) a new self-supervised approach with free-negative pairs; (ii) a new data augmentation technique. This approach shows a considerable gain in accuracy compared to several contrastive and self-supervised methods, across linear evaluation, semi-supervised learning, and transferability between sign languages.

Published in Transactions on Machine Learning Research (02/2026)
SSL-SLR: Self-Supervised Representation Learning for Sign Language Recognition
Ariel Basso Madjoukeng University of Namur
ariellussodeeu@@maiilcc com
Jérome Fink University of Namur
jerome..innkkummmaarr be
Pierre Poitier University of Namur
dierr.poolierrBunnannr be
Edith Bélise Kenmogne University of Dschong
elkenmmogee@gmaillccom
Benoit Frenay Universily of Namur
denoii..renayBBuammr
Reviewed OpenReview: https: // openreview. net/ forum? id= buT2kTXijy
Abstract
Sign language recognition (SLR) is machine learning task aiming to identify signs in videos. Due to the scarcity of annotated data, unsupervised methods like contrastive learn- ing have become promising in this field. They learn meaningful representations by pulling positive pairs (two augmented versions the same instance) closer and pushing negative pairs (different. from the positive pairs) apart. SLR, only certain parts the sign videos provide information that is truly useful for their recognition. Applying contrastive methods to SLR raises two issues: (i) contrastive learning methods treat all parts of video in the same way, without taking into account the relevance of certain parts over others; (ii) shared movements between different signs make negative pairs highly similar, complicating sign dis- crimination. These issues lead to learning non-discimiinativ features for sign recognition and poor results downstream tasks. In response, this paper proposes a sclf-supervised learning framework designed to learn meaningful representations for SLR. This framework consists of two key components designed to work together: (i) a new self-supervieed ap- proach with free-negative pairs; (ii) a new data augmentation technique. This approach shows considerable gain in accuracy compared to several contrastive and supervuu hods, across linear evaluation, semi-supervised learning, and transferability between sign languages.
1 Introduction
Sign language recognition is challenging task that involves identifying signs in videos. Depending on the data format, thore exist vidco-based SLR and imago-bascd SLR. Sign language recognition with machine learning rapidly expanding, but it faces several challenges, particularly in the acquisition annotated datasets. Indeed, collecting and annotating sign language data. requires linguistic expertise that is difficult to find, it costly and time-consuming (De Coster al. 2024). As an example, annotating hour of sign videos takes about 100 hours (Renz et al. 2021 For this reason, sign language suffers from scarcity of annotated data.

---

Published in Transactions on Machine Learning Research (02/2026)
Recent studies have turned to unsupervised methods such as contrastive representation learning, It offers a promising solution enabling deep learning models to train without supervision by leveraging data aug- mentations. Once pre-trained on unannotated sign languago data, the model can then be fine-tuned on less available data and achieve good results. Most contrastive approaches generale positive and negative pairs (Chen et al., 2020a, He et al.) (020)) then learn similarities and dissimilarities from them. A positive pair is usually created by applying an augmentation to an instance, while negative pairs consist of different instances from the dataset. As example, for image, positive pair consists of the image with new color distortion and the rotated image.
Li et al. 2020b) show that contrastive approaches can generate negative pairs with similar semantic char- acteristics, resulting in a poorly discriminated latent space. This issue is amplified in sign language where distinct signs may share similar movements (Vogler & Metaxas 2004] and there are several signs that share similar hand shapes and movements but convey different meanings (Zuo al.) 2023). Approaches without negative (e.g., BYOL (Grill et al.] 20200) pairs often require additional components (encoder, or other), increasing the complexity of the model. Additionally, sign language videos include repositioning (hand ad- justments following sign) and coarticulation (transitory motions between signs) (Poitier et al. 2024). These movements are generally present in sign language data, but they are not useful for sigu discrimination. Thus, not all parts of the signs are relevant for their identification. Contrastive learning approaches are designed to learn invariant representations by leveraging data augmentation. Thus, using contrastive approaches, all representations even those irrclovant for identification are learned. This results in low accuracy during the linear evaluation, poorly discriminated embedding space and limited transferability of learned representa-
tions. Methods that focus on local aspects (Xie et al., 2021 He et al. 2020 Bardes et al. 2022) of images are not suitable for this task, as they only concentrate on individual clements, rather than entire segments of the video recording.
In response to the above challenges, this paper proposes a self-supervised framework consisting of two key components: a new self-supervised approach and new augmentation method. The self-supervised method is designed to be invariant to augmentations by producing similar representations for a sign and its augmented variants. While the proposed data augmentation generates positive pairs by degrading the non-relevant parts of the signs, enabling the self-supervised method to become invariant to those non- relevant parts. The proposed method removes the need for negative pairs, additional encoders, or clustering mechanisms and surpasses several well-known contrastive and self-supervised approaches on several SLR tasks. The results show first an improvement in the accuracy and quality of representations learned by the most popular contrastive architectures such as SimCLR, MoCo, SimSiam, and BYOL, across several sign language datasets. Second, the achievement of state-of-thhe-ar results on several datasets.
The rest of this paper is organized as follows: Section 2 presents contrastive learning and several popular methods generally used in SLR. Section 3 presents previous works in SLR; Section f shoows that not all parts of a sign language video are equally relevant for the recognition; Section
5 presents the proposed approach; presents the ablation of the approach while Section g concludes and presents several further perspectives.
Section 6 presents experiments and the discussion; Section
7
2 Contrastive Representation Learning
Contrastive learning enables deep learning models to learn relevant representations without annotations. Many contrastive learning methods exist, each with its own particularities; SimCLR (Chen al. 2020a) is one of the most popular approaches. At each step, it generates positive pairs through data augmentations, then maximizes the similarity between positive pairs while minimizing the similarity between positive pairs and the negative others. However, in this approach, positive and negative pairs are restricted to the instances within the batch, which limits the learned diversity. MoCo v2 (Chon et al. 2020b) addresscs this limitation by introducing a quene to store the batch of the previous iteration and use two encoders. In SimCLR and MoCo v2, many negative pairs that often share similar characteristics are generated. This complicates the discrimination and often yields a poorly discriminated embedding space. In response, Li et al. (2020b) introduced the prototypical contrastive learning (PCL). It follows the MoCo approach, but at each step, it uses the k-means clustering (MacQueen 1967) on instances from the momentum encoder. The issue with
2

---

Published in Transactions on Machine Learning Research (02/2026)
the PCL is the use of a clustering function that requires a number of clusters. Modern contrastive methods such as BYOL or SimSiam avoid this issue by using only positive pairs. BYOL uses two encoders (the online and the target), at cach step it generates positive pairs which are then passed through the encoders. The ouline encoder is trained to predict the larget encoder. SimSiar for its part employs a straight Siamese architecture (Bromley et al., [993) with one encoder to learn representations from augmentations, These approaches are generally designed and validated for image classification and segmentation tasks. They have been used in the SLR and have yielded interesting results (Kothadiya et al. 2023 Madjoukeng et al.
2025a b
The next section presents previous works in SLR
3
Sign Language Recognition
Similar to spoken languages, there exist many sign languages that are different from one region to an- other. Several SLR approaches have been developed for varions sign languages. For the Argentinian sign language, Masood et al. (2018) proposed a two-level architecture (CNN and RNN) for the recognition of 46 different signs and achieved 95.2% accuracy. This accuracy can be attributed to the limited number of classes. Marais et al.] (0022 showed that an InceptionV3-GRU model (Szegedy ct al. scratch achieved 74.22% accuracy
2016) trained from vocabulary of 64 different signs (Ronchetti et al.
2023 In the same
sign language, Alyami et al. (2024) benchmarked several architectures and found that the best result (98.25% accuracy) was achieved with a transformer model.
introduced a comprehensive dataset containing 4,567 distinct signs, thereby providing a valuable resource for the development and evaluation of sign language recognition systems associated with this region. Leveraging this dataset, they trained Vision Transformer (Dosovitskiy et al., 2020 architecture from scratch. Despite the complexity and variability inherent in this dataset, their model achieved 54.4% accuracy on a subset of 700 signs.
For the French Belgian sign language (LSFB), Fink et al. 2021 2023
For the American sign language, Li et al. 2020a) proposed a pose-based Temporal Graph Convolutional Neural Network (Pose- TGCN) for the recognition of the signs of this language. They achieved 55.43 % accu- racy on a subset of 100 signs of Word Level American sign language (WLASL). Tunga et al. 2021 proposed GCN-BERT, a framework for poso-based SLR that combines a Graph Convolutional Network (GCN) and BERT and achieved 60.15% accuracy. Hu et. al.] (2021) presented SignBERT, a large-scale model pretrained a vast amount of data using a pretext task of masked visual token reconstruction. After pre-training and fine-tuning their model, they achieved an accuracy of 76.36% on the same subset (100 signs). Zhao et al. 2023) proposed BEST (BERT Pre-training for Sign Language Recognition with Coupling Tokenization), a framework pre-trained on a large amount of data with a pretext task of reconstructing the masked unit (left hand, right hand, etc.). With their approach, they obtained 77.91% accuracy on the same subset, Hu et al. (2023) proposed SignBERT+, an enhancement of SignBERT which incorporates model-aware hand prior. This improved model achieved 79.84% accuracy. More recently, [Jiang et al. (2024) proposed Sign- CLIP, a large model trained 500,000 videos and text descriptions from 44 different sign language corpora. The architecture involves two encoders: video encoder and text encoder. During the training, both encoders are optimized jointly using contrastive loss to align video and text representations in a shared latent space. With this approach, the model achieved 46% accuracy on the ASL Citizen dataset. Wong et al. (2025) proposed SignRop, a self-supervised lcarning framework for sign language recognition. Protrained on the YouTube-SL. dalaset (Tanzer & Zhang, 2024 (a large corpus containing more than 3000 hours of videos across 25 sign languages), they achieved 49.95% accuracy on this dataset during fine-tuning, Other works (Gueuwon et al. 2025a b) have also leveraged the Tubo-SS- dataset to pretrain models for signs recognition in this language.
For the Greek sign language, Adalogiou et al. 2021 conducted an extensive benchmark of several deep learning architcctures to determine the most effective model. In their experiments, the best performance was achieved using a hybrid architecture combining an Inception-3D with a Bidirectional Long Short-Ten Memory (BLSTM) module. This design allows the model to benefit both from spatiotemporal feature extrac- tion (via Inception-3D) and from temporal sequence modeling (via BLSTM). With their architecture, they achieved accuracy of 89.74% on a vocabulary of 310 isolated signs. Recently, on this dataset, Papaddim- itriou et al. 2024) introduced multimodal framework that leverages both appearance based information
3

---

Published in Transactions on Machine Learning Research (02/2026)
and skeleton- based information. For the appearance features, they used a ResNet2+1D network, while for skeleton sequences, they employed a spatio-temporal graph convolutional network. When applied to this datasot, the model achieved 96.25% accuracy.
Worldwide, there exist over 150 sign languages [iilee al. 2024), but only a few have enough annotated
data train deep learning models effectively. Existing models (qHH al. 2021 2023 Zhao al. 2023 Jiang
al. 2024 rely huge amounts of data from multiple sign languages to pre- pre-train models, which are then line-tuned with few annolated data. However, this presents some challenges, indeed pre-training requires large datasets (e.g., 500,000 videos for SignCLIP) from diverse sources (SignBERT, SignBERT+, etc.), which are sometimes not accessible and require considcrable computational resources; additionally, they often lack sign-specific features (Wong et al., 2025 For an effective solution to data scarcity in SLR, instead of pre- training models on vast amounts data from diverse sources, recent studies are focusing on pre-training the models on unannotated data from sign language or leverage self-supervised learning approaches (Kothadiya et al. 2023 Madjonkeng et al., 2025a b). In this context, Madjoukeng al. (2025a) leveraged contrastive learning techniques based on data augmentation (SimCLR, MoCo, SimSiam, etc.) to learn representations from unannotated sign language data. They pre-trained and fine-tuned several contrastive models on data from & specific sign language without requiring vast amounts of data from diverse sources. Conducted on image-basee based SLR, this work has yielded remarkable results. By enabling the training of models in an unsupervised manner on sign language, contrastive approaches provide an innovative solution to address the challenges of scarce annotated data in sign language. Madjoukeng et al. (2025b) show that by using contrastive learning approaches in image-based SLR, it sometimes occurs that the learned representations are not aligned with the sign in the image. They show that by enabling contrastive models to be more focused on the signs in the images, the accuracy and faithfulness of the contrastive models increase. As in image-based SLR, where the sign occupies only portion of the image, we observe that even for video-based SLR, not all frames in sequence are equally relevant for sign identification. The next section demonstrates that for sign identification, only certain frames are truly useful.
4 Not All Parts Of A Sign Are Relevant For Its Recognition
(a) Step
(b) Step
(c) Step 3
(d) Step
(e) Step 5
(f) Step 6
(g) Step 7
(h) Step 8
(i) Step 1
(i) Step 2
(k) Step 3
(1) Step 4
(m) Step 5
(n) Step 6
(o) Step 7
(p) Step 8
(q) Step
Step 2
(s) Step 3
(L) Step 4
(u) Step 5
(v) Step 6
(w) Step
Step 8
Figure 1: Examples of sign steps across three different datasets: LSA Ronchetti al. 2023 (first row)
ASL (Desai et al. 2023) (second row), and L.FB Fink et al. 2021 (third row). For privacy reasons, signers' faces have been blurred.

---

Published in Transactions on Machine Learning Research (02/2026)
Augmentation
Representation
Projection
F
g1
21
T
Ca
C1
T
h
X
y2
22
22
Predictor
Original Sample
Figure 2: SL-FPN architecture: A sign and its augmented variants are passed through an encoder. SL- FPN optimizes three objectives: (1) minimizing the distance between representations of the two augmented variants; (2) minimizing the distance between representations of one augmented variant and the original instance; and (3) minimizing the distance between representations of the original sample and the other augmented variant using a predictor with a stop-gradient operator.
Datasets for sign languages generally created in three ways. In cases, signers filmed in studio performing isolated signs. In other cases, isolated signs segmented from sign language videos and then annotated (c.g., LSFB). In other instances, the signers self-record the signs in a video with their equipment (e.g., ASL Citizen). In all scenarios, several issues occur: for signs recorded in a studio, repositioning movements appear. For signs resulting from segmentation due to segmentation errors, the beginning or end of a given sign often ends up another sign. For the signs recorded by the signers themselves, at the beginning of the signs, the signers often turn on the camera, then turn it off after the signs. This leads to movements that not truly useful for model for efficient and realistic discrimination. These movements neither the essence, nor the frames on which model should base for the signs discrimination.
To illustrate these movements, Figure I presents one instance of signs: Opaque (first), Sweet (second), Also (third), randomly selected from the LSA, LSFB, and ASL Citizen datasets. For each of these signs steps, several clements emerge: (i) at the start the signs, the frame sequences are continuous and follow consistent direction; (ii) at the end of each sign, repositioning movements are present (AST, Citizen) or sometimes preparing for the start of another sign (LSFB). This illustrates that, identify sign neither the initial position nor repositioning to the final position should matter, but rather what happens in between. To learn useful representations, models should focus more on the relevant parts of signs rather than learning representations that are irrelevant for sign identification. The next section presents efficient self-supervised framework that leverages this argument to produce better representations for SLR.
5 Self-Supervisel Framework For Sign Language Recognition (SSL-SLR)
This section presents the proposed self-supervised framework called SSL-SLR. This framework consists of self-supervised approach and novel data. augmentation. The first part of this section presents the self-supervised approach and the second the proposed data augmentation.
5.1 A novel self-supervised learning approach with free negative pairs: SL-FPN
The proposed SL-FPN aims to eliminate the need for negative pairs, additional clustering functions, or sup- plementary encoders that increase model complexity, while achieving higher accuracy than other negative- free methods. Generally, self-supervised methods only use either positive pairs positive and negative pairs without leveraging the original instance. However, since both positive pairs and the original instance repre- sent the same concept (i.e., the same sign in this case), it can be beneficial to leverage them simultaneously

---

Published in Transactions on Machine Learning Research (02/2026)
during the learning process. This approach leverages both positive pairs and the original instance to enhance training. Figure 2 presents the proposed architecture. For an input x; two augmented versions are generated by randomly applying two different augmentations T and T from an augmentation set. Then, the obtained versions 1 ø2 and the original instance x, are each passed through an encoder / and a projection head h. The representations 21 = h=((=))), 22 = h(f(z2)), and h(f())) are thus obtained. The goal of SL-FPN is to generate highly similar representations for instance and its augmented versions. To achieve this, it minimizes the difference between the representations of an input and its augmented counterparts, typically using the Mean Squared Error (MSE) as the loss function. The smaller this difference, the better ST-FPN can produce closely aligned representations in the embedding space for an instance and its augmented variants. As shown in Figure 2 the SL-FPN loss is threefold and consists of: (i) an MSE between the representation of iositive pairs (L1); ) (ii) an MSE between the representation of one of the positive pairs and the original instance (C2); (iii) an MSE between the output of the predictor and the representation of an augmented version. The stop-gradient operator defined as sg(21) = 21 is used to consider one representation as constant and prevent gradient propagation along this representation. Like SimSiam or BYOL, it is used as an asym- metrical component, breaking the symmetry between the branches. Hence, in the ST,FPN architecture, the asymmetry is ensured by C3; without it, SL-FPN does not benefit from any asymmetric component. For n-sized embeddings, the final loss is given by:
| ||2 2213, L2 71
- 22113 n
L3 = - -g(2))) C= C + +2+ L3.
L1
(1)
The training objective aims to minimize [(0), with 0 the model parameters. For a set of N samples and the MSE loss, L(0) is defined as:
1 N Σ +(C)))((((())++())))))))))))(()) ((P((((())
(())
(2)
By minimizing Equation 2 SL-FPN generates the closest representations for sign and its augmented versions in the embedding space. During our experiments, we assigned weight to each term of the objective function (equation 2), but this did not significantly affect the results. Therefore, for simplicity, we did not include any penalty term. The main difference between SL-FPN and other self-uupervised methods such as BYOL (which uses two branches and two encoders) or SimSiam (which uses two branches and a predictor), lies in the fact that SL-FPN uses three branches, a single encoder and a predictor. The novelty here does not simply consist of using three branches, but also lies in the way that they are combined during training. Section shows the effect of permuting the order of the three inputs on accuracy.
In reprosentation learning without negative pairs, representation collapse often occurs. It occurs when the model produces the same representations for all input instances. To avoid this issue, several methods exist. Wu et al. (2024) show that layer normalization and skip connections in a transformer encoder can mitigate collapse. Chen et al. (2020a), and Grill et al. (2020) highlight the importance of stop-gradient mechanisms coupled with a predictor to prevent collapse in self-supervised architectures like SimSiam and BYOL. Hence, SL.FPNN incorporates these components in its architecture to avoid the collapse solution. The ablation study in Doetion/sshove the impact of each of these components in avoiding collapse in the SL-FPN. This approach can be applied to wide range of computer vision tasks and can yield competitive results. In particular, it can be useful in situations where there is a risk of semantic inconsistency between positive pairs. This is well-known issue in contrastive learning, where positive pairs may no longer share the same semantic information iGuo & Shi 2023). In such cases, by leveraging the original instance, SL-FPN can prove to be very effective unlike other self-supervised approaches.
With the proposed ST.FPNN architecture alone, the challenge of ensuring the relevance of the learned repre- sentations remains. Indeed, there is no guarantee that it will focus on features truly discriminative for sign identification. To address this issue, the next section introduces new data augmentation strategy designed to help SL-FPN focus on the most relevant parts of the signs.

---

Published in Transactions on Machine Learning Research (02/2026)
5.2 A new augmentation method
In contrastive learning, the augmented variants are generally obtained by applying augmentations to all parts of sequence. By learning to produce similar representations between sign and its augmented variant. contrastive learning methods aim to build an invariant representation from the augmentations, Madjoukeng ett al. (025b) show that, for image-based SLR, applying augmentations while preserving the region of interest (sign in the image) helps contrastive models to better focus on it and improve the performance in downstream tasks. Hence, in the same way, the proposed augmentation method aims to generate positive pairs by preserving the parts of the signs that are more discriminant. for their identification. For image- based SLR, the region of interest can be identified using segmentation methods such as Mask R-CNN (He eet al. 2017) others. However, video-based SLR faces a major challenge: there is no established method to identify the frames that are relevant for sign recognition. To fill this gap, the first step of the proposed augmentation is to determine the relevant frames for the signs identification.
Determining the relevant frames for signs identification
In Section 4 it was shown that due to certain movements (repositioning, coarticulation, etc.), the information truly useful for discrininating signs is not found throughoul the entire sequence. The fundamental question is therefore to determine where these discriminative parts begin and end within the sequence. The main objective of this step is to determine the boundary importance. It is the boundary that marks when frames become and stop being relevant for sign identification. This step involves identifying (i) the point in the sequence from which frames contain sufficient discriminative information (k:); and (ii) the point until which these frames remain useful before losing their discriminative ability (k2). To date, the best of our knowl- edge, no standard method exists in the literature locate these discriminative regions within a sequence. In this research, we propose intuitive approach based on contrastive algorithm with transformer backbone, combined with temporal augmentation technique, denoted I
A sign is a sequence of frames starting from an initial frame hi to a final frame ke. The objective is to determine the optimal frames Kg and ke, which represent the boundary importance. Transformer-base architectures with positional encoding are particularly sensitive the positioning of elements within the sequence. Contrastive methods generate positive pairs through augmentations and learn representations that are invariant to these augmentations. Thus, contrastive approach equipped with transformer encoder offers two key properties: (i) invariance to augmentations, inherited from the contrastive loss; (ii) sensitivity to the order of elements in the sequence, induced by the transformer positional encoding. Moreover, it is well established that the performance of contrastive models in downstream tasks (SLR in this case) strongly depends on the quality of the learned representations (Tian et al. 2020 Roschewitz et al. 2024). The detection of relevant frames therefore relies on the combination of these two properties and on the exploitation of this information.
The idea is to use a contrastive algorithm to progressively degrade sign, first from the first to the last frame (to determine the frame kg), then from the last to the first frame (to determine kz). The impact of each degradation is then evaluated on linear evaluation. If the model becomes invariant to frames necessary for identifying a sign, its performance during linear evaluation will be low. Conversely, if it learns to be invariant to irrelevant framos, it will focus more on informative parts, leading to better downstream performance. For the degradation, several augmentations can be applied. Due to its capacity of modifying the temporal order of sequences while preserving the local distribution of the frames, temporal permutation is chosen, Additionally, as transformer encoder with positional encoding is sensitive to the positioning of cach element in the sequence, temporal permutation will induce diversity between the positive pairs from the point of view of the transformer encoder. Contrary to augmentations such as rotations, Gaussian blur, flips that are applied at the local frame level.
Algorithm presents an overview of this method. It takes as input a dataset D, a contrastive algorithm C, two augmentations Ti and no (typically two random permutations). Its goal to determine kg and ky (boundary importance) whose permutation maximizes the classification accuracy of C. For kg, the algorithm evaluates the accuracy after permuting the first k, frames, increasing k2 long accuracy increases. For ke it follows symmetric procedure on the last kg frames. The SegmentEval function applies ga and m2 to

---

Published in Transactions on Machine Learning Research (02/2026)
the selected segment, inserts the modified segments back into the sequence, and measures accuracy via C. The process stops for each parameter when further changes no longer improve accuracy and returns (kş k as the optimal lengths before stagnation. For datascts containing long sequences of signs, instcad of iterating over all frames, the evalualion can proceed in frame sleps (e.g., every p frame) which reduces computational cost.
After determining the boundary importance, the proposed augmentation consists of generating positive pairs by applying augmentation on the first kş and on the last kg to help the contrastive model to be more focused on the relevant part of the sign. The next part of this section presents two case studies as proof of concept for determining ky and : one on the GSL and the other on the LSFB datasct.
Proof of concept: a study on the GSL and LSFB datasets
11
N
20
N
(a)
(b)
(c)
(d)
Figure 3: Variation of the accuracy during linear evaluation protocol the LSFB 3a dataset based on the number of k shuffled frames the beginning (from the left to the right at the end (from the right to the left
3d
and GSL
13b
3c
3a
3b
and
33
3d
140
545
120
100
N/8
N/4
N/3
N/2
2N/3
N
N/8
N/4
N/3
N/2
2N/3
(a)
(b)
Figure 4: Histogram of class proportions function of permutations starting from the first frames (a) and from the last frames (b).
To give overview how Algorithm | performs and determine the default (k2, paramcters, this part of the paper provides empirical study of the application of this algorithm on the GSL (mediun) and the LSFB (large) datasets. The model parameters are described in section 6.2 and the contrastive method used was SL-FPN trained five consecutive times for each permutation. For computational cost reasons, permutations have been performed on each frames, with p in {1,
N
N 2N 3 NN.
8 1 3
Figure 3 shows the accuracy (linear evaluation) variations when applying permutations the first k, and last Ke frames of N-frame sequence. We observe that when incrementally permuting k, initial frames, the accuracy evolves progressively until kg reaches approximately N/3. Beyond this threshold (about the first third of the frames), the accuracy begins to decrease continuously, collapsing completely when k2 N (all

---

Published in Transactions on Machine Learning Research (02/2026)
30
20-
25
1.0
0.5
1/4
w/3
N22
28/3
N/8
N/4
N/3
N/2
2N/3
2
(a)
(b)
Figure 5: Impact of the global choice (kg and kz) on the failure cases.
frames permuted), which scems natural, as all temporal information is then lost. Similarly, when randomly permuting the kg last frames, the accuracy remains stable until lig R N/4 (the last quarter of frames). Beyond this point, accuracy decreases and eventually collapses when ke = N. These results suggest. that, the model is robust to temporal order loss in both the first third and last quarter of the sequence. Consequently, frame order appears most critical the central portion of the sequence (approximately between positions N/3 and - N/4). These frames are thus identified as the most. useful in the signs identification according to the dataset. After identifying the key frames, the generation of positive pairs is based on applying permutations different stages. Specifically, permutations are performed on the frames located between the first and N/3, as well on those between N N/4 and the last one.
Choosing parameters that maximize overall accuracy seems like a reasonable approach, but it is important to note that in a sign language dataset, not all signs usually follow the same temporal distribution (i.e., some may become relevant at frame t, while others at t + 1). There exists a global temporal trend in the dataset, as well as trends that may be specific to cortain classes or signs the dataset. Thercfore, it important to measure the impact of choosing kü and kg for the entire dataset. Hence, two further experiments were carried out. The first experiment consists of visualizing, the proportion of observations that achieve their best performance using the parameters k3 and kt, and those for which the best performances are not achieved with these parameters. The second consists of analyzing the effect of the parameters K and K on the signs that do not follow the global temporal distribution of the dataset. The reason behind this experiment is that this study considers the parameters h2 and he as those that maximize the overall accuracy. Hence, it is essential to assess the impact of this choice on instances whose optimal parameters do not correspond to k and kn
For the first experiment, Figure 4 shows two histograms, each representing the proportion of observations function of the values of their optimal kt (first histogram) and ke (second histogram) on the LSFB dataset. From this figure, we note that generally most observations reach their maximun at k, = k and ke = kg. However, we also observe that a subset of instances reaches its maximum at lower values, notably with k = ke = or with kg + kn and ke + k:- This indicates that a small but non-negligible proportion of signs relies primarily on the initial or the last frames for correct. identificat ion. Since the augmentation approach uses the optimal paraneters for the global trend in the dataset, this proportion of signs represents the failure cases for the proposed augmentation. For these failure cases, it is essential to assess the performance loss duc to the use of these global values instead of the sigu-specific imaimal ones. This means that, for any signs 8; whose optimal parameters are (k- o kg, k:f * kū), the impact of using (kt, in place of "eft,ke" should be measured. Hence, for each observation with the signs-specific optimal parameters (k",,,)) kot) and that obtained with the global parameters (k",k)) kT).
we measured the difference between the accuracy obtained
Figure reports the obtained results on the LSFB dataset. From this figure, we note that when Ky and kg are not optimal, it incurs a modest loss (I to 4%) depending on the observations. This loss reaches its maximun value for kg and ke 1, which corresponds to signs starting from the first frame and extending to the last frame. These cases represent the main failure cases for the proposed augmentation. We also observe

---

Published in Transactions on Machine Learning Research (02/2026)
that as the values of ke and ke approach k2 and ke the loss becomes less significant. This indicates that the approach not very sensitive to small variations around the optimal parameters. The fact that most observations reach their maximum at k2 = kn and hę = kg shows that even though some failure cases may occur, choosing kn and kn as optimal global parameters is justified. To account for these failure cases, new analytical augmentation strategies based on the specific behavior of the observations need to be developed.
Beyond isolated SLR, this augmentation can be apply in other area like continuous SLR. Indeed, in continuous SLR, non-relevant movements (such coarticulation and repositioning) are also present. In this field, the goal is to minimize the Word Error Rate (WER) (Chen et al. 2022] which is the proportion of signs that are inserted, substituted, or deleted relative to the reference sequence. These coarticulation movements can sometimes be mistakenly interpreted as signs, leading to an increase in WER. By properly calibrating this augmentation method, it. can therefore prove to be particularly useful this domain, as helps to better handle the natural variations occurring in continuous signing.
Note that, in the proposed augmentation, the parameters kn and kt can be modified according to each specific case and can be done a preprocessing task. For datasets containing long sign sequences, this procedure can lead to considerable computational cost. This cost can be estimated as 2 x k x T, where k is the number of (ests, T is the execution time per lest, and the factor 2 accounts for the lest being performed in both directions. To alleviate this issue, one solution is to apply the procedure to subsets of the sequence (subsets consisting of a certain number of frames, as we did this study), which significantly reduces the number of tests. Another approach to generalize based on prior information about the data. In our case, since repositioning, coarticulations and non-relevant movements are present across different datasets; the fact that the signers are natives, the parameters K2 NN/3 and K = N N-NNN N/4 will be used throughout all our experiments. The next section evaluates the SL-FPN, the proposed data augmentation and the SSL-SLR framework (SL-FFPN architecture coupled with the proposed augmentation).
Algorithm Search for boundary importance ki,
Require: D (dataset), C (algorithm), N (sequence length), position E ('first', 'last"
Ensure: ky kg: the boundary importance
1: function F'NDOPTIMAL((D,CC C, N, position)
k 1+1
Oprev SegmentEEaa(k; D, C, position) +-2
h
Ocurr f- SeamentEval(k, D,C, position)
6: while he < N and aeur > aprev do
7:
aprev <- aeurr
8: k k
aaur +- SegmentEval(k; D, C, position)
10: end while
11: return
12: end function
13: k + FINDOPTIMALK C, ., 'first")
14: k < FINDOPTIMALK C, N, 'last")
15: return k k
10

---

Published in Transactions on Machine Learning Research (02/2026)
function SEGMENTEVAL D, P,C,position for S = (S1..... .N) E 'D do
if position = 'first' then
segi - T seg2 52(f1..... fi)
b (Ji,..., are the frames.
8(1) (seg1. 1kt1,---:/1
8(2) + (seg2, fut1......
else
segı - Ti (1N k+1. SN)
-k+, {2) seg2 t x2(fN- BeffN-K+1,--
D. TT, T2 are two independent random permutations.
s(1) -- (fi....: -s segi)
((2) + (fi...., -k seg2)
end if
Apply C using (8(1, ((2) the positive pair
end for
return accuracy in linear evaluation end function
6 Experiments
This section first introduces the datasets used, then provides quantitative and qualitative evaluation of different methods. Finally, it benchmarks the propased approach against. state-of-the-art methods on dif- ferent datasets. The quantitative analysis assesses representation quality through performance on a linear evaluation protocol. This evaluation consists of protraining contrastive approach, froczing the backbone and train a simple multilayer perceptron on top of it. The qualitative analysis offers a visualization of the latent space and measure of intra-class inertia for each method. For fair comparison against the state-of- the-art methods, for datasets where existing methods did not benefit from pretraining (LSFB, LSA, GSL), the SSL-SLR approach was trained from scratch and then fine-tuned. For datasets (WLASL, (itizenn where state-of-the-art methods were first pretrained, SSL-SLR was pretrained and fine-tuned under the same conditions as the other methods (Wong et al. 2025 and its performances were then evaluated. The obtained results were subsequently compared with those reported in the lilerature for each dataset. Throughou this entire section, SL-FPN refers to the architecture described in Figure 2 without the proposed augmenta- tion, while SSL-SLR corresponds the SL-FPN combined with the proposed augmentation (described in Section 5.2
6.1 Datasets and splits
This study uses five datasets with different sizes. These datasets either made of video clips of signers executing signs independently sign language sentences with word annotations enabling the extraction of the isolated words. First, we use the French Belgian sign language (LSFB) Fiink et al. 2021 It is one of the largest sign language datasets the world. It contains 4,567 different classes (signs). Second, the Argentinian sign language (LSA) (Ronchetti et al., 2023) that is a partially recorded dataset in studio with uniform background, contains different classes. Third, the Greek sign language (GSL) (Adaloglou let al. 2021)) that captured in studio with a consistent background and regulated lighting, it contains 310 classes. Fourth, the American sign language Citizen (ASL Citizen) datasct (Desai et al. 2023) that is crowdsourced dataset comprising videos of isolated sigus performed at home by both native and non-native signers, it contains 2731 different classes. Fifth, the word level American sign language (WLAST.) [Li et al. 2020a), a variant of ASL containing 2,000 classes.
For this study, the different datasets were split according to their original papers to ensure fair evaluation and comparison. Some were split: 70% for training and 30% for testing (e-g., LSFB), others 80% for training and 20% for testing (e.g, LSA), and 80/10/10 for the GSL, For the state-offthe-art comparison, the 700 most represented classes of the LSFB their classes were used. For the WLASL dataset, used the WLASL-100 and WLASL-300 subsets, as it is
used (Fink et al. 2023): ; for the GSL, ASL Citizen, and LSA, all
11

---

Published in Transactions on Machine Learning Research (02/2026)
commonly considered a benchmark for evaluating downstream tasks. For the linear evaluation protocol, the 500 most represented signs of the LSFB and ASL were used.
6.2 Training parameters
The experiments use the Python language, typically the PyTorch 2.4 + cul18. For the implementation of the different contrastive learning approaches, we used the PyTorch Lightly library which proposes an implementation for the different methods. Given the sequential nature of data, a transformer encoder, primarily the Vision Transformer (VTT) (Dsovviskiy et al. 2020) provided by Fink et al. (2023) was used as the backbone for the different approaches. As a pro-processing task, the vidcos were first transformed into skeleton sequences using MediaPipe (Lugaresi et al. 2019) This choice is based on the lower computational cost and insensitivity to visual factors of skeletons compared to videos (Jiang et al.) 2021 The batch size was set to 512 signs, each with a maximum sequence length of 64 frames usual (Desai et al. 2023). During the unsupervised training, the models were trained on 200 epochs, each with parameters specified in their original papers (Chen et al. 2020a Chen & He 2021 Grill et al. 2020 He et al. 2020). For the SST-STR pretraining, the optimizer was SGD and the learning rate was fixed to 0.001. During the fine-tuning stage, the models were trained for 1000 epochs using SGD optimizer, along with linear warmup scheduler. The learning rate was progressively increased during the first 600 iterations, then gradually decreased over the remaining 400 (Goyal et al. 2017). The temperature parameter used in the different contrastive learning variants was fixed at 0.5. The wransoomerrbasee encoder consisted of 12 blocks, with 8 attention heads and an embedding dimension of 512. We used two layer normalizations at the input embedding stage (Dosovitskiy et al. 2020). The dropout rate was fixed at 0.1. The projection head of all the models was a 2-layer perceptron with a 512-dimensional input and a 128-dimensional output. The predictor P was also a 2-layer perceptron with a ReLU activation function.
6.3 Quantitative and time evaluation
For the quantitative evaluation, several types of evaluations were conducted. First, a linear evaluation was conducted that consists training a classifier on the frozen backbone trained with the unsupervised meth- ods on the training set. For this step, the ASL, LSA, LSFB and GSL were used. To assess the proposed augmentation method, a linear evaluation protocol using the proposed augmentation and the classical aug- mentations (rotation, Gaussian blur and flip) applied to the whole sequence with all the approaches conducted. Second, semi-supervised evaluation with 30% of the annotated data (30% of the training set) was also conducted. For the SLR, where the annotation process is scarce, semi-supervised evaluation aims to demonstrate the value of learning from unlabeled data to improve model performance when only a small amount of annotated data is available. Third, an evaluation of the possible transfer of the learned repre- sentations from a represented sigu language to another not represented by the approaches was conducted. For each model, we performed eight consecutive training runs and reported the average accuracy with 95% confidence.
Table [ presents the results obtained with the proposed SL-FPN compared to other contrastive methods, using first the proposed augmentation method and second standard image augmentation strategies applied to the entire sequence. From this table, we observe that the proposed augmentation helps all the contrastive approaches to better learn meaningful representations. When standard augmentations are applied, perfor- mance generally decreases across all methods. This indicates that the proposed data augmentation st.rategy enables the models to learn more discriminative representations across the different datasets. In some cases, the proposed augmentation yields substantial improvements, with gains of over 6% for the recognition of 500 classes and over 8% for the recognition of 310 classes, highlighting its importance. Therefore, in the following experiments, the proposed augmentation will be consistently applied to compare all the approaches. As the SL-FPN will be coupled with the proposed augmentation, it will be referred to as SSL-SLR.
To evaluate the transferability of learned representations from one sign language to another, we performed unsupervised training on the represented datasots (LSFB and ASI.) and conducted linear evaluation on less represented sign language datasets. Table 2 presents the results obtained using different approaches.
htttp:://oossiigghttt httpp///ccosssiiaaaaaaaaluuluued
12

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Lincar evaluation protocol with the proposed augmentation method (ours) versus classical image augmentations (rolation, Gaussian blur, flip, translation) applied to all the frames.
Datasets
SimCLR
MoCo v2
SimSiam
BYOL
SST-SL.
LSFB (our)
14.16% 0.24
13.68% = 0.48
15.26% 0.67
14.72% + 0.65
23.73% 0.53
LSFB
11.22% + 1.04
10.91% + 1.86
11.84% + 1.71
11.15% 1.12
16.07%
1.41
ASL (our)
14.13% = 0.42
14.69% = 0.39
15.91% = 0.56
16.43%
20.46%
1.21
ASL
11.04% = 0.69
11.09% = 1.12
12.17% = 1.36
12.78% 0.78
15.43% = 1.20
GSL (our)
34.19% = 0.85
36.15% 0.69
32.01% = 0.54
34.09% 0.93
47.76%
0.79
GSL
31.09% = 1.75
30.15% = 1.56
30.89% 30.11% 1.47
1.11
35.13% = 1.34
LSA (our)
34.02% + 1.24
35.69% = 1.06
30.06% + 2.14
37.47% + 1.51
41.74% = 1.08
LSA
29.10% = 2.89
27.17% = 2.76
26.16% = 2.46
32.37% + 2.61
36.71% = 1.97
Table 2: Transferring from one sign language to another with the proposed augmentation method.
Datasets
SimCLR
MoCo v2
SimSiam
BYOL
SSL-SLR
LSFB to LSA
0.89 33.40% + 1.16 32.47% = 31.15% 0.67 35.6%% 46.41%
LSFB to GSL
33.24% = 0.68 35.2%%
0.68 =
0.70 0..0 =
0.56 0033
ASL to LSA
34.74% = 0.473..3%% = 0.36
10.94%%0.000 0.37
ASL to GSL
32.46% + 0.4333.58% + 0.43 + 0.65 1.45 1.21
The SSL-SLR method clearly outperfosms the others. This result suggests that the representations learned by SST-SLR can also be more effectively transferable from one sign language to another compared to the representations learned by approaches like SimCLR, MoCo, BYOL and SimSiam.
The importance of representations learned from unannotated sign language was also evaluated in low-resouree scenarios. To simulate such settings, a fine-tuning using only 30% of the training set is conducted. The corresponding results are presented in Table 3 The proposed SSL-SLR consistently outperforms standard contrastive methods. This highlights the robustness and transferability of the representations learned through our method, even when limited labeled data is available.
The above results demonstrate the effectiveness of the proposed approach for SLR tasks compared to existing contrastive and supervied methods. They also highlight the impact that contrastive learning can have on reducing reliance on annotations in SLR. This advancement represents a significant contribution to the development of models that offer acceptable performance while alloviating the tedious and costly process of manual annotation.
To better position the SL-FPN approach within the self-supervised literature, it is essential to have an idea of the execution time during contrastive learning compared to other approaches such as BYOL and SimSiam, which also do not use negative pairs. For this purpose, the execution time of the self-supervised training for the SL-FPN, SimSiam, and BYOL approaches has been reported Table On the different datasets, we observe that SimSiam achieves the best execution time. This seems natural as it employs a single encoder without requiring negative pairs. In contrast, BYOL demonstrates longer execution times compared to both SL-FPN and SimSiam. This can be explained by the fact that it employs two distinct encoders, and one is updated using an exponential moving average. SL-FPN approach shows a slightly higher execution time than SimSiam, but lower than BYOL. This increase is due to the use of the original instance. The fact that it does not require performing augmentation on the original instance reduces moderate considerably its exccution time. Hence, despite its effectiveness, SL-FPN remains reasonable terms of exccution time.
6.3.1 Qualitative Evaluation
The qualitative evaluation consists in general of visualizing the embedding space to evaluate at which points the instances of the same class are close in the space. So, 2D visualization (MeInnes et al. 2018) was used to show the embedding space. Ideally, the signs with the same class should be closer in the embedding space.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Finc-tuning with 30% of annotations with our augmentation.
Datasets
SimCLR
MoCo v2
SimSiam
BYOL
SSL-SLR
LSFB
42.69% + 3.36
42.23% + 2.14
43.69% + 2.69
41.40% + 2.04
49.93% + 2.98
ASL
47.43% + 0.77
47.49% + 0.54
47.23% 0.63
47.02% 0.51
49.28% + 0.79
GSL
78.82% + 2.96
77.42% + 2.87
77.02%
2.95
78.04% + 2.65
83.86% + 2.01
LSA
87.69% + 1.48
88.04% + 1.69 87.96% + 1.36
88.64% + 1.36
92.76% + 1.63
Table 4: Linear evaluation time in second by the different approaches.
Dataset
SimSiam
BYOL
SL-FPN
LSFB
2.49 x 10a
2.67 x 10a
2.56 10a
GSL
9.77 x 103
1.04 104
1.02 104
LSA
2.67 x 103
2.86 x 10
2.77 103
ASL
2.22 x 104
2.46 x 104
2.43 x 10
Figure 6 illustrraess the 2D visualization of embedding spaces. Given the high number of classes, we randomly select and show the different instances of class across the datasets the embedding space. On the AST, GSL, and LSA datasels, for the randomly chosen class (sign), the embeddings produced by SSL-SLR are closer than those from the SimCLR, MoCo, BYOL, and SimSiam approaches. This demonstrates the effectiveness of the proposed method in generating higher-quality representations compared to the other approaches. However, on the LSFB dataset, the instances are scattered throughout the embedding space across the different approaches. This can be attributed to the complexity of this dataset. (many signs are visually similar, several variants of the same signs, a large number of classes). Additionally to this visualization, an intra-class inertia was computed for the different dataset. Table 5 presents the intra-class inertia obtained by the different approaches several datasets. It is observable from this table that the proposed approach performs better than the others.
6.4
Comparison Against State-of-the-art Methods
To demonstrate the effectiveness of the proposed SSL-SLR in the SLR literature, a benchmark and a com- parison against others state-of- the----ttttertttt methods on different datasets are conducted. As in the literature (Hul et al. 2021 2023 Jiang et al. 2024 Zhao al. 2023), SSL-SLR was first pre-trained and fine-tuned for the comparison. The goal is to compare the proposed approach with previous works done in terms of accuracy during the recognition task. Table i
presents the obtained results, the best results are in bold. From this table, it is observable that, on the LSFB, LSA, ASL Citizen and the GSL dataset, the proposed approach achioved several well-known state-of-tho-art methods. On the WLASL, the proposed SSL-SLR, surpassod several of them in top-5 accuracy.
7 Ablation Study
We examined the impact of the predictor equipped with the stop-gradient operator, the use of the original sample, layer normalization, and the pernutation of the components of the proposed SL-FPN during training, We used the LSFB dataset due to its size and diversity. Feature collapse occurs when the model produces the same representations for different signs. To determine when a model collapses, (Chen & He 2021 showed that, given the embedding size within the interval [0, Agl, the standard deviation of the model drops to zero. Following their protocol, we examine the model output in the same way. To investigate this, we trained the model in four different scttings: (i) without the predictor, the stop-gradient and layer normalization (without (p. 8t & LN)): (ii) without the predictor, the stop-gradient and with layer normalization (without p, $ı & with LN): (iii) without the original sample (without o); and (iv) by permuting the original sample and one of the positive pairs (perm) in the proposed SL-FPN. Figure shows the results obtained. When both the predictor and the layer normalization are used, there is sign of feature collapse. However, when both are removed, the standard deviation drops to zero after a few training steps, clearly indicating collapse.
14

---

Published in Transactions on Machine Learning Research (02/2026)
(a) SimCLR (GSL)
(b) MoCo v2 (GSL)
(c) BYOL (GSL)
(d) SimSiam (GSL)
(e) SSL-SLR (GSL)
(f) SimCLR (LSA)
(g) MoCo v2 (LSA)
(b) BYOL (LSA)
(1) SimSiam (LSA)
(i) SSL-SLR (LSA)
(k) SimCLR (AST.)
(I) MoCo v2 (AST.)
(m) BYOL (AST.)
(n) SimSiam (ASL.)
(o) SSL-SLR (ASL)
(p) SLM (LSFB) (q) MoCo v2 (LSFB) (r) BYOL (LSFB)
(s) SimSiam (LSFB) (t) SSL-SLR (LSFB)
Figure 6: Visualization of the embeddings using SimCLR, MoCo, BYOL, SimSiam, and the proposed method four datasets: GSL, LSA, ASL and LSFB.
When only layer normalization is used, the standard deviation gradually decreases, which suggests that it slows down the collapse but does not fully prevent. We also note that during the training, when the original sample not used and by permuting the order of the inputs in the proposed SL-FPN, the accuracy decreases significantly. This suggests that using the original input, when it is well-calibrated, can significantly improve the quality of the representations.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Intra-class inertia of diflerent approaches
Datasets
SimCLR
MoCo v2
SimSiam
BYOL
SSL-SLR (Ours)
LSFB
1.86 10*
1.84 x 10*
1.36 x 10*
1.84 x 10*
0.57 x 10
ASL
3.11 x 10"
1.37 x 10
2.81 x 10
1.54 x 10"
0.41 x 10
GSL
1.25 x 10a
0.91 x 10a
0.69 x 10a
0.67 x 10a
0.17 x 10a
LSA
3.45 x 10
2.72 x 102
2.27 x 102
1.22 x 102
0.25 x 102
Table 6: Top-1 and Top-5 accuracy of thate-of-the-art methods and the proposed SSL-SLR on various sign language datasets.
Dataset
Method
Top-1 (%)
Top-5 (%)
SignCLIP
Jiang et al. 2024
46.00
77.00
AST Citizen
SignRep (avg) (Wong al.
2025
37.47
68.77
SignRep (weighted)
[Wong et al.
2025
49.95
80.09
SST-SLR (Ours)
47.06
78.96
Fink et al. 20023
54.40
LSFB
SSL-SLR (Ours)
56.81
Masood et al.
2018
95.21
Alyami et al.
(024)
98.25
LSA
SSL-SLR (Ours)
99.07
Adaloglou et al. (2021)
89.74
GSL
Papadimitrion et al.
2024
96.25
SSL-SLR (Ours)
96.73
PSLR Tunga et al.
2021
60.15
83.98
BET Hu et al.
2021
76.36
91.09
WLASL-100
BEST Zhao et al.
2023)
77.91
91.47
SignBERT+ Hu et al.
2023
79.84
92.64
SSL-SLR (Ours)
77.95
93.02
PSLR Tunga et al. 2021
42.18
71.71
SignBERT (Hu et al. 2021
62.72
85.18
WLASL-300
BEST Zhao et al. 2023
67.66
89.22
SiguBERT+ (Hu al. 2023
73.20
90.42
SSL-SLR (Ours)
71.21
90.74
(p. 9g
without 1,..
without (p. s) & with
perm
without
1//
20
putpur 2t1
15
10
5-
0-
200
400
600
800
200
400
600
800
0
250
500
750
1000
1250
spochs
epochs
epachs
Figure 7: Standard deviation and linear evaluation accuracy of the model with ablation of predictor, stop- gradient, original sample and layer normalization on the LSFB dataset.
16

---

Published in Transactions on Machine Learning Research (02/2026)
8 Conclusion
This paper introduces new self-supervised framework for sign language recognition. The framework is based on a novel self-supervised approach and a new augmentation designed to degrade the non-informative part of a sign. The self-supervised approach leverages both positive pairs and the original instance. is designed to be invariant non-informative parts of signs while producing similar representations for a sign and its augmented variants. To validate the effectiveness of the proposed approach, several evaluation protocols, including linear evaluation, tross-lingual representation transfer and semi-supervised learning have been conducted. The results demonstrate the effectiveness of the proposed method compared to existing contrastive and state-of-the-art approaches across these different settings. This advancement represents a significant. contribution to the development of models that offer acceptable performance while alleviating the tedious and costly process manual annotation. Despite its effectiveness, the performance still has room for improvement on largo-scale datascts. Furthermore, the proposed method determine the boundary importance is currently determined empirically. In future work, we plan to develop non-empirical method to determine boundary importance. This will enable to mitigate the failure cases that face the current version. We also plan to extend this work for the cases like continuous sign language.
9 Acknowledgments
The present research benefited from computational resources made available on Lucia, the Tier-1 super- computer of the Walloon Region, infrastructure funded by the Walloon Region under the grant agreement 191910247. The authors thank Valentin Delchevalerie and Simon Lejoly for their insightful comments and feedback. They also thank the TMLR reviewers and the action editor for their constructive remarks on the manuscript.
17

---

Published in Transactions on Machine Learning Research (02/2026)
References
Nikolas Adaloglou, Thcocharis Chatzis, Iliss Papastratis, Andreas Stergioulas, Georgios Th Papadopoulos, Vassia Zacharopoulou, George J Xydopoulos, Klimnis Atzakas, Dimitris Papazachariou, and Petros Daras. A comprehensive study on deep learning- based methods for sign language recognition. 1EEE transactions on maltimedia, 24:1750-1762, 2021.
Sarah Alyami, Hamzah Luqman, and Mohammad Hammoudeh. Isolated arabic sign language recognition using a transformer-based model and landmark keypoints. ACM Transactions on Asian and Low- Resource Language Information Processing, 23(1):1-1 19, 2024.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicregl: Self-supervised learning of local visual features. Aduances in Newral Information Processing Systems, 35:8799-888 8810, 2022.
Yunus Can Bilge, Nazli Ikizler-Cinbis, and Ramazan Gokberk Cinbis. Cross-lingual few-shot sign language recognition. Pattern Recognation, 151:110374, 2024.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using& a" siamese' time delay neural network. Advances in neural information processing systems, 6, 1993.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Gcoffrey Hinton. simple framework for contrastive learning visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750 15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arx preprint arXiv:2003.0 20::2003304297, 2020b.
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems, 35:17043- 17056, 2022.
Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, and Joni Dambre. Machine translation from signed to spoken languages: State of the art and challenges. Universal Access in the Information Sociely, pp. 1305 1331, 2024.
Aashaka Desai, Lauren Berger, Fyodor Minakov, Nessa Milano, Chinmay Singh, Kriston Pumphrey, Richard Tadner, Hal Daumé III, Alex X Lu, Naomi Caselli, et al. Asl citizen: community-sourced dataset for advancing isolated sign language recognition. Advances in Neural Information Processing Systems, 36: 76893 76907, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Jérôme Fink, Benoît Frénay, Laurence Meurant, and Authony Cleve. Lsfb-cont and Isfb-isol: Two new datascts for vision-based sign language recognition. 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1 IEEE, 2021.
Jerome Fink, Pierre Poitier, Maxime André, Loup Meurice, Benoit Frénay, Anthony Cleve, Bruno Dumas, and Laurence Meurant. Sign language-to-text dictionary with lightweight transformer models. In 32nd International Joint Conference on Artificial Intelligence, IJCAI 2023, pp. 5968 5976. International Joint Conferences on Artificial Intelligence, 2023.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint ar Xiv: 1 706. 02677, 2017.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Yean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 21284, 2020.
Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, and Karen Livescu. Signmusketeers:: An efficient multi- stream approach for sign language translation at scale. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 22506-2252) 2025a.
Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, and Alexander H Liu. Shubert: Self- supervised sign language representation learning via multi-stream cluster prediction. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 28792 28810, 2025b.
Huijie Guo and Lei Shi. Contrastive learning with semantic consistency constraint. Image and Vision Computing, 136:104754, 2023.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proc. CVPR, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. Signbert: Pre-training of hand- model-aware representation for sign language recognition. In Proceedings of the IEEE/CVE international conference computer vision, pp. 11087-11096, 2021.
Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. Signbert+: Hand-moddl-aware self-supervised pre-training for sign language understanding. IEEE Transactions on Pattern Analysis and Machine In- telligence, 45(9): 11239, 2023.
Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li, and Yun Fu. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3413 3423, 2021.
Zifan Jiang, Gerard Sant,, Amit Moryossef, Mathias Müller, Rico Sennrich, and Sarah Ebling. Signclip: Connecting text and sign language by contrastive learning. orXiv preprint arXiv:2407.0126; 2024.
Deep R Kothadiya, Chintan M Bhatt, and Imad Rida. Simsiam network based self-supervised model for sign language recognition. In International Conference Intelligent Systems and Pattern Recognition, pp. 3-13. Springer, 2023.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1459-1469) 2020a.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arxiv preprint arXiv:2005. 0aoo:2005004966, 2020b.
Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ting Chang, Ming Guang Yong, Jubyun Lee, et al. Mediapipe: framework for building perception pipelines. arxie preprint Xrv66 008.906668777 08172, 2019.
James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Sympostum on Mathematical Statistics and Probability, Volume 1: Statistics, volume 5, pp. University of California press, 1967.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Ariel Basso Madjoukeng, Jerome Fink, Pierre Poitier, Edith Belise Kenmogne, and Benoît Frénay. Bench- marking data augmentation for contrastive learning in static sign language recognition. In ESANN 2024: 32nd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learn- ing. 16doc. com, 2025a.
Ariel Basso Madjoukeng, Edith Belise Kenmogne, Pierre Poitier, Benoît Frénay, and Jerome Fink. Local- global data augmentation for contrastive learning in static sign language recognition. In IDA 2025: Intel- ligent Data Analysis. 2025b.
Marc Marais, Dane Brown, James Comnan, Alden Boby, and Luxolo Lethukuthula Kuhlane. Investigating signer-independent sign language recognition on the 1sa64 dataset. In Southern Africa Telecommunicaatinn Networks and Applications Conference (SA TNAC). Rhodes University Grahamstown, South Africa, 2022.
Sarfaraz Masood, Adhyan Srivastava, Harish Chandra Thuwal, and Musheer Ahmad. Real-time sign language gesture (word) recognition from video sequences using cnn and rmn. In Intelligent Engineering Informatics: Proceedings of the 6th International Conference on PICTA, pp. 623 632. Springer, 2018.
Leland Mclnnes, John Healy, and James Melville. Umap: uniform manifold approximation and projection for dimension reduction. arxiv. arXiv preprint arXiv:1802.088 03426, 10, 2018.
Katerina Papadimitrion, Galini Sapountzaki, Kyriaki Vasilaki, Eleni Efthimiou, Stavroula-Evita Fotinea, and Gerasimos Potamianos. A large corpus for the recognition of greek sign language gestures. Compuler Vision and Image Understanding, 249:104212, 2024.
Pierre Poitier, Jérôme Fink, and Benoît Frénay. Towards better transition modeling in recurrent neural networks: The case of sign language tokenization. Neurocomputing, 567:127018, 2024.
Katrin Renz, Nicolaj C Stache, Samuel Albanie, and Gül Varol. Sign language segmentation with temporal convolutional networks. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2135 2139. IEEE, 2021.
Franco Ronchetti, Facundo Manuel Quiroga, César Estrebou, Laura Lanzarini, and Alejandro Rosete. Lsa64: An argentinian sign language dataset. arXiv preprint arXfe:2810.14499 2023.
Mélanic Roschewitz, Fabio De Sousa Ribeiro, Tian Xia, Galvin Khara, and Ben Glocker. Robust image representations with counterfactual contrastive learning (2024). URL htlps:/ /arriv. org/ubs htn/abs/24000 22409. 10365, 2024.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Garrett Tanzer and Biao Zhang. Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus. arXiv preprint arXiv:2407.1114. 2024.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in newral information processing systems, 33:6827-6839, 2020.
Anirudh Tunga, Sai Vidyaranya Nuthalapati, and Juan Wachs. Pose-based sign language recognition using gen and bert. In Proceedings of the TEFF/CVF winter conference on applications of computer vision, pp. 31 40, 2021.
Christian Vogler and Dimitris Metaxas. Handshapes and movements: Multiple-channel american sign lan- guage recognition. In Gesture-Basee Communication in Human-Computee Interaction: 5th International Gesture Workshop, GW 2003, Genova, Italy April 15-17, 2003, Selected Revised Pupers 5, pp. 247 258. Springer, 2004.
Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Signrep: Enhancing self-supervised sign represen- tations. Xrxiv preprint ar Xiv:2500 08529, 2025.
20

---

Published in Transactions on Machine Learning Research (02/2026)
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. URL. htlpss//arrin. org/abs/2/4 2205.. 18781, pp. 2, 2024.
Zhenda. Xie, Yutong Lin, Zheng Zhang, Yue Cao; Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16684-16693, 2021.
Weichao Zhao, Hezhen Hu, Wengang Zhou, Jiaxin Shi, and Houqiang Li. Best: Bert pre-training for sign language recognition with coupling lokenization. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 3597-3605, 2023.
Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural langunge-assisted sign language recognition. In Pro- ceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 14890-14900, 2023.
21


Reviews:
[{"review_id": "ZU0oFrlbPu", "reviewer": "Reviewer_hRW7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes SSL-SLR, a self-supervised learning framework designed specifically to address the scarcity of annotated data in Sign Language Recognition (SLR). The contributions are twofold:\n\nSL-FPN Architecture: A novel \"negative-free\" self-supervised approach that utilizes three branches: the original sample and two augmented versions. It minimizes the distance between these representations and uses a predictor with a stop-gradient operator to align the original sample with its augmented variants, eliminating the need for negative pairs or clustering mechanisms.\n\nTemporal Augmentation Strategy: A new data augmentation technique that empirically identifies \"boundary importance\" (frames $k_{s}^{*}$ and $k_{e}^{*}$). The method degrades non-relevant parts of the video (typically the start and end) via permutation, forcing the model to focus on the discriminative \"internal\" segments of the sign.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by experimentation, though with a reliance on heuristic derivation for the augmentation strategy.\n\nEmpirical Results: The authors provide comprehensive benchmarks across five datasets (LSFB, LSA, GSL, ASL Citizen, WLASL)3. The results show that SSL-SLR consistently outperforms standard contrastive baselines (SimCLR, MoCo v2, BYOL, SimSiam) in linear evaluation, semi-supervised learning (with only 30% labels), and cross-lingual transfer tasks.\n\nVisual Confirmation: The UMAP visualizations and intra-class inertia metrics convincingly demonstrate that SSL-SLR produces more compact and discriminative embedding spaces than competing methods.\n\nHeuristic Limitation: The determination of \"relevant frames\" is derived from aggregate statistics (Fig. 3), leading to a global rule of preserving the middle segment (between $N/3$ and $N-N/4$). While shown to be effective on average, the claim that this method accurately identifies relevant parts for all signs is not fully substantiated for outliers that may have different temporal dynamics.", "audience_yn": "Yes", "audience_explanation": "SLR Researchers will be interested in the data efficiency of the method, particularly the ability to achieve competitive results with only 30% of annotated data and the improved transferability between sign languages.\n\nThe SL-FPN architecture is interesting as a simplification of existing SSL methods (like BYOL/SimSiam) by integrating the \"original\" un-augmented sample directly into the loss function to prevent collapse and improve alignment.", "requested_changes": "Robustness to Skeleton Noise: The method relies on MediaPipe for skeleton extraction. It would be beneficial to discuss if the \"degradation\" via permutation affects the model's robustness to skeleton estimation errors, which are common in sign language videos.\n\nClarification of Heuristic Generalization: The paper applies a fixed temporal crop ($N/3$ and $N/4$) across all datasets based on the \"average\" performance shown in Figure 3. The authors must address or analyze the failure cases where specific signs do not follow this temporal distribution (e.g., signs that start immediately without a setup phase). A critical discussion on the variance of $k_{s}^{*}$ and $k_{e}^{*}$ across different sign classes is needed.\n\nComputational Cost Analysis: While the method eliminates negative pairs (reducing batch complexity), it adds a third branch for the original sample. A comparison of training time and memory usage against SimSiam/BYOL would strengthen the efficiency claims.", "broader_impact_concerns": "Bias in Heuristics: There is a minor concern regarding the \"boundary importance\" heuristic18. If the timing of signs ($k_{s}^{*}$ and $k_{e}^{*}$) varies significantly between native signers and learners (or across different demographics), hard-coding these thresholds could introduce bias, optimizing the model for a specific style of signing. Adding a brief statement on the demographic diversity of the signers used to derive these thresholds would address this.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EMWGNnHoNZ", "reviewer": "Reviewer_b4zV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper introduces SSL-SLR, a self-supervised learning framework for sign language recognition that addresses two key limitations of existing contrastive methods: the high semantic similarity of negative pairs in SLR and the presence of non-discriminative temporal segments within sign videos. The framework combines a negative-free three-branch architecture (SL-FPN) that jointly leverages augmented views and the original sample, with a boundary-aware temporal augmentation strategy that degrades non-informative frames. Experiments on multiple sign language datasets show consistent improvements over standard contrastive and self-supervised baselines under linear probing, semi-supervised learning, and cross-lingual transfer.\n\nStrengths:\n\n1. The paper provides a clear and accurate diagnosis of why standard instance-discrimination methods (e.g., SimCLR, MoCo v2) underperform in sign language recognition, where different signs often share similar hand shapes and motion patterns, motivating the move away from explicit negative pairs.\n\n2. The proposed SL-FPN architecture is simple yet well motivated, distinguishing itself from BYOL and SimSiam by explicitly incorporating the original sample alongside augmented views, thereby improving the utilization of training signals within a single-encoder, negative-free framework.\n\n3. The boundary-aware temporal augmentation strategy is a strong domain-specific contribution, effectively addressing coarticulation and repositioning effects by identifying and degrading non-discriminative temporal segments using transformer sensitivity to temporal order.\n\n4. The experimental evaluation is thorough and well designed, covering multiple datasets (LSFB, ASL, GSL, and etc.) and learning settings, including linear probing, semi-supervised learning, and cross-lingual transfer, with consistent performance gains.\n\n\nWeaknesses:\n\n1. The positioning relative to recent self-supervised SLR-specific methods (e.g., SignRep) is limited, and a clearer comparison would help disentangle the benefits of the proposed augmentation strategy from architectural choices.\n\n2. The boundary importance selection relies on empirical ratios (N/3, N/4) derived from observed trends, and its robustness across datasets with different signing styles or continuous signing settings remains unclear.\n\n3. The computational cost of the boundary estimation procedure is not discussed in detail, and its scalability to longer sequences or larger datasets would benefit from further clarification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported by comprehensive empirical evidence across multiple datasets and evaluation settings.\n\n1. Comparisons with standard contrastive and negative-free baselines, together with ablation studies, demonstrate the effectiveness of the proposed SL-FPN architecture and boundary-aware temporal augmentation.\n2. Both quantitative results (linear probing, semi-supervised learning, cross-lingual transfer) and qualitative analyses (embedding visualization, intra-class inertia) support the claimed improvements in representation quality.", "audience_yn": "Yes", "audience_explanation": "The paper directly addresses representation learning under limited supervision, a topic of broad interest to the TMLR community. The paper will be of interest to researchers working on self-supervised learning, video representation learning, and low-resource sequence modeling. In particular, the domain-aware augmentation strategy and the analysis of why standard contrastive assumptions fail in SLR provide insights that extend beyond sign language recognition.", "requested_changes": "1. Add a related work section would benefit from a more explicit discussion of recent self-supervised methods tailored for sign language representation learning, such as SignRep, and other emerging approaches that focus on sign-specific inductive biases rather than generic visual SSL. Including these works would help clarify how SSL-SLR differs from existing SLR-focused representation learning methods, particularly in terms of boundary-aware augmentation versus architectural design.\n\n2. While the paper positions SL-FPN as a negative-free approach distinct from BYOL and SimSiam, the conceptual differences could be articulated more clearly. Discuss why incorporating the original (non-augmented) sample provides additional benefits beyond existing asymmetric Siamese frameworks, and under what conditions this design choice is expected to be most effective.\n\n3. The boundary importance parameters(N/3, N/4) are empirically derived from selected datasets(GSL, LSFB). The paper would benefit from a more detailed discussion on how sensitive performance is to deviations from these ratios, whether similar trends are observed across various datasets with different frame rates or signing styles, and how this procedure might generalize to continuous or sentence-level sign language recognition.\n\n4. The current experiments focus on isolated sign recognition. A brief discussion on how the proposed SL-FPN architecture and boundary-aware augmentation might extend to continuous signing, sentence-level SLR, or sign language translation pipelines would strengthen the broader relevance of the work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yi98SVSOns", "reviewer": "Reviewer_uxhV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a self-supervised method for sign recognition that is composed of two parts:\n- a three-loss architecture that tries to map positive augmented samples close to the original sample in the latent space, followed by a predictor for classification\n- an augmentation scheme that relies on a contrastive method to determine the most important frames of the sample that should be augmented\n\nI am not an expert in the field, so I cannot comment properly on the body of literature cited; however citations seem abundant especially as regards baselines the method is compared to.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper is well-written, organized, and with a reader-oriented style that makes it easy to read and understand the new contributions. The authors have justified every claim made in the paper with either empirical evidence or by supporting citations. It was a pleasure to read it.\n\nI believe the claims of better performance are properly supported. I specifically appreciate the intuition behind the use of a contrastive method to detect the best parts to augment, which exploits the inductive bias of certain architectures (Transformer) to achieve the intended goal.", "audience_yn": "Yes", "audience_explanation": "Avoiding the need to craft negative samples takes a huge burden off practitioners, which makes the method more appealing to a broader audience. The topic is in line with TMLR's scope, and it is well executed.\n\nI think this paper should be accepted following minor adjustments mentioned below.", "requested_changes": "I would like the authors to further clarify certain aspects:\n- Sec 5.1 (4th line): contrastive methods do not only use positive pairs. I think the sentence might be ambiguously written\n- Sec 5.1: \"randomly applying\" -> please specify in which sense: do you pick at random between T and T' ?\n- Please discuss why you do not utilize a specular loss as L2 using the other augmentation\n- End of page 4: n should be $n$\n- Equation 1: please provide an intuitive understanding for L3\n- Sec 5.2: \"it is well-established that...\" please provide a reference supporting your statement\n- I believe the authors should discuss the limitations of using a pre-trained contrastive method that is not necessarily readily available. Actually it was not clear if the contrastive method was trained or not. What is the trade-off here?\n- Please improve figures by adding labels (e.g. Fig 3) as well as not using screenshots (Fig 6)\n- Sec 5.2 \"the accuracy evolves\": does it increase/decrease? why do you think it should increase as in Figure 3 when you apply permutations of the first/last elements? This is not intuitive to me\n- Please clarify what \"accuracy in linear evaluation\" means. I could not find a proper explanation in the paper.\n- Page 9: there is a 80/10/10 split reported: how did you use the validation set?\n- I think the authors should discuss why they did not perform hyper-parameter tuning, including whether hyper-parameter tuning was performed for other methods. If no method performed a proper tuning of the hyper-parameters, it might happen that the ranking of the various models is biased by a certain architectural choice.", "broader_impact_concerns": "I have no specific concerns.", "additional_comments": "I think this work should be considered for a TMLR certification.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)