Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
Authors: Jianyuan Guo
Peike Li
Trevor Cohn
Venue: NeurIPS 2025 poster ()
Keywords: Sign Language Translation
Pseudo Gloss Generation
Large Language Model
Primary area: applications

Abstract: Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation.
Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. 
To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process.
This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss.
We train our SLT model—consisting of a vision encoder and a translator—through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language.
Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.

Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
Jianyuan Guo
Peike Li²
Trevor Cohn
City University of Hong Kong
2
Googge
Abstract
Sign Language Translation (SL.T) aims map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert- annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose gloss-free free pseudo gloss generation framework that eliminates the need for while preserving the structured intermediate representation. Specifically, we prompt Large Language Model (L..M) with few example text-gloss pairs using context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between --nerrrrdee pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT model conesincnsiiting of vision encoder and translator thhnuugg three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, approach outperforms previous state- of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.
glosses
Introduction
Sign languages are the primary means of communication for the deaf and hard-of-hearing com- munities [47], functioning as independent linguistic systems with unique lexicons, grammars and syntax [4]. Unlike spoken languages, they convey meaning using combination of manual gestures such as hand shapes, movements, and positioning, and non-manual cues, including facial expression, mouthing and eyebrow movements [33]. Despite their crucial role in communication, sign languages often lack standardized written forms (so-called glossing, which we discuss shortly), creating signifi- cant challenges for both accessibility and automatic translation. Sign Language Translation (SLT) has emerged as key research area aimed at converting sign language videos into textual representa- tion [32, 9, 8, 57, 10]. However, for many reasons, ranging from the complexity of the input (video) to their linguistic complexity, SLT remains a challenging problem within the field of computer vision and natural language processing.
The task of SLT is typically framed as sequence-to-sequeeeee problem, taking video input of a signer, which is processed by vision encoder, and then generating a text output in written language (e.g., English) using a translation decoder, such as an LSTM or Transformer. These approaches can be broadly categorized into gloss-based and gloss-free methods, based whether they incorporate gloss
Work done during an internship at Google. Correspondence to jianyguodcityu.co edu.hk
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Duration (h)
Vocabulary (K)
SOTA BLEU4
Dataset
Gloss
Language
train
val
test
total
train
val
test
total
Gloss-based
Gloss-free
A
Phocnix14T [8]
DGS
9.2
0.6
0.7
11
2
0.9
1
2.9
29.0' [20]
24.3 [26]
4.7
CSL DSily [60]
CSL
20.6
1.2
1,4
23
2
1.3
1.3
2.3
25.8 [11]
20.6 [26]
5.2
BOBSL
BSL
1236
20
204
1467
72
14
35
78
7.3 [27]
3.3 [27]
4.0
How2Sign [16]
ASL
69.6
3.9
5.6
79
15.6
3.2
3.6
16
15.5 [45]
OpenASI. [46]
ASL
288
33
21.2 [28]
Youtube-ASL [51]
ASL
984
60
Table 1: Comparison of sign language translation datasets. Results sourced from Papers With Code as of
April 2025 (large-scale pre-training methods [38] are excluded). Indicatess the oracle setting where previous
ground truth sentences are used in [27].
annotations. Glosses are written representation of sign language, denoting the sequences of signs used, which often differs substantially from the words used in the written language text, in terms of word choice, grammar and ordering, alongside sign-language specific concepts such as pronominal referents, classifiers, and non-manual signs. Overall, as linguistic description of sign language, glosses should provide a much richer source of supervision for understanding signed video input compared to the use of written language (closed captions).
Gloss-based methods typically involve pre-training the visual encoder on Sign Language Recognition (SLR) [13, 43], converting videos into gloss sequences. The task then becomes a standard machine translation problem, mapping these glosses spoken language text. However, these methods are limited by their reliance on expert-annee annotated gloss labels, which are time-consuming and costly to produce, as glossing not widely used in the deaf community. Accordingly, glosses are scarce in existing datasets [16, 51, 46]. In contrast, gloss-free methods [10, 58] have gained attention for directly training models on sign videos and corresponding spoken language text, bypassing the need for gloss annotations. While these methods streamline the pipeline and reduce annotation burdens, they face performance gap compared to gloss- based approaches, as shown in Table 1. This discrepancy arises from the inherent differences between sign and spoken languages, making direct translation more challenging. In this context, gloss annotations remain vital for bridging these linguistic gaps and ensuring effective translation.
Recent approaches seek to leverage approximate pseudo glosses to bridge the semantic gap between visual and textual features [59, 28J. For example. Sign2GPT [53] synthesizes pseudo glosses by filtering out function words based on their part-of-speech tags, retaining only nouns, adjectives, and numerals. However, real gloss sequences often contain terms not in the original text, and their order often differs from the written text. This misalignment between pseudo-glossed and video inputs complicates the use of conventional training methods, such as Connectionist Temporal Classification (CTC) loss [19], reducing their effectiveness.
To combine the advantages of gloss-free methods with improved performance, we propose a frame- work that generates pseudo glosses using LLMs. This approach enables us decompose the SLT process into pre-training and fine-tuning stages, as illustrated in Figure 1, bridging the gap between visual signals and spoken language, and allowing the SLT model to leverage gloss-like structures for enhanced performance. The core idea is that the semantic information of sign language is inherently embedded in reference translations provided alongside the video in standard SLT datasets, making possible to systematically derive approximate glosses from it. By harnessing the reasoning capabilittes of LLMs [49, 1]. we extract key information from the text generate pseudo glosses. To further improve this process, we utilize the in-context learning ability of LLMs, providing small set of example text-gloss pairs (e.g., 30). This allows the model to better capture the underlying structure and generate sbeudo-glossed that align with sign representations.
To mitigate the potential misalignment between LLM-generated glosses and the actual sign sequences in vidcos, we introduce reordering method, framed as a weakly supervised learning problem. We train an order-invariant classifier predict the set of glosses in each video, which then used to infer temporal alignment. Armed with reordered pseudo glosses, we decompose the SLT task into three key stages [10]: video to pseudo gloss recognition (Sign2Gloss), pseudo gloss to text translation (Gloss2Sign), and final fine-tuning (Sign2Text), as illustrated in Figure 1. This structured breakdown alleviates the challenges of direct video-to-text mapping and improves translation per- formance. This structured breakdown follows a curriculum learning [5] paradigm, where the model gradually progresses from simpler synthetic tasks to more challenging real-world translation tasks, thereby alleviating the challenges of direct vidco-to-text mapping and enhancing overall translation

---

Translation Loss
Transiation Loss
Pacudo Gloss
CTC Loss
Trunslation Lass
CTC Loss
Pscudo Gless
LM (Translator)
Pscudo Gloss
LM (Translatoc)
Text
LLM
classifier
Projector
classiffer
Projector
LM (Translator)
Visual Encoder
Visual Encader
Viaual Eucoder
Text (spoken language)
Frames
Pseudo Gloss
Video
Video
Pseudo Gloss Gemeration
Stage I: Sign2Gloss
Stage 2: Texss
Stage à.1: Sign2Test
Stage 3.2: Sigu2 Test
Figure I:
The training pipeline comprises three stages. Pseudo glosses are generated by LLMs and reordered using weakly supervised learning paradigms (see Sec. 3).
performance. Experiments demonstrate that our method outperforms existing gloss-free approaches and provides viable alternative to traditional ploss-basck methods.
Related Work
Gloss Supervision in SLT. Gloss-based SLT employs intermediate textual representation, known as gloss, to bridge sign video sequences and spoken language text. Gloss provide a precise, tem- porally aligned transcription of sign video sequences, offering explicit linguistic supervision that facilitates the learning of structured representations in SLT models. Early works [60, 8] adopted a two-stage approach, where SLT is decomposed into gloss recognition [15, 35J followed by machine translation [30, 2, 411. Inference proceeds in two steps: a Sign2Gloss module first predicts the gloss sequence, which then translated into text by Gloss2Text module. Later studies [10, 11] revealed that an end-to-end Sign2Text model can outperform the traditional Sign2Gloss2Tex pipeline by jointly optimizing both stages. However, the intermediate gloss remains essential for pre- pre-training Sign2Gloss and Gloss2Text modules, aiding in disentangling visual feature extraction from language modeling and enhancing structured learning.
Gloss-free SLT Framework. While gloss annotations offer precise alignment of keywords with sign videos, they are costly and require expert annotators with domain knowledge to transcribe videos at the lexical level. Consequently, many large-scale sign language datasets [16, 46, 51] lack gloss annotations. Recent methods [28, 56, 37, 59, 53] have explored gloss-free SLTT by learning implicit vision-text alignment without explicit gloss supervision. One line of research leverages pseudo- labeling techniques to approximate gloss supervision from raw vidco-text pairs. For example, GFSLT- VLP [59] pre-trained the visual encoder and language decoder using a CLIP-based contrastive learning method. Sign2GPT [53] heuristically retained only content words (noun, numeral, adjective, etc.) from the text to generate glosses, which are then used to pre-train the visual encoder. VAP [28] utilized CLIP-based word representations, pre-trained on large-scale corpora, to improve the alignment between visual frames and lexical words in the paired text. Other studies aimed to scale up the amount of training data [51, 38] and employed stronger pre-training strategies [45] to enhance SLT models.
Unlike previous gloss generation methods employing contrastive learning to roughly align frames with corresponding words in text, we propose to use LLMs to generate pseudo glosses". Sign language possesses its own grammar, vocabulary, and expression patterns. Leveraging in-context learning and linguistic understanding of spoken language (text), LLMs can akin human expert, infer reliable gloss-like annotations from text when provided with some gloss-text examples. These generated glosses serve as effective supervision signals, enhancing alignment between visual encoder and language decoder during the training of SLT model.
Pre-training for SLT. Another research direction focuses on enhancing SLT performance through pre-training. SignBERT [25] first proposed a self-supervised pre-training followed by downstream- task fine-tuning framework. Building on this, SignBERT+ [24] modeled hand poses visual tokens using multi-level masked modeling strategies and incorporated model-aware hand priors to capture hierarchical context. Meanwhile, SHuBERT [21] adapted masked token prediction objectives to multi-
*Our generated glosses are not tideo-derived true glosses (should only be translated from videos). Instead, we leverage LLMs' strength in key info extraction, core content is reflected the corresponding spoken text.
3

---

stream visual sign language inputs, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams. SSVP-SLT [45] leveraged MAE [23. 22] to pre-train a capable sign language encoder on video data alone without gloss or text annotations, achieving stat-of-the-art results on How2Sign.
Weakly Supervised Sequence Alignment. Connectionist Temporal Classification (CTC) [19] loss widely used in speech recognition and sequence modeling, allowing training without frame-level alignments. In our case, LLM-generated pseudo glosses may not match the actual temporal order of the signing sequence. To address this, we treat these pseudo glosses as an unordered label set and align them with the video using weakly supervised learning [6, 17, 34]. This process learns the correct temporal aligmment before applying the CTC loss. Similar weakly supervised methods have been explored in action segmentation [7, 36, 44], where only unordered action labels are available, with no temporal boundaries or ordering information. This line of work is closely related to our objective, where we seek to recover the temporal structure of L...-generated glosses from unordered supervision to enhance alignment between the visual encoder and language decoder.
3 Methodology
In this section, we first revisit previous gloss-based methods that decompose SL. into three training stages in Sec. 3.1. Then, we introduce our pseudo gloss generation method in Sec. 3.2. We describe how to reorder the pseudo glosses via weakly supervised learning in Sec. 3.3. Finally, we present the overall training and inference details of our approach in Sec. 3.4.
3.1 Preliminaries: Gloss-based Frameworks
Given an input sign video V fui.... ..., Ur } with T frames, the objective is to learn neural network (.) that directly translates the sign video into the corresponding spoken language sentence S = (81, } with U words (punctuation marks omitted here):
S NN,()),
To facilitate learning, prior works [10, 11] have decomposed SLT into three sub-tasks: (1) a visual recognition task that converts sign videos into glosses (Sign2Gloss), (2) a translation task that maps glosses to spoken language texts (Gloss2Tex)), and (3) an end-to-end SLT model that integrates both steps (Sign2Text). This hierarchical structure enables pre-training of the first two tasks independently before fine-tuning the joint SLT model.
Stage 1: Sign2Gloss. The goal of this stage, also known sign language recognition (SLR), is to predict the gloss sequence 9 - {gi,.... 9Ms } with M glosses from the sign video V. This is achieved using a vision encoder dh,, trained with the following objective:
CSLR
log (G vv)).
Stage 2: Gloss2Text. Given the gloss sequence G, a natural language translation model vhr generates the spoken language sentence S trained with the objective:
LSLT log p(S|g: s/r).
Stage 3: Sign2Text. To enable end-to-end finettuning an additional randomly initialized projector ø used map the visual features into the latent space of the translation model's input. This allows for joint optimization of both recognition and translation, with the overall objective formulated as:
(4) Usually, the CTC loss [19] is employed to compute V)G)V, while a cross-entropy (CH) loss with next-word prediction is used for p(S(g) and p(S]v). More details can be found in the appendix.
L = LSLT + LSLR = ( thr, vv, Φ) - log P(G)V; v/).
Traditional SLT methods heavily rely gloss annotations to align visual and textual representations. However, in a gloss-free setting. G is unavailable, posing a significant challenge to this paradigm. To address this, we propose PGG-SLT, annotation-efficient (applicable most gloss-free settings)* framework that does not require gloss annotations.
*Specifically, our approach is not strictly gloss- free on PHOENIX14T dataset, as we use small portion of available true gloss to generate pseudo- pseudo-glosses. glosses. For datasets like How that inherently lack gloss annotations, gloss-tex text examples from other datasets, making our method gloss-free free relative the target dataset.

---

3.2 Generating Pseudo Gloss via Large Language Model
The glosses provide more concise and direct representations of information that align with the video, compared to the full text. More importantly, their content remains semantically grounded in the original text, enabling a systematic extraction of key information to generate pseudo glosses. As shown in Table 2, certain function words in the text (e.g., articles and auxiliary verbs) are often redundant in glosses and can be omitted without affecting the conveyed mcaning.
Fortunately. well-trained multilingual large language models (LLMs) can serve as surrogate experts proficient in various sign languages, and are capable of performing the basic tasks of glossing such as filtering out unimportant words while preserving key content. Furthermore, by providing a few reference text-gloss example pairs, we can hamness the in-context learning ability of LLMs to generalize the gloss extraction process. Even with a minimal number of examples (e.g., just 30 pairs from Phoenix 14T, accounting for no more than 0.4% of the training split), the LLM can accurately identify and structure essential content words that convey meaningful information. Additionally, it can perform lemmatization, ensuring verb normalization and consistency across different tenses. This approach enables the efficient generation of reasonable quality pseudo glosses (refer to LLM Gloss) without requiring explicit linguistic annotations or expert intervention. More details about the prompts are illustrated in Figure 3, Figure 4, and Figure 6 in appendix.
im Nordwesten und Westen gibt cs vorübbrgehhend auch gefticrenden Regen. the northwest and there temporarily also freezing rain.) REGION (region) AUCH (also) / FROST (frost) REGEN (rain) / /MT (with) / FROST (frost)
Text
Phuemix 14T [8]
Gloss
Text
Fish often die because of the high concentrations of the toxin in the waters. WATER
Belebele [14]
Gloss
FISH/ OFTEN / DIE/ WHY TOO / MUCH/ TT-O-X-I-N
Table 2: Example text and gloss from German Sign Language (DGS) [8] and American Sign Language
(ASL) [14]. The dashes in 'toxin' denote finger spelling.
3.3 Reorder of LLM-Generated Pseudo Gloss
Although the provided text-gloss example pairs may inherently reflect some structural characteristics of gloss annotation, the pseudo glosses generated by the LLM tend to follow the word order of the spoken language. As a result, they often fail to align properly with the sign sequence in the corresponding video. Consequently, supervision approaches such as CTC loss, which rely on strict sequential alignment, are not suitable for training with current pseudo glosses.
To address this issue, we need to correct the order of pseudo glosses. Currently, the pscudo glosses generated purely from text, lacking any temporal grounding from the vidco itself. However, the video inherently contains rich temporal cues that can guide the correct sequencing of signs. Given the LLM Gloss Gy= = {9. 9,. ..., 9r} generated in Sec 3.2, we propose a video-guided refinement process in a weakly supervised manner to better align Sy: with the natural temporal structure of the video. To achieve this, we first denote a video-specific vocabulary set as Uy: = (or,.., ..., In J. where is the number of unique glosses for video V Aggregating the pseudo glosses across the entire dataset, we construct global vocabulary set denoted as L/ {91 .... 9KK } size K, representing all unique glosses generated by the LLM. Then, we extract frame-wise visual features from every video ' using vision encoder v/v, yielding feature sequence F' (fl (f...... fr) Tx where D is the feature dimension. We then use classifier dets to map these features to the global pseudo gloss vocabulary 21 producing sequence of logits Z' E T K Since each frame should correspond to at most one gloss label, we apply softmax activation along the vocabulary dimension (K):
P softmax(Z),
(5)
where P represents the frame-wise probability distribution over the global pseudo gloss vocabulary.
Weakly Supervised Multi-Label Classification. Here, we describe the training process for the classifier dds. Since only have weak supervision, i.e., we have an estimated set (by LLM) of which words (Uyy)) are present in the video but lack precise timestamp annotations, classification (MLC) problem [40]. During the training phase, we first consolidate the predictions P E T into a video-level representation by applying max-pooling operation over the temporal dimension (T):
frame this as multi-label
8 = max PLk IE [T]]
Vk E (l, K),
(6)
5

---

where ÿ E IK denotes the predicted presence probabilities for each gloss in the global vocabulary. We then optimize the classifier using Binary Cross- Cooss-Entoopy (BCE) loss:
LACE - - Σ we [y% log Ur + (1 - y/.) log(1 - 9x)]
(7)
where yk E {0,1)
}} denotes whether gloss & appears the draft (I.LM- --genrrrtted pseudo gloss, and Wg is a weighting factor. To address the long-tailed distribution of pseudo gloss words words-where only a small subset of the vocabulary appears in each video (c.g., 7 vs. 1, 100 in Phoenix 14T [8])-we introduce frequency-ave awware weighting to reweight the BCE loss:
Wek =
Whase log (Smax/ [k)
(8)
where f& is the frequency of gloss k and fmax the frequency of the most common gloss. These frequencies are computed based on the initial pseudo gloss drafts. This encourages the model to pay more attention to rare glosses, mitigating the imbalance problem.
Temporal Smoothing Constrain. To enforce temporal consistency and encourage smooth transitions between consecutive frames, we introduce an LI regularization loss on the frame-wise logits:
T Σ IP, P.till
Samoohh
This penalizes abrupt changes in gloss predictions across adjacent frames. The final loss function combines the BCE loss for weakly supervised gloss prediction and the L1 smoothness loss:
LBCE + Camooth
(10)
Frame-wise Gloss. Based on the frame-wise probability distribution P E T' we can perform max- max-pooling operation along the vocabulary dimension (K) derive the frame-wise gloss ý glosss max Pi, E R.?.
Greedy Algorithm for Reordering LLM Gloss. With the frame-wise probability distribution Pa obtained from Eq. 5, our goal is to refine the LLM Gloss to better align with P, by designing greedy algorithm that iteratively adjusts the LLM Gloss to resemble the frame-wise predictions. The process involves the following steps:
(i) Filter irrelevant words. We first filter out words from the Frame-wise Gloss that are not present in the LLM Gloss, ensuring that only relevant terms are considered.
(ii) Merge consecutive duplicates. Since certain glosses may be predicted for multiple consecutive frames, we consolidate adjacent duplicates to form a streamlined Ref Gloss.
(iii) Greedy reordering. We perform a greedy reordering strategy to better align the L.MM Gloss with the temporal structure suggested by the Ref Gloss. The key intuition is to leverage the order of words in the Ref Gloss as a soft temporal reference to reorganize the LLM Gloss, resulting refined sequence denoted as ςy Sy.tagget If a word in the LLM Gloss is absent from the Ref Gloss, it is directly retained in its original order. For words that appear in both glosses but are out of order, we reorder them according to their first appearance in the Ref Gloss. More details are shown in Sec. A.2 in the appendix. This adaptive alignment ensures that the final sequence respects the temporal cues inferred from the vidco, enhancing consistency and coherence with the visual content.
Table and Algorithm provide a illustration of the reordering process. Once reordered, the resulting Vvaagg is used to train the SLT model, following the strategy described Sec. 3.4.
3.4 Training and Inference
Our training pipeline is illustrated in Figure 1. After generating the reordered gloss, we decompose the network training into three stages: i.e., Sign2Gloss, Gloss2Text, and Sign2Text, following prior gloss- based approaches [10, 11] discussed in Sec. 3.1. In stage (Sign2Gloss), we train a vision encoder to map sign video features to pseudo gloss sequences using the CTC loss. In stage (Gloss2Text), we leverage pre-trained multilingual language model, such mBART [41] or Gemma2 [50], and fine-tune it on the gloss-to-text translation task. In stage 3 (Sign2Text), we introduce an ad- ditional projection layer, implemented as MLP with two hidden layers randomly initialized.

---

Text (S) Gloss (C7)
in norden und westen mal somne mal walken (In the north and west someetimes sun somelimes clouds.)
NORDWEST/ WECHSEL / SONNE/ WOLKE (northwest changeable cloud)
Gloss (9lv)
NORD/ WEST SONNE/ WOLKE/ WECHSELHAFT (morth west sun cloud changeable)
NORD/ NOOR /WEST/ / WECHSELHAFFT WECHSELHAFT
/SONNE/ SONNE WWOLKE
Frame wise Gloss
REGION NORD/ /EST WECHSEL HAFT/ / SONNE/ WOLKE
Ref Gloss
Reordered Gloss (Gv.earget)
NORD/ / WECHSELHAFT/ /SONNE WOLKE (oorth / west changeable/ / sun cloud)
Table 3: Example of the L.L--generaed pseudo gloss (LLM Gloss) and corresponding reorder operation. NORD means merging consecutive duplicates. REGION means discarding words not present the LLM Gloss.
The pseudo gloss serves as an auxiliary supervision signal via the CTC loss, similar to stage However, since the pseudo gloss is not an exact transcrip- tion compared to the true gloss, we remove the CTC loss in the final epochs (Stage 3.2) of training to prevent over-reliance on its potentially noisy supervi- sion.
Algorithm 1: Reordering of LLM Gloss
Input: LLM Gloss: C [m.. Ref Gloss: R [ri...,,Te Tul
Output: Reordered Gloss: farget [f1, ..., Im.
Initialize largel <- []] (empty sequence of length [C]):
Initialize two pointers: + 0. + 0:
while < [2] do
i, = then
Append 1, to target;
i i 1,j + j
else if & R then
Append I, to target;
i ++;
else r appears at later position in & (denoted as ly) then Append largel;
At test time, the model takes video frames as input and di- rectly generates spoken lan- guage text, bypassing the in- termediate pseudo gloss. This approach allows the model to leverage structured pseudo gloss supervision during training to bridge vision and language, while maintaining the flexibility of end-to-end inference for fluent and accurate text generation.
Remove by from L;
j+
return target
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate our proposed method on two widely used SLT datasets. Phoenix14T [8] consists of 8.257 fully glossed German Sign Language (DGS) videos with corresponding German translations and sign glosses, sourced from weather forecast programs. The dataset is split into train/dev/test sets, containing 7,096/5 19/642 videos, respectively. How2Sign [16] is large-scale American Sign Language (ASL) dataset without gloss annotation, featuring 79 hours of videos with English trans- lations, containing a vocabulary of approximately 16,000 unique English words. We utilize the manually re-aligned video clips, with train/dev/and test splits consisting of 31, 11.10,1,,527/2.349 samples, respectively.
Evaluation Metrics. Following prior works [10, 12, 28], we adopt BLEU [42] and ROUGE [39] to assess SLT performance. Higher BLEU and ROUGE scores indicate better translation quality.
Implementation Details. Following [11, 10]. we adopt S3D [54], pre-trained on Kinetics-400 [31], as visual encoder for sign videos. Each input video of shape T x H x W x 3 is fed into the encoder, where T denotes the number of frames, and H and W represent the height and width of the video, respectively. The encoder extracts spatiotemporal features, which are then spatially pooled to size of T/4 x 512. These features are passed through a projector consisting two fully connected layers. For the translator, initialize either m [41] or Gemma2 [50] with pre-trained checkpoints from HuggingFace [52]. We use AdamW optimizer along with a cosine annealing learning rate schedule. The total training process is divided into four stages, with 40 epochs allocated stages 1, 2, 3.1, and 3.2, respectively. More implementation details are reported Appendix.
httpe:///hooetggggiiiee google/ 2-2b-it
htt.://gww.ggaaaocoo//ccccaaaaaa large- cc25, "tttps://bugiigiacco..
7

---

4.2 Comparison against State-of-the-art Methods
Results on Phoenix14T. We present our main results in Table 4. Our proposed PGG-SLT significantly
outperforms previous gloss-free methods. When using the same mBART [41] language decoder, PGG-SLT improves the BLEU4 score by 5.0/5.4 over GFSLT-VLP [59] and 0.5/0.8 over VAP [28] on the Dev/Test sets, respectively. Compared to the previous state-of-ther of-the-art VAP [28], method does not require either a skeleton-specific encoder [29] or any additional training of a translation model. Surprisingly, PGG-SLT achieves performance competitive with gloss-based methods such as MMTLB [10], while using only 30 gloss annotations out of 7K available in the dataset. For any SLT dataset, the cost of annotating few dozen glosses is negligible. This result suggests that our pseudo- glosses closely approximate real gloss annotations, significantly boosting performance compared other greudo-gloss-basse
approaches like Sign2GPT [53]. Furthermore, when equipped with stronger LLM translator such as Gemma2-2B [50], PGG-SLT achieves even better
gloss-based methods on metrics such as BLEUI score. This improvement can likely be attributed to Gemma2's s richer vocabulary (tokenizer), enabling more precise and diverse language generation.
Results on How2Sign. We further evaluate our method on the more challenging How2Sign [16] dataset (more details in Sec A. in appendix), with results summarized in Table 5. Since How2Sign does not provide gloss annotations, closs-based methods are not applicable in this setting. Compared to competitor methods, our models consistently outperform VAP [28] in both BLEU4 and ROUGE scores. When compared SSVP-SLT [45], which leverages self-supervised learning to pre-train the visual encoder, our best model achieves a 6.1 BLEU4 improvement on the test set. While prior works [51, 38, 45] have demonstrated that large-scale training benefits SLT models, our approach remains highly competitive-eve even surpassing Uthus er al. [51] by significant margin. These results highlight the effectiveness of our pseudo gloss generation in facilitating better alignment and training of SLT models. Importantly, the strong performance on both Phoenix 14T [8] and How2Sign [16] demonstrates that LLMs can serve as powerful assistants for generating righ-quality pseudo gloss annotations. Furthermore, our method scales effectively with dataset size, making it well-suited for worldw large-scale SLT applications.
Dev Set BLEU3
Test Set BLEU3
Method
BLEUI BLEU2
BLEU4 ROUGE BLEUI
BLEU2
BLEU4 ROUGE
Gloss-based
MMTLB [1]]
33.95
41.12
33.14
27.61
53.10
53.97
41.75
33.84
28.39
52.65
SLTUNet [57]
27.87
52.23
52.92
41.76
33.99
28.47
52.11
IP-SLT [55]
54.10
41.56
33.66
28.22
54.43
54.25
41.51
33.45
27.97
53.72
TS-SLT [11]
54.32
41.99
34.15
28.66
54.08
54.90
42.43
34.46
28.95
53,48
Gloss-free
NSLT [8]
31.58
18.98
13.22
10.00
32.60
29.86
17.52
11.96
9.00
30.70
GASLT [56]
39.07
26.74
21.86
15.74
39.86
GFSLT-VI.P [59]
44.08
33.56
26.74
22.12
43.72
43.71
33.18
26.11
21.44
42.49
SignLLM [18]
46.88
36.59
29.91
25.25
47.23
45.21
34.78
28.05
23.40
44.49
Sign2GPT [53]
49,54
35.96
28.83
22.52
48.90
FLa-LLM [12]
46.29
35.33
28.03
23.09
45.27
VAP [28]
52.78
26.62
51.47
53.07
26.02
51.28
PGG-SLT (mBART)
53.16
40.38
32.66
27.09
52.01
53.45
40.55
32.65
26.85
$1.85
PGG-SLT (Gemma2)
53.84
41.08
32.82
27.53
52.85
54.02
41.23
32.89
27.32
52.56
Table 4: Comparison with state-of-the-ar art methods on Phoenix We report results using two translators: mBART [41] and Gemma2-28 [50]. Equipped with generated pscudo gloss, our approach surpasses previous gloss-free methods and achieves performance comparable to gloss-based MMTLB.
4.3 Ablation Study
In-context Learning Ability of LLM. Our key hypothesis is that LLMs can act as human experts to extract key lexical information from a sentence and generate corresponding gloss annotations. Although sign language differs systematically from spoken language, e.g., in grammar or the use of named entities, these modality-specific biases may not be immediately captured by LLMs. However, such gaps can be narrowed down by providing LLM with small number of gloss-text example pairs, allowing it to adapt to the unique linguistie patterns of sign language through in-context learning. As shown in Table 6a, even without any examples, LLM-generated glosses achieve BLEU4 score
8

---

Dev Set BLEU3
Test Set
Method
BLEU1 BLEU2
BLEU4 ROUGE
BLEUI
BLEU2
BLEU3
BLEU4
ROUGE
Uthus el al. [51]
15.0
5.1
2.3
1.2
SSVP-SLT [45]
30.2
16.7
10.5
7.0
25.7
Tarrés al. [48]
35.2
20.6
13.3
8.9
34.0
19.3
12.2
8.0
FLa-LLM [12]
29.8
19.0
13.3
9.7
27.8
VAP [28]
42.3
14.9
30.3
39.2
12.9
27.8
Uthus et al. [51]
37.8
24.1
16.9
12.4
Uni-Sign [38]
40.2
14.9
36,0
SSVP-SLT [45]
43.2
28.8
20.8
15,5
38.4
PGG-SIT (mBART))
41.8
26.5
19.7
15.2
32.6
38.9
25.4
18.1
13.1
31.5
PGG-SLT (Gemma2)
43.4
27.7
21.1
15,9
34.8
40.8
25.9
18.6
13.7
32.9
Table 5:
Comparison with SOTA methods on How2Sign. We report results using two translators: mBART [41] indicates the models use extra large-scale datasct [46, 51] for pre-training.
and Gemma2-2B [50].
Gloss (Train)
Gloss (Test)
Test set n4
Gloss (Test)
# Pairs
B11
B44
WER-
B1"
B44
Pseudo Gloss
B1
LLM
WER B1
B4
POS [53]
95.8 28.7
2.1
96.5 28.2
2.0
POS [53]]
46.3 20.8
BB
65.9 48.5 11.9
0
67.5 44.6
10.7
67.1
43,2
10.5
# Pairs = 0
52.5 25.6
Gemma2-27B
65.2 49.2 12.2
30
62.8
53.1
14.3
62.1
53.6
13.8
# Pairs 30
53.1 26.2
GeminiXL-IT
63.9 52.8 12.6
61.2
55.2
17.9
60.9 55.7 17.2
True Gloss (7K)
53.9 28.2
Geminil 1,5ro
62.1 53.7 13.8
(a) Quality of generated pseudo gloss.
(b) SLT results.
(c) Different LLMs (30 pairs).
Reorder
Gloss (Train)
PHX [8]
H2S [16]
Emmonts We
Prec.
Gloss (Tran)
Rec.
WER
Bat
Prec. f
Gloss (Test)
Ree. f WER
B4*
WFR
B13
B4"
Bi
B4
B11
B4"
302
833
X
62.8
53.1
14.3
53.1
26.0
38.5
12.6
862
W9
3000
33.
13.
84.8
42.9
36
90.7
99.2
25.6
57.6
76.6
84.1
37.0
4144
60.1
53.1
17.4
53.5
26.9
38.9
13.1
91.7
99.3
228 61.9
78.1
85.2
36.1
T3.
(d) Reorder operation for pseudo gloss (30 paris).
(e) Components for training the reordering classifier.
Table 6: Ablation study on Phoenix14T: (a) the similarity between generated pseudo gloss and true gloss. measured by Word Error Rate (WER) and BLEU score, "POS" means the Parts-of-Speech tagging in [53]; (b) SLT performance using diffcrent pscudo gloss variants; (c) the similarity between pscudo gloss generated by different LLMs and the true gloss. Our default setting is shown by gray...
of 10.7 when compared with ground-trutt glosses, significantly outperforming prior beuristic-based pseudo gloss methods [53] (labelled "POS" in the table), with over 30% relative improvement in WER and an 8% increase in BLEU4. When provided with 30 example pairs, the BLEU4 score of generated glosses increases 14%. The performance continues to improve as more examples are given. While there's s still headroom for improvement, the progress so far is already significant, demonstrating the effectiveness of LLMs' in-context learning capabilities.
Effectiveness of the Pseudo Gloss. We present the SLT results trained (excluding stage 3.2) with different types of pseudo glosses in Table 6b. The glosses generated by the LLM already provide strong baseline under a fully gloss-free setting. When 30 ground-truth gloss annotations are included in-context examples, the final translation quality further improves. Using the full set of ground- truth glosses serves as the performance upper bound of our approach.
Ablation on Different L.Mss. To isolate the impact of L.MMs, we compare different model versions and parameter scales while using the same 30 pairs for long-context in-context learning. As shown in Table 6c, larger models, such Gemma2-27B [50], tend to generate higher-quality glosses. More advanced models, such as Gemini 1.5 Pro [49], further improve the quality of generated glosses and achieve the best performance.
Effectiveness of the Reordering Operation. We evaluate the proposed reordering operation in Table 6d. The core improvement introduced by this operation lies in reducing the linguistic asymmetry between L-M--gnerated glosses, which follow spoken language syntax, and ground-truth sign language glosses, which are aligned with the visual signs in the video. After reordering, the similarity between the pseudo glosses and the ground-truth glosses is improved, leading to better performance during the training of the SLT model.

---

Ablation on Weakly Supervised Classification. We employ two constraints to assist in training the classifier for reordering (See 3.3): temporal smoothing constraint Ssmotth and the frequency- aware weighting coefficient wi. To evaluate their effectiveness, we conduct this ablation study using ground-truth glosses while excluding their temporal order information. As shown in Table 6e, the baseline MLC solution, using BCE loss, achieves satisfactory precision and recall. With the addition of Smmothh, both WER and BLEU4 scores significantly improve, demonstrating that the predicted order is closer to the ground-truth The wy further enhances the accuracy of the pseudo gloss.
Effectiveness of Vision- Vision-Language Alignment in Training. Our key contribution lies in generating high-quality pseudo glosses to enhance the training of SLT models. Figure 1 outlines the overall training pipeline. A key question is: what happens if no gloss annotations are used? As shown in Figure 2 and Table 7, directly training the Sign2Texe Text model using only sentence-level supervision (Stage 3.2) results in poor cerrommanne-yilddiie and How2Sign, respectively. Increasing training epochs does not help and even degrades performance. In contrast, incorporating our pseudo glosses (via Stage and Stage 2) leads substantial performance gains, underscoring the critical role of vision-language alignment in SLT training. Moreover, higher- quality glosses yield stronger alignment signals, further boosting model performance.
BLEU4 scores of just 8.2 and 2.3 on PHOENIX 14T
Train Stage
PHX [8]
H2S [16]
10
2
3.1
3.2
BLEU1
BLEU4
BLEU1
BLEU4
X
X
X
/
24.2
8.2
17.5
2.3
/
/
/
50.9
24.8
37.2
11.5
X
52.3
26.2
38.2
12.7
53.5
26.9
38.9
13.1
Table Impact of different training stages.
20
30
Figure 2: BLEU4 epochs Phoenix 14T test set.
5
Conclusion
In this paper, we propose leveraging LLMS to address the challenging problem of Sign Language Translation (SLT) by generating pseudo glosses in a gloss-free setting. Our method, PGG-SLT, demonstrates significant performance improvements on the Phoenix147 and How2Sign datasets. We show that LLMs can serve as experts in extracting key information from sentences, and their in-context learning capabilities help capture potential grammatical patterns and glossing conventions. Moreover, introduce weakly supervised setting to reorder the generated pseudo glosses, mitigating the misalignment between LLM -eM----rettted glosses and the sign sequences in the video. We believe this approach presents a promising direction for leveraging LLMs to enhance SLT model training.
Limitations and Future Work. One limitation of this work is that we only scaled the dataset from Phoenix 14T to How2Sign, and the performance of our proposed PGG-SLT when incorporating additional training data remains unvalidated. As shown in Table 5, two other methods that leverage large-scale datasets for pre-training achieve slightly better results than model. Moving forward, we plan to evaluate the effectiveness our pseudo-gloss generation method on larger ASL benchmarks. In addition, this paper is mainly about pseudo gloss generation, we first want to clarify how glossing are defined and described here. Indeed, reducing certain action video frames to a few "words" inherently limits the expressiveness of sign language in such representations, This is why resources like 2M- -l--Fooss-ALL incorporate markers such as "+++" to indicate repeated actions, "cl:" to denote specific handshape classifiers, as well as markers like "//WHATW" and "//WHWW" to represent NMMs (with // and WI indicating their span) in their dataset. Further, sign language exhibits substantial variability, different sign sequences
convey the same meaning, much like how spoken English uses diverse vocabulary for identical concepts. Crucially, text (spoken language) alone cannot perfectly reverse-engineer the specific visual details (e.g., precise hand movements, spatial configurations) captured in video. That said, given current constraints (e.g., dataset scale and model capabilities). we think that our current relatively "simple" glossing approach offers practical advantages: lowers the learning barrier for models and facilitates more measurable improvements in SLT performance (e.g., BLEU scores). Implement a more nuanced, expressive representation will also be our future work.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilee Akkaya, Florencia Iconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08772 2023.
[2] Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884, 2019.
[3] Samuel Alhanic, Gäl Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al. Bbe-oxford british sign language dataset. arXiv preprint Xrx::21 1,00635, 2021.
[4] David F Armstrong. William C Stokoe, and Sherman E Wilcox. Gesture and the nature of language. Cambridge University Press, 1995.
[5] Yoshua Bengin, Jérôme Louradour, Ronan Collohert, and Jason Weston, Curriculum learning. In Proceed- ings of the 26th annual international conference on machine learning, pages 41-48, 2009.
[6] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2846-2855 2016.
[7] Piotr Bojanowski, Rémi Lajugie, Francis Bach, Ivan Laptev, Jean Ponce, Cordelia Schmid, and Josef Sivic. Weakly supervised action labeling in videos under ordering constraints. In Computer Vision-ECC ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 628 643. Springer, 2014.
[8] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784 7793, 2018
[9] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages NU023-10033, 2020.
[10] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5120-5130, 2022.
[11] Yutong Chen, Ronglai Zuo, Fangyun Wei. Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems, 15:17043-17056, 2022.
[12] Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learning assisted with large language model for gloss-free sign language translation. arXiv preprint arXiv:2403. 12556. 2024.
[13] Helen Cooper, Brian Holt, Richard Bowden. Sign language recognition. Visual Analysis of Humans: Looking at People, pages 539-562. Springer, 2011.
[14] Marta R Costa-jussà, Bokai Yu, Pierre Andrews, Belen Alastruey, Necati Cihan Camgoz, Joe Chuang, Jean Maillard, Christophe Ropers, Arina Turkantenko, and Carleigh Wood. 2m-belebele: Highly multilingual speech and american sign language comprehension dataset. arXiv preprint wwxxx:2412.082722 2024.
[15] Runpeng Cui, Hu Liu, and Changshui Zhang, Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. Proceddiggs of the IEEE conference on computer vision and pattern recognition, pages 7361-7369,
[16] Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze. Jordi Torres, and Xavier Giro-i Nicto. How2sign: large-scale multimodal dataset for continuous american sign language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2735 27445, 2021.
[17] Thibaut Durand, Taylor Mordan, Nicolas Thome, and Matthieu Cord. Wildcat: Weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 642 651, 2017.
11

---

[18] Jia Gong, Lin Geng Foo. Yixuan He. Hossein Rahmani, and Jun Liu. Lims are good sign language translators. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 18362-18372, 2024.
[19] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: lahelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 233r international conference on Machine learning, pages 369-376, 2006.
[20] Mo Guan, Yan Wang. Guangkun Ma, Jiarui Liu, and Mingzu Sun. Multi-stream keypoint attention network for sign language recognition and translation, arXiv preprint arXiv:2405.05662, 2024.
[21] Shester Gueuwou, Xiaodan Du, Greg Shakhnarovich, Karen Livescu, and Alexander HI Liu. Shubert: Self-supervised sign language representation learning via multi-strean cluster prediction. arXiv preprint urXiv:2411.16755. 2024.
[22] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Yunhe Wang, and Chang Xu. Fastmim: Expediting masked image modeling pre-training for vision. arXiv preprint wrXiv:2212.06555 2022.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[24] Hezhen Hu, Weichao Zhao, Wengang Zhou, and Houqiang Li. Signbert+: Hand-model-awe -ware self- supervised pre-training for sign language understanding. IEEE Transactions on Puttern Analysis and Machine Intelligence, (999):1122111229 2023.
[25] Hezhen Hu, Weichao Zhao, Wengang Zhou, Yuechen Wang, and Houqiang Li. Signbert: Pre-training of aaree representation for sign language recognition. In Proceedings of the IEEE/CVF international conference computer vision, pages 11087-11096 2021.
[26] Eui Jun Hwang. Sukmin Cho. Junmyeong Lee, and Jong C Park. An efficient sign language translation using spatial configuration and motion dynamics with llss. arXiv preprint arXiv:2408. 10593, 2024.
[27] Youngjoon Jang, Haran Raajesh, Liliane Momeni, Gül Varol, and Andrew Zisserman. Lost in translation, found in context: Sign language translation with contextual cues. arXiv preprint arXiv:2501.097: 2025.
[28] Peiqi Jiao, Yuecong Min, and Xilin Chen. Visual alignment pre-training for sign language translation. In European Conference on Compuder Vision, pages 349-367. Springer, 2024.
[29] Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, and Xilin Chen. Cosign: Exploring co- occurrence signals in skeleton based continuous sign language recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
[30] Melvin Johnson, Mike Schuster, Quoe V Lc, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, al. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics. 5:339-351, 2017.
[31] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 17X5::7005006500, 2017.
[32] Sang- Ki Ko. Chang Kim, Hyedong Jung, and Choongsang Cho. Neural sign language translation based on human keypoint estimation. Applied sciences, 9(3))2663, 2019.
[33] Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. Weakly supervised learning with sulti--tream cnn-Istm- hmms to discover sequential parallelism in sign language videos. IEEE transactions on pattern analysis and machine intelligence, 42(9):2306-2223 2019.
[34] Oscar Koller, Hermann Ney, and Richard Bowden. Deep hand: How to train a enn on million hand images when your data is continuous and weakly labelled. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3793-3802, 2016.
[35] Oscar Koller, Sepehr Zargaran, and Hermann Ney. Re-sign: Re-aligned end-to-end sequence modelling with deep recurrent can-hmms. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 4297-4305, 2017.
[36] Hilde Kuehne, Alexander Richard, and Juergen Gall. Weakly supervised learning of actions from transcripts. Computer Vision and Image Understanding, 163:78 89, 2017.
12

---

[37] Dongxu Li, Chenchen Xu, Xin Yu. Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. Advances in Neural Information Processing Systems, 11204-1204455 2020.
[38] Zecheng Li, Wengang Zhou, Weichao Zhao, Kepeng Wu, Hezhen Hu, and Houqiang Li. Uni-sign: Toward unified sign language understanding at scale. arXiv preprint (rXii:2501.5156 2025.
[39] Shin-Yew Lin. Rouge; A package for automatic evaluation of summarics. In Text summarization branches out, pages 74-81, 2004.
[40] Weiwei Liu, Haobo Wang, Xiaobo Shen, and Ivor W Tsang. The emerging trends of multi multi-label learning. IEEE transactions on pattern analysis and machine intelligence, 44(11)):955-799 7974, 2021.
[41] Yinhan Liu, Jiatao Gu, Naman Goyal. Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742, 2020.
[42] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318,22 2002.
[43] Razieh Rastgoo, Kourosh Kiani, and Sergio Escalera. Sign language recognition: A deep survey. Expert Systems with Applications, 164:11379A 2021.
[44] Alexander Richard, Hilde Kuehne, and Juergen Gall. Action sets: Weakly supervised action segmentation without ordering constraints. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 5987-5996, 2018.
[45] Phillip Rust, Bowen Shi, Skyler Wang. Necati Cihan Camgākz, and Jean Maillard. Towards privacy-aware sign language translation at scale. arXiv preprint 1rXiv:2402.00661, 2024.
[46] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Liveseu, Open-domain sign language translation Iearned from online vidco. arXiv preprint arXiv:2205. 12870, 2022.
[47] Richard JH Smith, A Eliot Shearer, Michael S Hildebrand, Guy Van Camp, et al. Deafness and hereditary hearing loss overview. GeneReviews, 1993.
[48] Laia Tarrés, Gerard Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró-i Nieto. Sign language translation from instructional videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3625-5635 2023.
[49] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[50] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at a practical size. arXiv preprint 24Xiv:2408.00//6 18, 2024.
[51] Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtuhe-asl: A large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems, 36:29029-29047, 2023.
[52] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint 1rXivv1910003371, 2019.
[53] Ryan Wong, Necati Cihan Camgoz, and Richard Bowden Sign2gnt: I Leveraging large language models for gloss-free sign language translation. arXiv preprint arXiv:2405.04 046404 2024.
[54] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed- accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), pages 305-32. 321, 2018
[55] Huijie Yao, Wengang Zhou, Hao Feng. Hezhen Hu, Hao Zhou, and Houqiang Li. Sign language translation with iterative prototype. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
13

---

[56] Aoxiong Yin. Tianyun Zhong. Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF conference computer vision and puttern recogmition, pages 2551-2562, 2023.
[57] Biao Zhang, Mathias Müller, and Rico Sennrich. Situnet: A simple unitied model for sign language translation. arXiv preprint arXiv:2305.01770 2023.
[58] Jian Zhao, Weizhen Qi. Wengang Zhou, Nan Duan, Ming Zhou, and Houqiang Li. Conditional sentence generation and cross-modee reranking for sign language translation. IEEE Transactions on Multimedia, 24:2662 2672, 2021.
[59] Benjia Zhou, Zhigang Chen, Albert Clapés. Wan. Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. In Pruceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 20871-20880 2023.
[60]
Hao Zhou, Wengang Zhou, Weizhen Qi. Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign tack-translation In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1316-1325, 2021.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope. Specifically, the use of LLM- LuM-ggnerated pseudo glosses and the refinement process highlighted key strategy for enhancing model training, effectively bridging the gap between video content and linguistic representation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: This paper discusses the limitations of the proposed method in various sec- tions. For instance, we address the dependency on LLM-ggnnrrted pseudo glosses and acknowledge potential noise and misalignment issues due L.MM's lack precise temporal understanding.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex" system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledger in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: We provide essential assumption and validation in both the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems. The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: For the reproducibility, we show the details in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
The data and corresponding model, e.g.. LLMs, are all available and open- sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nig) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtpppccc/ppppccccccccccccccceee hellliccguiiesss weweeeee/oooeSumieeeeeeeeeee
ceriiisssoooiicyyy.. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and basclines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We show the details of all the experimental settings in both main paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We follow the common practice in this field, egg., BLEU and ROUGE score, and do not include error bars in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All experiments are conducted using 8 GPUs. We have included the information in our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps://nets http///eerrip..ee httpr//s......ublleecchccccaaaaa
Answer:
[Yes]
Justification:
All research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: It's discussed in the end of each section and the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks. All data and LLM checkpoint are public available. We also provide every details of the used LLMs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets are properly cited and credited.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww com/datasess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve such aspects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve such aspects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
One of the core contributions in this research leverages publicly available LLMs to generate pseudo glosses based on open-source benchmarks. All necessary details and explanations are thoroughly presented our paper.
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy (https: / //neuripss... wwtpeiipp.cc/Coooo for what should or should be described.
21

---

Appendix
A Training Details
A.1 Dataset Preparation
"role": "user",
"content": "<Example: [Text, Gloss), .... [Text, Gloss]>, <Instruction>> <Query Text>"
"role": "assistant",
"content": "de..M-Geerrtted Pseudo Glosss"
Figure 3:
Example prompt used for pscudo gloss generation with Gemini 1.5 Pro [49]. The prompt contains few example text-gloss pairs to guide the LLM in generating well-structured glosses for the query text. Detailed prompt formatting be found in Figure 4 and Figure 6.
Phoenix 14T. Figure presents examples of prompts used to generate pseudo glosses with an LLM. The content under the user role serves as the input to the LLM, while the content under the assistant role represents the I.LM's output.
In practice, we randomly select 30 true glosses from the Phoemix14T training set to create example text-gloss pairs. The guantitative results presented in Table 6a of the main paper. Figure illustrates the prompt without text-gloss pairs, corresponding to the second row of Table 6a.
Prompt (w/ example pairs) and LLM's answer (sbown in bold)
LLM Inpor:
"role": "user".
"contont":
German German gloss examples:
Example text: [und wettervorherage morgen donnerstag
august] gloss: (ETTZ
WETTER MORGEN DONNERSTAG ZWOELF FEBRUAR]
Example text: [mancherorts regnet es such länger ergicbig auch lokale überschwemmungen wieder
HEBERSC möglichl gloss: (ORT REGEN DURCH REGEN KOENNEN UEBERSCHW EMMUNG KOENNENI
These examples demonstrate transformation natural based on the benchmark for sign language translation in German weather forecasts.
structured gloss, specifically
The ourput gloss should be optimized for sign langunge video translation by considering which words (e.g.. nouns, adjectives, adverbs, numbers) are crucial the context weather forecasting.
The gloss should:
Exclude function
articles
and auxiliary unless necessary.
Preserve key content words such weather conditions, time expressions, and geographic references.
Follow the typical structure of German sign language glosses video.
Now, based on examples provided above, please infer the most suitable German gloss sequence the following German text:
donnerstag
regenwolken allmühhich richtung süddosten aus]
The ourput should follow the following format without any text formatting: Output: the generated gloss.""
LLM Output:
"role": "assistant"
"content": "DONNNRSTAGG REGEN WOLKE LANGSAM KOMMEN SUEDOST]'
Figure 4:
The prompt sent to the LLM for generating pseudo glosses includes two example pairs. These two examples serve as references to guide the LLM in producing accurate pseudo glosses during generation. The text marked in red represents spoken language from the datasets, which should be replaced during each iteration.
22

---

How2Sign. How2Sign [16] is an ASL benchmark without gloss annotations, we randomly sample 20 glosses from the 2M-Flores-ASL dataset form the example pairs. The constructed prompt and representative text-gloss pairs for How2Sign are illustrated in Figure 6.
Table 8 presents examples of the final pseudo glosses for the How2Sign dataset. In cases where the generated and reordered gloss annotations exceed the total sequential length of the video-making the application of CTC loss infeasible- remove redundant words from the generated glosses. The list of such words is provided below.
Prompt (w/ example pains) and L.L.'s answer (shown bold)
LLM Inpur:
"role": "user",
"content":
: 1 Your task is to generate German Sign Language gloss based on the given German text. Your gloss generation should align with the standard glossng conventions used in Gemman Language, particularly in the context of weather forecusting. The output gloss should be optimized for sign language video translation by considering which words (c.g.. nouns, adjectives, adverbs, numbers) crucial the context of weather forecasting.
gloss should:
All numbers must he written in German words (e.g. "eins", "rwei", "drei").
Exclude function words like articles ( l'der, die, das and auxiliary verbs unless necessary.
Preserve content words such weather conditions, time expressions, and geographic references.
Follow the typical structure of German sign langunge glosses for video.
text: Jab donnersiag breiten sich regenwolken allmähhich weiter richtung südosten aus
The ourput should follow the following formar without any text formarting: Output: the generated gloss." "**
LLM Output:
"assistant"
"content": "[DONNERSTAG REGENWOLKEN ALLMÁHLICH SÜDOSTEN AUSBREITENJ
The prompt sent to the LLM for generating pscudo glosses without providing example pairs. The text marked in red represents spoken language from the datasets, which should be replaced during each iteration.
Figure 5:
Discussion choosing text-gloss pairs. this paper, we directly use random selection choose the text-gloss pairs. We conducted experiments comparing text-gloss vs. gloss-text pair formats in prompt, e.g., "Examplel: text: gloss: [ (as shown Figure 4) vs. "Example 1: gloss: [ text: []". For generated gloss quality, using text-gloss pairs prompts yields ascudo-glosses closer to true glosses (with better BLEUI and BLEU4 scores) than gloss-text prompts. showing that prompt engincering is critical factor in optimizing performance. We leave other selecting approaches for future study; selecting better text-gloss example pairs may ensure greater diversity and reduce bias.
A.2 Greedy Reordering Strategy
Here, we detail our Greedy Reordering Algorithm for LLM Gloss, outlined in Sec. 3.3 and illustrated Algorithm This strategy aims to optimally align the LLM Gloss with the temporal structure suggested by the Ref Gloss. The process starts by initializing as target, with the same length as the LLM Gloss. Two pointers, i and j. are set to 0, representing the current positions in LLM Gloss and Ref Gloss, respectively. At each step. words from both sequences: 1 from the LLM Gloss and ry from the Ref Gloss. 4 = rj, we append 1; target and increment both pointers (i < 1, < J 1). If I, + ry and does not exist in the reference gloss, we directly append I, to target and move the LLM pointer forward (i + i + 1). In append
empty sequence, denoted
compare the current
the final case, if 1, # T; b is found at later position in the LLM Gloss, denoted as ly., rj to largel, remove the occurrence of l from the LLM Gloss, and increment the reference pointer (+j+ + 1). This process continues until all elements in the LLM Gloss are processed, resulting in reordered version that aligns more accurately with the temporal structure suggested by Ref Gloss.
23

---

Example Prompt for How2Sign dataset
"Pronntt Content":
Your task is to generate an American Sign Language (ASL) gloss based on the given English sentence.
Glossing Guidelines: Preserve key content words, such nouns, adjectives, verbs, adverbs, and numbers, while omitting function words like articles ("a, the") and auxiliary verbs unless necessary.
Convert numbeers into fingerpelllngg friendly formats (e.g.. "21" should be "TWO ONE"). Spell out names and place names using ASI. fingerspelling conventions, unless estahlished ASI. sign exists. Use ASI, gloes abbreviations where applicable (e.g., 'IX" for pronouns).
Examples:
sentence: power failure following routine fire-commaadd system lest caused relief valves open and crude oil overllowed the Fort Greell pamp station 9.] gloss: [ELECTRICTY lights go out AFTER DAILY FIREMAN MAN CONTROL SYSTEM TEST CAUSE V-A-L-V-E OPEN, ROUGH OIL OVERFLOW AREA NEAR F-O-R-T G-R-E-E-LLY P-U-M-P S-T-A-T-L-O-N NINE.]
vents were still leaking prohably from expansion inside tank]] gloss: [WEDNESDAY AFTERNOON, T-A-N-K V-E-N-T-S STILL LEAK POSSIBLE CAUSE T-H-E-R-M-AALL EXPANSION INSIDE T-A-N-K.I
sentence:
[As Wednesday aftemoon,
[Another secondary containment area helow the tanks capable of holding harrels was not yet filled to capacity. gloss: [ANOTHER SECONDARY LIMITED AREA BELOW T-A-N-K-S CAN HOLD MAXIMUM TWO BARRELS NOT-YET MAXIMUM FULL]
sentence:
Now, infer the most suitable ASL gloss sequence for the following two English sentenees:
sentence [query
sentenec2: [query2]
sentence:: [query3]
The ouipul should be a single line conlaining the generated gloss without any additionnll text formalting:
[generated gloss
Igenerated gloss]]
Igenerated "loss3).""
Figure 6: Example prompt that we feed to Gemini 1.5 Pro [49] for pseudo gloss generation How2Sign [16].
Text
this is di bowler and this has been how tell you have low self esteem a
Text
so you know i had to just light thnough the cleaning and had to huve lock smith come out and open my doors and hod to go anollier jub fight through clean must locksmith open door must another go
Gloss
ix know
Text
again we'se talking about cross stitc cross stitch craft kits repeat discuss cruss sttcheross stitch crafl kit
Gloss
Text
uniquue headstooes custerniee your yard me o le g gey y vavin in this clip show how use unique headstone custornize yard
grey
this
you
Gloss
Text
what she's 's displaving the angry colors
she show angry color
Text
try it with a little tempo try little
Gloss
Text
and most the time you can see that you look hopefally you can pick this up here but you Il see high speed it usually be stamped right on it time can see look hope you see high speed usually stamp right there
Gless
Table 8: Generated pseudo glosses for How2Sign (punctuation marks are omitted).
Remove words = { "the", "a", "an", "on", "in", "to", "of", "at", "by", "this", "that", "and", "but", "or", "so", "because", "for". "with", "about", "as", "if", "then", "just", "like", "very", "really", "actually", "alright", "well", "now", "there", "here", "we", "you", "my", "your", "our", "us". "it", "its", "they", "them", "their", "is", "are", "was", "were", "be", "been", "being", "do", "does", "did", "done", "can", "could", "will", "would", "shall", "should", "might", "must", "let", "make", "get")
Table 9 summarizes the number of generated pseudo glosses based on the approach detailed in Sec. 3.
Datasct
# Vocab
Pseudo Gloss Č
Phoemix147 How2Sign
2,887
1.120
15,541
9,796
Table Vocabulary size of text vs. pseudo glosses.
24

---

A.3 Architecture Overview
Our framework follows the gloss-based architecture MMTL [10]. The visual encoder øy consists of S3D backbone [54] and head network. Given input video V X2x222x222x3 only the first four blocks of S3D are used, producing features Z E RT/44832 after spatial pooling. These features are then processed by a head network, comprising a projection block (temporal linear layer, BN, and ReLU), followed by temporal convolutional block (two convolutional layers with kernel size 3, a linear layer, and ReLU). The final output is in the shape of RT/1x512 In Stage I and Stage 3.1, 1, a linear classifier and Softmax are applied to Z, resulting in frame-level gloss probabilities P TR. T/4X/K where denotes the gloss vocabulary size. In Stage 3.1, we introduce an additional randomly initialized projector, consisting of two fully connected layers. For the translation network (referred as the "Translator" in Figure 1), we adopt mBART [41], a sequence-tor sequence-to-sequees -o-enneeee--e denoising autoencoder, and Gemma-2 [50], decoder-only architecture. Both models are initialized with pretrained weights from large-scale multilingual corpora, providing a strong backbone for our translation process. Table 10 provides comprehensive breakdown of the number of trainable parameters for each component of our SLT model.
Component
# Params
# Trainable
Component
Params
# Trainable
Visual Encoder
4,734.272
4,695,936
Visual Encoder
4.734,272
4,695,936
Visual Head
8,280,906
8,280,906
Visual Head
8,276,868
8,276,868
Projector
1,574.912
1,574,912
Projector
6,492.672
6,492,672
Translater (mBART)
358,461,888
358,461,880 ,888
Translater (Gemma2)
2,045,709,312
12,779,520
Total
373,013,642
Total
32,244,996
(a) Model parameters of mBART based translator.
(b) Model parameters of Gemma2-2B based translator.
Table 10: Training settings including (a) the number of pseudo-glosses during pre-training and (b)
parameter counts during downstream translation.
A.4 Training Configurations
We report our training configurations for our PGG-SLT Table 11.
Parameter
Stage 1
Stage
Stage 3.1
Stage 3.2
Model Attention drop
S3D
mBART
S3D mBART
S3D & mBART
0.1
Dropout
0.3
Label smoothing
0.2
0.2
0.2
Number of beams
5
Video size (T, C. H. W)
(T, 3. 224, 224) X [0.5, 1.5]
(I, 224, 224)
(T, 3, 224, 224) /
Horizontal Flip
X
Temporal Sampling Range
[0.5, 1.5]
[0.7. 1.3)
[0.7, 1.3)
Optimizer
Adam W 0.9/0.999
AdamW
AdamW
AdamW
Optimizer momentum (3, /32)
0.9/0.999
0.9/0.999
0.9/0.999
Learning rate schedule
Cosine
Cosine
Cosine
Cosine 0.05
Weight decay
0.05
0.05
0.05
Warmup epochs
5
0 40
Epochs
40
40
Batch size
8
8 x 10-5
Peak learning rate
5 x 10-3
5 x 10-5
1 x 10-4
Minimum learning rate
1 x 10-7
x 10-7
1 x 10-7
1 1 10-7
Table 11: Training settings on Phoenix 14T.
A.5 Connectionist Temporal Classification (CTC) Loss Formulation
The goal of the Sign2Gloss stage, also known as Sign Language Recognition (SLR), is to predict the gloss sequence G = (0), , ..., ..., 9M) from the sign video 1/ containing 7 frames. We employ the CTC loss [19], which allows for sequence-level supervision without requiring precise frame-to- gloss alignments. Specifically, CTC computes the probability of by summing over all feasible
25

---

alignments:
(( Σ D) TES
(11)
where x denotes frame-level gloss path, and S is the set of all valid mappings between the visual frames and the gloss sequence. The probability p(r[V) is obtained by applying Softmax activation
to the output of the visual encoder gvv. The CTC loss is then defined as the negative log-likelihood:
LSl
log p (9VV;
(12)
In the Gloss2Text stage, the recognized gloss sequence G is translated into the target spoken language sentence ($1,..., SU) using a neural translation model thr. This is achieved through sequence- to-sequence learning objective:
SSL Σ
log 1ogp(siss<,, G:
(13)
where p< 9) represents the probability of generating the i-th word given its preceding words and the gloss sequence.
A.6 More Results on CSL-Daily.
Dev Set BLEU4
Test Set
Method
BLEU1
ROUGE
BLEUI
BLEU4
ROUGE
TS-SLT [11]
58.24
29.18
57.81
58.64
29.55
58.62
GFSLT-VLP" [59]
39.20
11.07
36.70
39.37
11.00
36.44
Sign2GPT [53]
41.75
15.40
42.36
Lig [18]
42.45
12.23
39.18
39.55
15.75
39.91
VAP [28]
53.31 53.58
23.84
$1.19
52.98
23.65
51.09
PGG-SLT" (mBART)
24.05
52.34
53.29
23.70
52.88
Table 12: Comparison with SOTA methods on CSL-Daily [60].
iiniicates the Gloss-free method.
A.7 Qualitative Results
Table 13 and Table 16 provide qualitative examples of our mBART-based PGG-SLT model on the Phoenix 14T and How2Sign datasets, respectively. The reordering of gloss annotations is illustrated in Table 14. We also compare our approach with the best-performing models, including Tarr'es er al. [48], Uthus et al. [51]. SSVP-SLT [45], and the reference translations on How2Sign. As shown in Table 15, although our method achieves a slightly lower BLEU4 score compared to SSVP-SLT, the generated predictions remain highly relevant and on-topic, comparable to SSVP-SLT. Notably, all current models still face challenges with repetitions and sign mix-ups in various instances.
Val Set
Reference
sonst regner tellweise kriftipp (Otherwise mins heavlyy times) sonst regner teilweise krättig. (Otherwise rains heavily times.)
Prodiction
Reference
tag wechseln
und walken einander teilweise with each
längere
Val
Prediction
am tag some und wolken im wechsel gehiessweise zeigt sich die sonne für längere zei.
(During day sun clouds
it sanny longer period time.)
(During the day sun and clouds altomate is soms treas the sun appears for longer period of time.)
Reference
das boch high- presure system brings predominanlly plessant weather rising temperatures umill least Friday)
bis mindestens karfreinag überwiegend froundliches wester und steigende temperaluren.
Val (3)
Prodiction
his freitgg wird unter hwwhkdruckeiiffee langsom wieder freundlicher und much tempenituen steigen winder.
(By Friday under high pressure influence it will slowly become more pleasant again and the temperatures will also rise again.)
Reference
am dremerstag regen der nordhalfti in der sidfhilhe mal soone mal wolken thhlichee wetter dann auch am froitng. Thunday
Test
Prodiction
donnersag rognet
the northem half southem half sometimes wechsel feiigg clouds similar weather then also Friday.)
(On Thurslay it rains the coests otherwise
clouds alternate Friday changeable weather.)
Test Set (3)
Reference Prediction
teilweise ist auch klaz. (Partly is also clear.)
trilweise klart such auf. (Punlyy cllurs
Table 13: Phoenix 14T qualitative results.
26

---

()) Text (S)
Begen abend rieht von westen wieder schnce heran der sich am sonntag weiter aushreitet. (Toward evening moves again from the west which will spread further on Sunday.)
True Gloss (9)
IX ABEND/ / KOMMEN/ WIEDER / SCHNEE/ / SCHNEIEN / /SONNTAG/ / KOMMEN (ix eveaing come again snow snowing / suuday / comne)
LLM Gloss (Gv)
ABEND/ WEST / SCHNEE/ / KOMMEN / SONNTAG/ KOMMEN (evening /est// snow come / sunday / (oome)
Frame wise Gloss
ABEND/ ABENDD/ WEST KOMMEN KOMMEN/ WEST SCHNEE/ SCHNEE// / SCHNEE/ /
SONNTAG / /DEUTSCHLAN / REGION/ / KOMMEN / KOMMEN / REGION / REGION
Ref Gloss (ev
ABEND/ / WEST/ / KOMMEN/ / WEST /
SCHNEE/ / SONNTAG REGION / KOMMEN / REGION ABEND/ / WEST / KOMMEN SCHNEE/ / SONNTAG/ / KOMMEN (evening / /west/ come / snow I
Reardered Gloss (Gy.ttrrr .target)
sunday come)
(2) Text (S)
fouchte kalte luft strömt zu uns nach deutschland. (Moist cold air flows to us into germany.)
True Gloss (g)
TROCKEN/ / KALT LUFT// / KOMMEN/ / DEUTSCHLAND (dry cold / air / come / germany)
LLM Gloss (Gy)
/ALT LUPT / DEUTSCHLAND /
KOMMEN (cold / air germany / come)
Frame-wise Gloss
KALT KALT/ / KALT LUFF / /UUFT / KOMMEN / KOMMEN / KOMMEN/ DEUTSCHLAND/
DEUTSCHLAND
Ref Gloss (GV re
KALT/ / LUFT/ KOMMEN/ DEUTSCHLLAND LNND
Reordered Gloss (9v.targe)
KALT/ / LUFT/ KOMMEN/ / DEUTSCHLAND (cold/ / air I come / germany)
(3) Text (S)
der donnerstag begimat oll Ireundlich später ziehi
regen heran. (Thursday begins oflen
friendly later rain moves in from the west.)
True Gloss (i)
DONNERSTAG/ FREUNDLICH SONNE/ /ANNN/ / SPAETER KOMMEN/ REGEN (thursday / fricadly sun / then / later / come rain)
I.I.M Gloss (Gv)
DONNERSTAG/ / FREUNDLICH SPÄTER WEST / REGEN/ / KOMMEN (thurdaay / friendly / later Twsst/ rain / come)
Frame wise Gloss
DONNERSTAG/ / FREUNDLICH WENDDICC/SPPAA SPÄTER / SPÄTER / /WEST/ / KOMMEN / REGEN / REGEN/ /
REGEN / NUR
Ref Gloss rev.
DONNERSTAG/ FREUNDLICH/ SPÄTER/ / WEST KOMMEN / RRGGN NUR
Reardered Gloss (Varggge))
DONNERSTAG/ / FREUNDLICH / SPÄTER WEST / KOMMEN REGEN (thrrday/ friendly later [ /west/ come / rain)
(4) Text (S)
und die wettervothersagee für morgen samstag den fünfzehnton august. (And now the weather Forecasi for lomorrow, Saturday the filtennth of Augusi.)
True Gloss (G)
JETZT / /WTE AUSSEHEN WETTTER MORGEN / SAMSTAG / /LUNNZZEN AUGUST (now / how-lock/ / weather/ / tomorrow / saturday/ / fifteen / angust)
LLM Gloss (Gvv
MORGEN WETTER / SAMSTAG / FÜNFZEIN AUGUST (Lomortww weather / saturday /
fifieenth / angust)
Frame-wise Gloss
WETTER WETTER/ / MORGEN MORGEN/ / MORGEN/ / MORGEN/ SAMSTAG/ WETTER/ /
WETTER/ WETTER/ WETTER/ AUGUST
Ref Gloss (vyref)) Reordered Gloss (vvtaareet)
WETTER /MORGEN/ / SAMSTAG/ / WETTER / AUGUST
WETTER / MORGEN SAMSTAG / FÜNFZEHN AUGUST (weather tomorrow / saturday /
1ifteenth / august)
(5) (S)
der beutige zwischen hitze im uslen und abkühlung im westen. (Todaa is a day between heat in the east and cooling
west.)
True Gloss (C)
HEISS/ HIER / OST/ / REGION/ / KUEHL (hot here/ / cast / region/ / cool)
LLM Gloss (Gr)
HEUTE/ / OST / HIESSS / /WEST/ / KALT (today / east / bot west/ / cold)
Frame-wise wise Gloss
AUCH/ HEISS HEISS /ST /ST HELTF. HEUTE// WEST/ WEST / KALT/ /KLLT /KOMENN
Ref Gloss (Sv.re))
HEISS/ / /ST/ HEUTE/ WEST/ KALT
Reurdered Gloss (Vytegget
HEISS/ OST/ HEUTE/ WEST KALT (hot / east/ today west/ / cold)
(6) Text (S)
am montag sonne teils wolken und einzelne gewitterschauer. (On Monday partly sunny partly cloudy and isolated thunderstorms.)
True Gloss (g)
MONTAG/ / SONNE/ / WOLKE/ WECHSELHAFT KOENNNNNNNNNNNNNNNNN / GEWITTER (monday sun / cloud chamgeeblle///// / (hhudderstorn)
LLM Gloss (Gv)
MONTAG TEILL SONNE/ / TEILS/ / WOLKE/ / UND/ / GEWITTER/ / KOENNEN (monday / party/ / sun partly / cloud and thunderstorm/ / can)
Frame wise Gloss
MONTAG/ MONTAG SONNE, UND WOLKE/ / WOLKE/ WOLKE/ AUCHI/ / KOENNEN/ / KOENNEN/ / GEWITTER GEWITTER / GEWITTER
Rel Gloss (ey)
M / SONNE/ / UND/ / WOLKE/ KOENNEN / GEWITTER MONTAG/ TEILS/ SONNE// TEILS UND/ WOLKE/ KOENNEN GEWITTER (monday / party/ / sun partly/ and cloud/ / can / thunderstorm)
Reardered Gloss (Cy. )argeet
(7) Text (S)
ärtlich muss mit bodenfrost gerechnet werden. (Locally ground frost must be expected.)
True Gloss (g)
OKT / MOEGLICH / BODEN / /BOST splace possible / ground/ / frest)
LLM Gloss (Gv)
ORT / FROST BODEN (place/ / frost / ground)
Framc-wise Gloss
ORT/ KOENNEN/ BODEN FRROST/ KOENNEN
Ret Gloss ( reff)))
ORT/ BODEN/ FROST
Reordered Gloss (vvtargee) ORT / /BODEN///// / FROST (place / ground / frost)
Table 14: More examples of LLM generated pscudo gloss and corresponding reordering operation on Phoenix14T.
27

---

(1) Reference
And that's great vital point technique for women's 's self defense.
Tarrés al. [48]
It's really great point for women's 's self defense.
Uthus [51]
It's really great for women's 's self defense.
SSVP-SLT [45]
This is a really great point for women's 's self defense.
PGG-SLT (ours)
These are great things for women's 's self defense.
Reference
In this clip I'm going to show you how to tape your cables down. In this clip I' I'm going to show you how to improve push ups. In this clip we're going to show you how cut piece of clay.
(2) Tarrés et al. [48]
Uthus al. [51]
SSVP-SIT [45]
In this clip I'm going show you how to clip the cable, the cable. In this clip I'm going to show you how to name the clipper wire.
PGG-SLT (ours)
(3) Reference
this segment we're going talk about how to load your still for distillation lavender essential oil.
Tarrés et al. [48]
Ok, in this clip, we're going to talk about how to fold the ink for the lid of the oil.
Uthus al. [51]
In this clip we're going to talk about how to feed a set of baiting lizards for a lava field oil.
SSVP-SLT [45]
In this clip we' re going to talk about how to feed the trail for draining clean for laborer oil.
PGG-SLT (ours)
In this clip we're going talk about how to smooth out the sidewall for the dust wash for the lavatory oil.
(4)
Reference
You dancing, and now you going need the veil and you going just grab the veil as far as possible.
Tarrés et al. [48]
So, once you'se 're helly dancing, once you' 've got to have the strap, you' 're going to need to grah the thumb, and try to avoid it.
Uthus al. [51]
Their hopping and dancing is now, they're going need their squat and squat and they's re going to able to move independently.
SSVP-SLT [45]
So that she's going to get her hips up as far as she can, and now she's going to lift her head up as far as possible.
PGG-SLT (ours)
Your belly dancing right now should be going up and down as far as you can.
(5)
Reference
But if you have to setup new campfire. there's two ways to do it in a very low impact;
is with mound fire, which should in the campfire segment carlier and the other way to setup low impact campfire is to have a fire pan, which is just a steel pan like the top of a trash can.
Tarrés et al. [48]
And other thing I'm going to talk to you is little bit more space, a space that's what it's going to do, it's kind of quick, and then don't 't want to take spray skirt off, and then I don't want it take it the top of it.
Uthus et al. [51]
But if you have to set up a new campfire, there are two ways to do low impact fire, one is a cone fire, which we have to do in the tent earlier. and the other one is to set up a campfire in fire pan.
SSVP-SLT [45]
But if you have set up One is monk fire. One a campfire. The other one is to campfire in a campfire. That's 's just a post like the top of the post.
campfire, this is one way to do in low impact.
PGG-SLT (ours)
But if we'r re going to set up new campfire, there's 's two ways to do a low impact fire which is marking fire which we should do in the center of the campfire, and another is just to
fire pan, and that's just the top of the pan.
(6)
Reference
So, this is very important part of the process.
Tarrés er al. [48]
It's a very important part of the process.
Uthus et al. [51]
Alright, let's get started.
SSVP-SLT [45]
It's an important part of the pracess.
PGG-SIT (ours)
This is good part of the process.
Table 15: Qualitative translation examples from our mBART based version compared to Tarrés et al. [48], Uthus et al. [51], SSVP-SLT [45], and the reference translations on How2Sign [16]. The examples were picked from the How2Sign test set by [48]. Although the BLEU4 score of our method is slightly lower than that of SSVP-SLT, our model's 'ss predictions remain largely on-topic, similar to SSVP-SLT. Moreover, all current models can struggle with repetitions and sign mix-ups in many cases.
28

---

Reference
Beautiful.
Prediction
Beautiful.
Reference
One more time.
Prediction
One more time.
(3)
Reference
Inhaling all the way up.
Prediction
And he breathes all the way up.
(4)
Reference
Drag that energy down to the heart.
Prediction
Draw the energy down the heart.
(5)
Reference
So I'm shuffling this deck at the start of this segment because instead of laying out bad hands, Jacks to Open, I'm going to show them you right off the deck.
Prediction
And I'm going to talk about the amount that I'm going to be spending to get started because I'm going to need to get rid of the seeds, I'm going to get rid of the bad soil in the open, and I'm going to show you how to draw it.
(6)
Reference
And so let me show you how to set it up. Maybe should show you how to this up.
Prediction
(7) Reference
I am going to take these parts of my hand right there and I am going to squeeze
the clay and push it up.
Prediction
We're re going to take our bowl and remove some of the clay and we're 'r going to squeeze our bowl up.
(8)
Reference
Make sure that when you are placing your veil, you need to grab your veil like this.
Prediction
So make that when you up your tent you should take hat like this.
(9)
Reference
So again, just practice plunking out notes, singing some scales, and some differ- ent notes, and sooner or later you will have melody. It just takes a little practice and you can get some slims different parts the body, and eventually they can become very slippery.
Prediction
(10)
Reference
We'll have Emily standing about hip width apart. So daniel's going to stand with his feet about hip width apart.
Prediction
(11) Reference
Now I'm ready for my legs. Now we' re ready for the legs.
Prediction
(12)
Reference
I'll start with my upper hody even though my legs are going to he the ones that will really be the focus.
Prediction
We're going to start with our higher body kind of whatever the leg is that's really great.
(13) Reference
you have a queen, and let's say you have ten, it's irrelevant what the third card is 'cause it's already better than a queen-six-four If you're re queen of 10 let's say you're re queen of 3 it's much better than a queen of 4.
Prediction
(14)
Reference
Typically if you have higher than a queen six- four, you will want to stay you'll II want put your bet in the play column. If it's little bit longer than your sleeve, you might want to stay with and that's how you play line up for your sleeve.
Prediction
(15) Reference
it's under that, you want to fold, you will lose your ante. If it's below your comfort level you can wear it while you' re dancing.
Prediction
(16) Reference
you have a queen, but your two cards are less then the dealers, you will lose and the dealer will take both of your bets. If you'le re queen, you can only have two cards, you have to either lose your opponent's 's hand you have the rhythm of losing your opponent's hand.
Prediction
(17)
Reference Prediction
Again want make faces.
And again you just want make sure you're putting it right on the face.
Table 16: More qualitative translation examples from our mBART based version on How2Sign [16].
29


Reviews:
[{"review_id": "RCe6MESzpl", "reviewer": "Reviewer_exiF", "summary": "This paper addresses the challenge of Sign Language Translation (SLT), which converts sign language videos into spoken language text. Traditional SLT methods rely on costly, expert-annotated glosses as intermediate supervision, limiting scalability. To handle this, PGG-SLT is proposed in this paper, a gloss-free framework that leverages Large Language Models (LLMs) to generate pseudo glosses from spoken language text. These pseudo glosses are refined via a weakly supervised reordering mechanism to better align with sign video sequences, including three training stages, video-to-pseudo-gloss recognition, pseudo-gloss-to-text translation, and end-to-end fine-tuning. PGG-SLT provides a scalable, annotation-efficient solution for SLT by harnessing LLMs for pseudo gloss generation and weakly supervised alignment, achieving competitive accuracy without gloss supervision. The framework demonstrates strong generalization across languages and datasets.", "questions": "1.\tHow do LLM-generated pseudo-glosses capture sign language-specific linguistic features, such as spatial verbs, non-manual markers, that diverge fundamentally from spoken language syntax? Quantitative analysis comparing pseudo-glosses with ground truth for linguistic features, including missing classifiers, incorrect verbs, is necessary.\n2.\tDoes the method maintain performance on sign languages with radically different grammar, such as Chinese and Japanese, or noisy real-world videos? More tests and evaluations could be implemented on datasets based on different languages, like CSL.\n3.\tA few illustrative figures are contained in this paper, and the existing figures fail to effectively convey the core idea and explain the algorithm.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n+A Gloss-Free method is proposed to bypass the costly need for expert-annotated glosses, leveraging LLMs to generate pseudo glosses from spoken language text, which addresses a critical scalability bottleneck in SLT.\n+A weakly supervised video-guided refinement process is introduced, using multi-label classification and temporal smoothing, to align LLM-generated pseudo glosses with sign sequences, enabling effective CTC loss supervision.\n+The results outperform gloss-free methods on Phoenix14T, specifically, BLEU4 +5.0 over GFSLT-VLP, and +6.1 BLEU4 over SSVP-SLT, closing the gap with gloss-based methods.\nWeaknesses\n-LLM-generated glosses inherit spoken language syntax, failing to fully capture sign language-specific grammar. Reordering mitigates but does not eliminate misalignment. \n-Performance highly depends on LLM capabilities, but smaller LLMs might underperform larger counterparts.\n-Experiments are limited to German (Phoenix14T) and English (How2Sign) sign languages. Robustness regarding non-Western or low-resource sign languages is untested.", "was_revised": true, "final_justification": "The authors' response has solved most of my concerns. Based on the response and other reviewers' comments, I would raise my rating to Borderline accept.", "rebuttal": "We sincerely thank the reviewer for providing valuable comments. We respond to your questions and provide clarifications as follows: \n\n>Q1.1 How do LLM-generated pseudo-glosses capture sign language-specific linguistic features, such as spatial verbs, non-manual markers, that diverge fundamentally from spoken language syntax? & W1 LLM-generated glosses inherit spoken language syntax, failing to fully capture sign language-specific grammar.\n\nThanks for your comments. We acknowledge that our current LLM is not yet capable of generating glosses that include non-manual markers or classifier annotations. For example, some glosses in the 2M-Flores-ASL dataset use \"+++\" to indicate repeated actions and \"cl:\" to denote specific handshape classifiers, as well as markers like / /WHAT\\ \\ and / /WHY\\ \\ to represent NMMs (with / / and \\ \\ indicating their span). We think this task is quite difficult without access to video information.\n\nIn our study, most glossing requires only a single word to accurately describe what is being signed. The proposed LLM-generated glosses perform well in handling the unique grammatical structure of sign languages, which often differ significantly from spoken language word order, while effectively capturing some sign-specific patterns and filtering out spoken language artifacts. This leads to a more robust and scalable pipeline for automated sign language processing. We believe that, with sufficient text-gloss examples, LLMs can learn to produce such features through in-context learning, functioning as a dynamic \"lookup table\" that is continually refined by contextual cues. Our current focus is for the LLM to capture two key aspects: (1) differences in lexical grammar and (2) divergent expressive patterns. For example, the spoken phrase \"In the north and west sometimes sun sometimes clouds\" is glossed in sign language as \"northwest / changeable / sun / cloud\"—using \"changeable\" to condense the spoken meaning, while omitting spoken-specific prepositions. The LLM can filter out such spoken language elements, including prepositions and other structures tied to spoken language that are irrelevant to sign glosses. We show that a small set of examples explicitly containing these features allows LLM to infer unique sign language structures without explicit rule engineering. We observe performance improves with more examples, as the model refines its grasp of how sign language syntax diverges from spoken norms.\n\n>Q1.2 Quantitative analysis comparing pseudo-glosses with ground truth for linguistic features, including missing classifiers, incorrect verbs, is necessary.\n\nThank you for your advice. Since the How2Sign dataset does not provide corresponding gloss annotations (including classifier markings), we currently offer only a lexical analysis, which serves as an essential first step toward the deeper linguistic analysis requested. We have conducted quantitative comparisons between the generated pseudo-glosses and the ground truth glosses, with the results summarized as follows:\n\n|Dataset| | |Phoenix14T| | | CSL-Daily |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|        | Overall | Noun | Numerical | Adjective | Verb | Overall |\n| Precision| 58.2 | 63.8 | 43.2       | 61.2    | 55.9    | 72.4  |\n| Recall     | 63.5 | 66.9 | 45.1       | 63.5    | 59.1    | 82.8  |\n\nFor computational convenience, we first processed both the generated glosses and ground truth by removing duplicate words, then calculated the precision and recall between each generated gloss and its corresponding ground truth gloss (denoted as overall). Additionally, we used the spaCy natural language processing library to perform lemmatization and categorize words into several part-of-speech classes, and further analyzed the generation quality of several key categories.\n\nAs shown in the table above,  our method achieves quite strong precision and recall, especially for key content words such as nouns. We will include the full analysis in the appendix. We acknowledge that this evaluation is conducted at the lexical level. A more in-depth analysis addressing missing classifiers or incorrect verb types, as you suggest, represents a valuable direction for future linguistic evaluation of sign language translation systems.\n\n> Q2. Does the method maintain performance on sign languages with radically different grammar, such as Chinese and Japanese, or noisy real-world videos? More tests and evaluations could be implemented on datasets based on different languages, like CSL.\n\nWe further evaluate our method (mBART-based version) on CSL-Daily dataset, * means the Gloss-free method.\n\n|Model||Dev Set|||Test Set||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||BLEU1|BLEU4|ROUGE|BLEU1|BLEU4|ROUGE|\n|TS-SLT|58.24|29.18|57.81|58.64|29.55|58.62|\n|GFSLT-VLP*|39.20|11.07|36.70|39.37|11.00|36.44|\n|Sign2GPT*|-|-|-|41.75|15.40|42.36|\n|SignLLM*|42.45|12.23|39.18|39.55|15.75|39.91|\n|VAP*|53.31|23.84|51.19|52.98|23.65|51.09|\n|Ours*|53.58|24.05|52.34|53.29|23.70|52.88|\n\n\n>Q3. A few illustrative figures are contained in this paper, and the existing figures fail to effectively convey the core idea and explain the algorithm.\n\nThank you for this feedback. We agree that the figures can be improved. In our revision, we will create a new, comprehensive Figure 1 that visually integrates the entire pipeline: (1) Prompting the LLM with text-gloss pairs, (2) showing the initial LLM-generated gloss, (3) illustrating the video-guided refinement process from Algorithm 1, and (4) showing the final reordered gloss used for training. This will provide a much clearer, end-to-end overview of our core contribution."}, {"review_id": "2xf5dHfbSd", "reviewer": "Reviewer_ujwE", "summary": "This paper aims to alleviate the reliance on costly gloss annotation with the proposed pseudo gloss generation framework. It starts by prompting LLM with a few samples and utilizes weakly-supervised ordering for better alignment. Experiments are conducted on two datasets to validate the effectiveness.", "questions": "The questions are raised in weakness part.", "limitations": "Please discuss the limitations in a section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe idea is clearly presented.\n2.\tThe paper is well-organized.\n3.\tThe experiment results look promising. \n\nWeaknesses:\n\n1.\tThe authors should ensure the rigor of language expression. It should not be claimed as a gloss-free generation framework since it leverages several gloss-text pairs as the prompt of LLMs.\n2.\tIt is suggested to discuss/demonstrate whether the proposed method could be generalizable to more sign languages like CSL.\n3.\tHow do authors choose the gloss-text pairs to ensure more diversity and less bias?\n4.\tFrom Table 6a, it seems that the upper bound of gloss generation is still low but does not have a great impact on the final SLT performance. Could the authors provide more explanation on it?\n5.\tCould the authors analyze the failure cases?\n6.\tSome related SLT pre-training works are suggested to cite, SignBERT+, SHuBERT…", "was_revised": true, "final_justification": "Thank the authors for their detailed rebuttal. It addressed most of my concerns. I confirm to keep my initial score.", "rebuttal": "We sincerely thank the reviewer for providing supportive comments and acknowledging the strengths of our work. We respond to your questions as follows: \n\n>Q1. The authors should ensure the rigor of language expression. It should not be claimed as a gloss-free generation framework since it leverages several gloss-text pairs as the prompt of LLMs.\n\nThank you for your advice. You raise an excellent point regarding the precision of terminology. We agree that the term \"gloss-free\" can be misleading, as our method does leverage a small number of example gloss-text pairs. We will clarify this when introducing our method (e.g., Line 62), and revise relevant sections of the paper to better contextualize our approach. Specifically, for the PHOENIX14T dataset, our approach is not gloss-free, as we use available gloss annotations during training. However, for datasets such as How2Sign, where no gloss annotations are inherently provided, we instead utilize gloss-text examples from other datasets. In this case, we consider our method to be gloss-free with respect to the target dataset. Our original intention was to emphasize that our approach eliminates the need for gloss annotations across the vast majority of the training data, particularly for domains where such annotations are unavailable or costly to obtain. To more accurately reflect this, we will revise the paper to replace \"gloss-free\" with more precise terms such as \"annotation-efficient\" or \"minimally supervised\", as appropriate. Thank you for this important clarification—it will help us better articulate the strengths and limitations of our methodology.\n\n>Q2. It is suggested to discuss/demonstrate whether the proposed method could be generalizable to more sign languages like CSL.\n\nWe further evaluate our method (mBART-based version) on CSL-Daily dataset, * means the Gloss-free method.\n\n|Model||Dev Set|||Test Set||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||BLEU1|BLEU4|ROUGE|BLEU1|BLEU4|ROUGE|\n|TS-SLT|58.24|29.18|57.81|58.64|29.55|58.62|\n|GFSLT-VLP*|39.20|11.07|36.70|39.37|11.00|36.44|\n|Sign2GPT*|-|-|-|41.75|15.40|42.36|\n|SignLLM*|42.45|12.23|39.18|39.55|15.75|39.91|\n|VAP*|53.31|23.84|51.19|52.98|23.65|51.09|\n|Ours*|53.58|24.05|52.34|53.29|23.70|52.88|\n\n>Q3. How do authors choose the gloss-text pairs to ensure more diversity and less bias?\n\nThis is a good question, and one we haven’t yet considered. Currently, we use random selection, which we will explicitly note and discuss in our Future Work section.\nTo date, we have only conducted experiments comparing text-gloss vs. gloss-text pair formats in prompt, e.g., “Example1: text: [] gloss: []” vs. “Example 1: gloss: [] text: []”. For generated gloss quality, using text-gloss pairs as prompts yields pseudo-glosses closer to true glosses (with better BLEU1 and BLEU4 scores) than gloss-text prompts. We agree that prompt engineering is a critical factor in optimizing performance, and we will add a relevant discussion of this point in the paper’s appendix.\n\n>Q4. From Table 6a, it seems that the upper bound of gloss generation is still low but does not have a great impact on the final SLT performance. Could the authors provide more explanation on it?\n\nThank you for your valuable comments. The key insight behind our approach is that even imperfect pseudo-glosses provide a far more informative training signal than having no glosses at all, as they help bridge the modality gap between sign language and spoken language.\nRegarding concerns about the quality ceiling of gloss generation, the 700 pairs of data (last row in Table 6a) represent less than 10% of the entire training set. We believe gloss generation quality will continue to improve with a larger proportion of samples used. However, we have also observed that even when test samples were referenced during the in-context phase, the LLM still failed to generate 100% accurate glosses in some cases. We attribute this to two factors: current LLMs are not specifically trained for such tasks, and glosses inherently have a “misaligned” relationship with spoken language. Additionally, the logical reasoning capabilities of current LLMs are not flawless (e.g., RAG systems cannot achieve 100% accuracy), resulting in certain generation limitations.\n\nAs for the impact on final SLT performance, Table 6b still shows some improvements. Since this paper does not modify the training method of the baseline model, the upper bound of the gloss-free approach remains consistent with previous gloss-based methods.\n\n>Q5. Could the authors analyze the failure cases?\n\nWe present here several typical error cases in How2Sign dataset here:\n\n- Reference：We’ll have Emily standing about hip width apart.\n\n- Prediction：So daniel’s going to stand with his feet about hip width apart. \n\n- Reference：Now I’m ready for my legs.\n\n- Prediction：Now we’re ready for the legs.\n\nThe example pairs we use convert demonstrative pronouns to \"ix\" and render names in finger spelling form (e.g., \"Emily\" becomes \"E-M-I-L-Y\"). However, in practice, some videos may not spell out names via finger spelling. This mismatch leads to obvious referential errors in the model’s outputs.\n\n- Reference: So again, just practice plunking out notes, singing some scales, and some different notes, and sooner or later you will have a melody.\n\n- Prediction: It just takes a little practice and you can get some slims in different parts of the\nbody, and eventually they can become very slippery.\n\nIn this case, the model appears to have missed the core meaning of the input. As a result, it resolved the uncertain or ambiguous signs into semantically unrelated or nonsensical outputs. In How2Sign, there are also more complex scenarios, typically involving lengthy videos and texts—many of which result in translation errors. This suggests that current models may have limitations when handling longer text inputs and extended video frames.\n\n>Q6. Some related SLT pre-training works are suggested to cite, SignBERT+, SHuBERT…\n\nThank you for pointing this out. We will incorporate discussions of these papers into Section 2 (Related Work):\n\nIn addition, another research direction focuses on enhancing SLT and fingerspelling model performance through pre-training. SignBERT first proposed a self-supervised pre-training followed by downstream-task fine-tuning framework. Building on this, SignBERT+ introduced a self-supervised pre-training approach that models hand poses as visual tokens using multi-level masked modeling strategies and incorporates model-aware hand priors to capture hierarchical context. Meanwhile, SHuBERT adapted masked token prediction objectives to multi-stream visual sign language inputs, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams."}, {"review_id": "MnQtE0Ktea", "reviewer": "Reviewer_UNMc", "summary": "This work explores generation of pseudo-glosses to support sign language translation in settings without gloss annotation, using LLMs for pseudo gloss generation", "questions": "Questions:\n- The description of stage 2: gloss2sign is slightly confusing, unclear why we are going back to sign – could you clarify?\n- German and English have different performance for mBART and Gemma, how do those impact corresponding glossing?\n- Which sentences / example pairs are used to train the LLMs? How does this impact the performance? \n\nSuggestions:\n- Improved discussion of glossing (as suggested above)", "limitations": "See above related to glossing.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "As data annotation (glossing) for sign languages is fairly laborious task, the gloss-free sign language translation task explored here is very valuable, particularly for lower resource settings. Notably their results show performance on par with gloss-based approaches. Their pseudo gloss-reordering approach is interesting, the authors conduct a comprehensive set of experiments and the work is well written. \n\nMy main concerns for this work are related to some of the nuances with claims made and framing, particularly related to how glosses are defined and described.  In the introduction, glossing is framed as something specific to sign languages – but in fact, glosses are an intermediary representation used in linguistics when considering translations between languages. Therefore, glosses are not a written form of sign languages (there do exist sign language writing systems, not adopted widely).\nAdditionally, many of the issues described about the different syntax and ordering between signed and spoken languages are pretty much translation aspects applicable to translation between all languages (e.g., between two spoken languages).  I think accurately framing the problem is necessary to evaluate claims but also to situate the work correctly amongst other related research areas. \n\nGlossing for sign language is typically done with reference to sign language, so as to correctly mark the signs present in the sentence. In this work, they instead use the  english translations to approximate the glosses present in the signed sentence. My main concern with this approach is that it seems that the expressiveness of sign language is greatly reduced in the representations. For example, the same English sentence may be signed many different ways in ASL – using different signs, perhaps incorporating classifiers, 3D space, idioms and other unique linguistic features. Using the english translation for glossing then would fail to capture any of this linguistic diversity (e.g., consider many ASL idioms like TOUCH-FINISH, TRAIN-GONE, that are not represented in corresponding english translation at all). Should these learned models and representations ever be used to in sign language generation, the produced sign language would be very homogenized, and bring in spoken language bias.\n\nI am also additionally skeptical of the claim that LLMs are surrogate experts proficient in sign languages – what is the basis of this claim? While LLMs might be good at tasks of “filtering out unimportant words from text” that is not glossing which linguists do on while referring to the sign language video alone. The results are further used to claim that the generated pseudo glosses approximate real gloss annotations – but the experiments do not substantiate this claim. It would require a comparison of generated pseudo glosses to existing glosses. Perhaps a more accurate claim would be that pseudo glosses have comparable value in supporting sign language translation. I believe that the distinction is important, given that glosses have use across different languages SLRGT and linguistic tasks. \n\nOverall, I do think that this work is valuable – I would love to see a more accurate discussion of glossing, and limitations of this gloss-free / pseudo-glossing approach (e.g., linguistic limitations, spoken language bias etc).", "was_revised": true, "final_justification": "I recommend Accept. The proposed changes (additions to the limitations and future work section, a more nuanced discussion of glossing, and addition to the appendix) satisfies my concerns.", "rebuttal": "We sincerely thank the reviewer for providing supportive comments and acknowledging the strengths of our work. We respond to your questions as follows:\n\n> Discussion about the glossing.\n\nThank you for pointing out this point. We agree that it addresses a fundamental aspect of the sign language domain.\n\nFirst, regarding glossing: Indeed, reducing certain action video frames to a few \"words\" inherently limits the expressiveness of sign language in such representations. This is why resources like 2M-Flores-ASL incorporate markers such as \"+++\" to indicate repeated actions, \"cl:\" to denote specific handshape classifiers, as well as markers like / /WHAT\\ \\ and / /WHY\\ \\ to represent NMMs (with / / and \\ \\ indicating their span) in their dataset. Further, sign language exhibits substantial variability, different sign sequences can convey the same meaning, much like how spoken English uses diverse vocabulary for identical concepts. Crucially, text (spoken language) alone cannot perfectly reverse-engineer the specific visual details (e.g., precise hand movements, spatial configurations) captured in video. That said, given current constraints (e.g., dataset scale and model capabilities), we think that our current relatively \"simple\" glossing approach offers practical advantages: it lowers the learning barrier for models and facilitates more measurable improvements in SLT performance (e.g., BLEU scores). We acknowledge the validity of your concerns, but implementing a more nuanced, expressive representation remains challenging with existing model and data. To address this, we will add an additional section in Limitations and Future Work to discuss these trade-offs, along with avenues for advancing glossing methods to better capture sign language’s rich expressivity.\n\nRegarding \"filtering out unimportant words from text,\" we agree that a true gloss translates only from the sign language video. In this paper, our key point is that core video content in SLT is likely reflected in the corresponding spoken text, playing to LLMs' strength in extracting key info. Table 6a shows our generated glosses outperform Sign2GPT’s (POS row) via WER and BlEU1/BLEU4 comparisons with true glosses. In our next version, we’ll clarify our outputs are not video-derived \"true\" glosses, but argue these pseudo-glosses have comparable value for SLT, with discussion on glossing nuances in the related work section.\n\n> Q1. The description of stage 2: gloss2sign.\n\nSorry for the confusion. Figure 1 should be \"gloss2text,\" consistent with Line 152.\n\n> Q2. German and English have different performance for mBART and Gemma, how do those impact corresponding glossing?\n\nFor current glossing (specifically, generating LLM-based pseudo-glosses), we employ the Gemini 1.5 Pro model which is trained in a multilingual setting, it natively understands German and English inputs, including the example text-gloss pairs. For the mBART- and Gemma-based SLT models, their respective tokenizers are used to process German and English, respectively.\n\n> Q3. Which sentences / example pairs are used to train the LLMs? How does this impact the performance?\n\nThis is a good question. We randomly select example pairs to prompt the LLM for pseudo-gloss generation, and we didn’t fine-tune the LLM, relying solely on these examples as prompts. To date, we have not explored how different pair selections might affect results, but we have conducted experiments comparing text-gloss vs. gloss-text pair formats, e.g., “Example1: text: [] gloss: []” vs. “Example 1: gloss: [] text: []”. For gloss quality, using text-gloss pairs as prompts yields pseudo-glosses closer to true glosses (with better BLEU1 and BLEU4 scores) than gloss-text prompts. We will add relevant discussion in the paper’s appendix."}, {"review_id": "grtRyLlVWP", "reviewer": "Reviewer_tD2m", "summary": "This paper introduces a novel gloss-free pseudo gloss generation framework for Sign Language Translation (SLT) that addresses the limitations of relying on costly and scarce human-annotated gloss labels. The proposed method, PGG-SLT, leverages Large Language Models (LLMs) to produce \"pseudo glosses\" from spoken language text using in-context learning with a few example pairs. To overcome the issue of LLM-generated glosses not aligning temporally with sign videos, a weakly supervised learning process is introduced to reorder these pseudo glosses.", "questions": "See weaknesses.", "limitations": "Since, for various reasons(Like missing large-scale pertaining), the method hasn't reached SOTA performance, I recommend adding a limitation section and listing some potential directions for future work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n1. The paper is clearly written and easy to follow.\n2. The proposed method for pseudo-labeling and weakly supervised order refinement. \n3. The experiment is solid and promising. The ablation analysis is thorough, examining various important design choices in depth.\n### Weaknesses:\n1. The main advantage of the gloss-free method is the potential for scaling up data.  The paper would be more complete if an experiment about the scalability were conducted, for example, by improving performance through introducing extra data or showing the capability to generalize over different domains. \n2. I am a little doubtful whether the order is a real problem. From table(d), the order refinement results in about a 0.5 to 1 BLEU4 difference. I am curious whether this difference can be narrowed if we use a structure that does not require strict alignment, such as RNNT and the Transformer decoder, for predicting the gloss.", "was_revised": true, "final_justification": "I am satisfied with the paper. And since the work has not reached SOTA and weakness 2 in my review is not solved, I will keep my score.", "rebuttal": "We sincerely thank the reviewer for providing supportive comments and acknowledging the strengths of our work. We respond to your questions as follows:\n\n>W1. About scaling up data.\n\nThanks for your comments. We agree that evaluating on even larger web-scale datasets (e.g., YouTubeSL25) is an important next step. We will explicitly highlight this as a key direction for future work in the Limitations and Future Work section. To date, our experimental validation on scaled-up datasets has been limited to Phoenix14T and How2Sign. That said, the How2Sign dataset (being approximately five times the size of Phoenix14T) does provide preliminary evidence of our method’s scalability within a certain range. As for the generalization over different domains, we further conduct the experiments on CSL-Daily dataset, which is based on Chinese, expanding beyond the German and English contexts. The results are shown below:\n\n|Model||Dev Set|||Test Set||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||BLEU1|BLEU4|ROUGE|BLEU1|BLEU4|ROUGE|\n|TS-SLT|58.24|29.18|57.81|58.64|29.55|58.62|\n|GFSLT-VLP*|39.20|11.07|36.70|39.37|11.00|36.44|\n|Sign2GPT*|-|-|-|41.75|15.40|42.36|\n|SignLLM*|42.45|12.23|39.18|39.55|15.75|39.91|\n|VAP*|53.31|23.84|51.19|52.98|23.65|51.09|\n|Ours*|53.58|24.05|52.34|53.29|23.70|52.88|\n\n>W2. RNNT and the Transformer decoder for predicting the gloss.\n\nThis is indeed an intriguing direction. In our current setting, we don’t modify the baseline SLT model itself. This makes the generate-then-reorder a practical and effective solution within the existing experimental setup, as it can be seamlessly integrated without requiring architectural adjustments to the underlying translation framework. We will also discuss this in our Future Work section.\n\n>Q1 Adding a limitation section.\n\nThanks for your valuable suggestion. We will add following discussion to an additional section Limitation and Future Work:\n\nOne limitation of this work is that we only scaled the dataset from Phoenix14T to How2Sign, and the performance of our proposed PGG-SLT when incorporating additional training data remains unvalidated. As shown in Table 5, two other methods that leverage large-scale datasets for pre-training achieve slightly better results than our model. Moving forward, we plan to evaluate the effectiveness of our pseudo-gloss generation method on larger ASL benchmarks. Additionally, we will explore whether mixing diverse large-scale datasets (e.g., ASL and BSL) can further benefit SLT model training, given their shared connection to English as a spoken language."}]

Author rebuttal:

Meta-review: To reduce the costly-annotation in gloss labeling for sign language recognition, this submission proposes a gloss-free pseudo gloss generation method with a few example text-gloss pairs. The proposed method is very clear and easy to follow, and result is very promising. However, sign language as a low resource language, the result on sign language highly dependences the LLM. The submission received one borderline reject and three borderline accepts before rebuttal. However, after the deeply discussion between authors and reviewers, two reviewers increased their ratings, and this submission received one accept and three borderline accepts. All the reviewers reach a consensus of acceptance. The AC agree with the reviewers and recommend accepting this submission.
Decision: Accept (poster)