Title: Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sign Language Translation

Pseudo Gloss Generation

Large Language Model
Abstract: Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation.
Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. 
To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process.
This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss.
We train our SLT model—consisting of a vision encoder and a translator—through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language.
Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.


Peer reviews:
comments: SUMMARY:
This paper introduces a novel gloss-free pseudo gloss generation framework for Sign Language Translation (SLT) that addresses the limitations of relying on costly and scarce human-annotated gloss labels. The proposed method, PGG-SLT, leverages Large Language Models (LLMs) to produce "pseudo glosses" from spoken language text using in-context learning with a few example pairs. To overcome the issue of LLM-generated glosses not aligning temporally with sign videos, a weakly supervised learning process is introduced to reorder these pseudo glosses.

STRENGTHS AND WEAKNESSES:
### Strengths:
1. The paper is clearly written and easy to follow.
2. The proposed method for pseudo-labeling and weakly supervised order refinement. 
3. The experiment is solid and promising. The ablation analysis is thorough, examining various important design choices in depth.
### Weaknesses:
1. The main advantage of the gloss-free method is the potential for scaling up data.  The paper would be more complete if an experiment about the scalability were conducted, for example, by improving performance through introducing extra data or showing the capability to generalize over different domains. 
2. I am a little doubtful whether the order is a real problem. From table(d), the order refinement results in about a 0.5 to 1 BLEU4 difference. I am curious whether this difference can be narrowed if we use a structure that does not require strict alignment, such as RNNT and the Transformer decoder, for predicting the gloss.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Since, for various reasons(Like missing large-scale pertaining), the method hasn't reached SOTA performance, I recommend adding a limitation section and listing some potential directions for future work.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores generation of pseudo-glosses to support sign language translation in settings without gloss annotation, using LLMs for pseudo gloss generation\n\nSTRENGTHS AND WEAKNESSES:\nAs data annotation (glossing) for sign languages is fairly laborious task, the gloss-free sign language translation task explored here is very valuable, particularly for lower resource settings. Notably their results show performance on par with gloss-based approaches. Their pseudo gloss-reordering approach is interesting, the authors conduct a comprehensive set of experiments and the work is well written. \n\nMy main concerns for this work are related to some of the nuances with claims made and framing, particularly related to how glosses are defined and described.  In the introduction, glossing is framed as something specific to sign languages – but in fact, glosses are an intermediary representation used in linguistics when considering translations between languages. Therefore, glosses are not a written form of sign languages (there do exist sign language writing systems, not adopted widely).\nAdditionally, many of the issues described about the different syntax and ordering between signed and spoken languages are pretty much translation aspects applicable to translation between all languages (e.g., between two spoken languages).  I think accurately framing the problem is necessary to evaluate claims but also to situate the work correctly amongst other related research areas. \n\nGlossing for sign language is typically done with reference to sign language, so as to correctly mark the signs present in the sentence. In this work, they instead use the  english translations to approximate the glosses present in the signed sentence. My main concern with this approach is that it seems that the expressiveness of sign language is greatly reduced in the representations. For example, the same English sentence may be signed many different ways in ASL – using different signs, perhaps incorporating classifiers, 3D space, idioms and other unique linguistic features. Using the english translation for glossing then would fail to capture any of this linguistic diversity (e.g., consider many ASL idioms like TOUCH-FINISH, TRAIN-GONE, that are not represented in corresponding english translation at all). Should these learned models and representations ever be used to in sign language generation, the produced sign language would be very homogenized, and bring in spoken language bias.\n\nI am also additionally skeptical of the claim that LLMs are surrogate experts proficient in sign languages – what is the basis of this claim? While LLMs might be good at tasks of “filtering out unimportant words from text” that is not glossing which linguists do on while referring to the sign language video alone. The results are further used to claim that the generated pseudo glosses approximate real gloss annotations – but the experiments do not substantiate this claim. It would require a comparison of generated pseudo glosses to existing glosses. Perhaps a more accurate claim would be that pseudo glosses have comparable value in supporting sign language translation. I believe that the distinction is important, given that glosses have use across different languages SLRGT and linguistic tasks. \n\nOverall, I do think that this work is valuable – I would love to see a more accurate discussion of glossing, and limitations of this gloss-free / pseudo-glossing approach (e.g., linguistic limitations, spoken language bias etc).\n\nQUESTIONS:\nQuestions:\n- The description of stage 2: gloss2sign is slightly confusing, unclear why we are going back to sign – could you clarify?\n- German and English have different performance for mBART and Gemma, how do those impact corresponding glossing?\n- Which sentences / example pairs are used to train the LLMs? How does this impact the performance? \n\nSuggestions:\n- Improved discussion of glossing (as suggested above)\n\nLIMITATIONS:\nSee above related to glossing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper aims to alleviate the reliance on costly gloss annotation with the proposed pseudo gloss generation framework. It starts by prompting LLM with a few samples and utilizes weakly-supervised ordering for better alignment. Experiments are conducted on two datasets to validate the effectiveness.

STRENGTHS AND WEAKNESSES:
Strengths:

1.	The idea is clearly presented.
2.	The paper is well-organized.
3.	The experiment results look promising. 

Weaknesses:

1.	The authors should ensure the rigor of language expression. It should not be claimed as a gloss-free generation framework since it leverages several gloss-text pairs as the prompt of LLMs.
2.	It is suggested to discuss/demonstrate whether the proposed method could be generalizable to more sign languages like CSL.
3.	How do authors choose the gloss-text pairs to ensure more diversity and less bias?
4.	From Table 6a, it seems that the upper bound of gloss generation is still low but does not have a great impact on the final SLT performance. Could the authors provide more explanation on it?
5.	Could the authors analyze the failure cases?
6.	Some related SLT pre-training works are suggested to cite, SignBERT+, SHuBERT…

QUESTIONS:
The questions are raised in weakness part.

LIMITATIONS:
Please discuss the limitations in a section.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of Sign Language Translation (SLT), which converts sign language videos into spoken language text. Traditional SLT methods rely on costly, expert-annotated glosses as intermediate supervision, limiting scalability. To handle this, PGG-SLT is proposed in this paper, a gloss-free framework that leverages Large Language Models (LLMs) to generate pseudo glosses from spoken language text. These pseudo glosses are refined via a weakly supervised reordering mechanism to better align with sign video sequences, including three training stages, video-to-pseudo-gloss recognition, pseudo-gloss-to-text translation, and end-to-end fine-tuning. PGG-SLT provides a scalable, annotation-efficient solution for SLT by harnessing LLMs for pseudo gloss generation and weakly supervised alignment, achieving competitive accuracy without gloss supervision. The framework demonstrates strong generalization across languages and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+A Gloss-Free method is proposed to bypass the costly need for expert-annotated glosses, leveraging LLMs to generate pseudo glosses from spoken language text, which addresses a critical scalability bottleneck in SLT.\n+A weakly supervised video-guided refinement process is introduced, using multi-label classification and temporal smoothing, to align LLM-generated pseudo glosses with sign sequences, enabling effective CTC loss supervision.\n+The results outperform gloss-free methods on Phoenix14T, specifically, BLEU4 +5.0 over GFSLT-VLP, and +6.1 BLEU4 over SSVP-SLT, closing the gap with gloss-based methods.\nWeaknesses\n-LLM-generated glosses inherit spoken language syntax, failing to fully capture sign language-specific grammar. Reordering mitigates but does not eliminate misalignment. \n-Performance highly depends on LLM capabilities, but smaller LLMs might underperform larger counterparts.\n-Experiments are limited to German (Phoenix14T) and English (How2Sign) sign languages. Robustness regarding non-Western or low-resource sign languages is untested.\n\nQUESTIONS:\n1.\tHow do LLM-generated pseudo-glosses capture sign language-specific linguistic features, such as spatial verbs, non-manual markers, that diverge fundamentally from spoken language syntax? Quantitative analysis comparing pseudo-glosses with ground truth for linguistic features, including missing classifiers, incorrect verbs, is necessary.\n2.\tDoes the method maintain performance on sign languages with radically different grammar, such as Chinese and Japanese, or noisy real-world videos? More tests and evaluations could be implemented on datasets based on different languages, like CSL.\n3.\tA few illustrative figures are contained in this paper, and the existing figures fail to effectively convey the core idea and explain the algorithm.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}