Venue: NAACL 2025

An Efficient Gloss-Free Sign Language Translation Using Spatial Configurations and Motion Dynamics with LLMs

Eui Jun Hwang Sukmin Cho Jumyeong Lee Jong C. Park School of Computing Korea Advanced Institute of Science and Technology {ehwa20,nellpic,david516,jongpark}@kaist.ac.kr

###### Abstract

Gloss-free Sign Language Translation (SLT) converts sign videos into spoken language sentences without relying on glosses, which are the written representations of signs. Recently, Large Language Models (LLMs) have shown remarkable translation performance in gloss-free methods by harnessing their powerful natural language generation capabilities. However, these methods often rely on domain-specific fine-tuning of visual encoders to achieve optimal results. By contrast, we emphasize the importance of capturing the spatial configurations and motion dynamics in sign language. With this in mind, we introduce **S**patial and **M**otion-based Sign Language Translation (**SpaMo**), a novel LLM-based SLT framework. The core idea of SpaMo is simple yet effective: instead of domain-specific tuning, we use off-the-shelf visual encoders to extract spatial and motion features, which are then input into an LLM along with a language prompt. Additionally, we employ a visual-text alignment process as a lightweight warm-up step before applying SLT supervision. Our experiments demonstrate that SpaMo achieves state-of-the-art performance on three popular datasets--PHOENIX14T, CSL-Daily, and How2Sign--without visual fine-tuning1.

Footnote 1: cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote †: Cornegonding author.

+
Footnote†: footnotetext: Cornegonding author.

+
Footnote †: Cornegonding author.
 

capabilities. However, the modality gap between the continuous sign videos and discrete text poses a challenge for the LLMs in effectively understanding the sign videos. To address this, many methods fine-tune their visual encoders to be more domain-specific to sign language [24, 10, 35].

That said, fine-tuning visual encoders can be resource-heavy and time-consuming, making it impractical for real-world applications, especially when considering the diversity of sign languages. This leads to an important question: _Is domain-specific tuning of visual encoders necessary to achieve optimal performance in LLM-based SLT?_ We argue that focusing on the inherent characteristics of sign language could reduce the need for such extensive fine-tuning. First, visual encoders trained on general domains [7, 6] have already proven highly effective in downstream tasks such as action recognition [8, 9] and video captioning [11, 12]. Moreover, LLMs are capable of retaining rich visual information from these general encoders in their latent space [11]. Rather than emphasizing fine-tuning, we shift our attention to the crucial roles of **spatial configurations** and **motion dynamics** in sign language.

Spatial configurations include the arranging and positioning of signs within the signing space, including hand shapes, facial expressions, and body postures. These components work together to distinguish different signs and convey their intended meanings [13]. As shown in Figure (a)a, the signs for "cold" and "winter" both use the same handshape, with a shivering motion of the fits. The primary difference lies in the facial expressions: "cold" is typically accompanied by a tense or grimaced expression, while "winter" may feature a more neutral expression. Motion dynamics, on the other hand, involve the path, speed, and rhythm of hand movements, illustrating how movements alter the meanings of signs over time [10]. As shown in Figure (b)b, the signs for "chair" and "sit" both use the same "H" handshape and involve the interaction of both hands. However, the motion differentiates these signs: "chair" involves a repetitive tapping motion, while "sit" involves a single, smooth motion. These examples highlight the importance of the spatial configurations and motion dynamics in conveying accurate messages in sign language.

To this end, we introduce a novel gloss-free framework, **Spati** and **Motion**-based Sign Language Translation (**SpaMo**). SpaMo is designed to fully exploit the spatial configurations and motion dynamics in the sign videos using off-the-shelf visual encoders, all without the need for domain-specific fine-tuning. As shown in Figure 2, the core idea is simple: We extract spatial features (spatial configurations) and motion features (motion dynamics) using two different visual encoders, and feed these into an LLM with a language prompt. Specifically, we use a pre-trained image encoder (e.g., ViT) as **Spatial Encoder (SE)** to individually encode each frame for its spatial features. To further refine the spatial features, we apply \(S^{2}\) scaling [15], which processes a sign image at multiple scales. Additionally, we use a video encoder (e.g., VideoMAE) as **Motion Encoder (ME)** to encode sign clips (groups of sign frames) into the motion features. To capture finer motion dynamics, we apply a sliding window approach, which results in implicit gloss-level representations [12, 13]. Next, **Sign Adapter (SA)**, comprising Multi-Layer Perceptron (MLP) layers, transfers these features to the LLM. To further bridge the modality gap, we propose **Visual-Text Alignment (VT-Align)**, a training strategy that aligns the visual features with the LLM's embedding space, promoting more efficient training and improved translation performance.

In all, our contributions can be summarized as:

* We introduce SPaMo, a novel gloss-free framework based on LLMs. Our method is simple yet effective, focusing on conveying core elements of sign language to LLMs without domain-specific tuning of visual encoders.
* Our proposed method achieves state-of-the-art performance on three popular sign language datasets: PHOENIX14T, CSL-Daily, and How2Sign.
* We provide a novel and comprehensive analysis of how the LLM interprets the sign videos within its embedding space and translates them into corresponding text.

## 2 Related Work

### Gloss-free Sign Language Translation

Gloss-free SLT directly converts sign videos into spoken language sentences without relying on glosses. These methods, however, often underperform compared to gloss-based methods [Cam 

goz et al., 2020; Zhou et al., 2021b, a; Yin et al., 2021; Chen et al., 2022a, b; Zhang et al., 2023b; Jing et al., 2024). To address the performance gap, recent work has focused on several key areas: enhancing the temporal semantic structure Li et al. (2020), improving the alignment between visual and textual modalities Zhao et al. (2021); Lin et al. (2023); Fu et al. (2023), leveraging LLMs Wong et al. (2024); Gong et al. (2024); Chen et al. (2024), and scaling efforts by utilizing larger sign language datasets Uthus et al. (2024); Rust et al. (2024). Despite these advancements, most gloss-free methods depend on fine-tuning visual encoders using the glosses Li et al. (2020); Yin et al. (2023); Fu et al. (2023), target translations Zhou et al. (2023); Wong et al. (2024), or self-supervised learning Gong et al. (2024); Rust et al. (2024). In particular, fine-tuning with the glosses helps the visual encoders to offer more domain-specific training on continuous or isolated Sign Language Recognition (SLR) datasets, such as WLASL Li et al. (2020) and PHOENIX14T Camgoz et al. (2018).

Consequently, we classify these methods as _weakly gloss-free_ due to the implicit involvement of the glosses, as further elaborated in Section 4.2. On the other hand, the rest of the fine-tuning methods eliminate reliance on these annotations. However, they often require substantial resources. As a results, it can be difficult to achieve robust visual representations and improve translation performance without access to a sufficiently large dataset. To address this limitation, our approach diverges from this norm by focusing on capturing the spatial configurations and motion dynamics through off-the-shelf visual encoders. This allows us to bypass the need for resource-intensive fine-tuning.

### Scaling Language Models in SLT

The scaling laws in language models Kaplan et al. (2020) have been pivotal in the rise of Large Language Models (LLMs) Touvron et al. (2023); Chiang et al. (2023); Chung et al. (2024). Leveraging their strong generation capabilities, LLMs have been applied across diverse domains: multilingual translation Zhu et al. (2023); Zhang et al. (2023); Gao et al. (2024), pose generation Feng et al. (2024); Zhang et al. (2024), and visual question answering Li et al. (2023); Liu et al. (2024, 2024), extending their impact beyond Natural Language Processing. LLMs have also demonstrated impressive translation performance in the SLT domain. SLT methods using the LLMs focus on aligning high-dimensional visual features with inputs comprehensible to LLMs. These methods involve fine-tuning visual encoders to produce language-like tokens Gong et al. (2024), using pseudoglosses Wong et al. (2024), or performing video-grounded text generation tasks Chen et al. (2024).

In this work, we take a different approach by focusing on spatial configurations and motion dynamics. We extract spatial and motion features and pass them to LLMs with a light warm-up process. This method is simple yet effective, demonstrating that an extensive pre-training for the visual encoders is unnecessary to achieve peak performance.

## 3 Method

We first give an overview of our framework in Section 3.1. We then explain Spatial Encoder (SE) and Motion Encoder (ME) in Sections 3.2 and 3.3, respectively. Next, we discuss Sign Adapter (SA) in Section 3.4 and VT-Align in Section 3.5. Finally, we explain the training details in Section 3.6.

### Framework Overview

Given a sign video \(X=\{x_{i}\}^{T}_{i=1}\), where each frame \(x_{i}\in\mathbb{R}^{H\times W}\) represents a frame with height \(H\) and width \(W\), the objective of SLT is to generate a corresponding translation \(Y=\{y_{j}\}^{U}_{j=1}\), composed of \(U\) words. Previous gloss-free methods Zhou et al. (2023); Wong et al. (2024); Gong et al. (2024); Chen et al. (2024) have involved fine-tuning visual encoders using sign language data to make them more domain-specific, leading to improvements in translation accuracy. However, while this fine-tuning introduces more domain knowledge at the feature extraction level, it is unnecessary, especially with LLMs, which already maintain rich visual information from the visual encoder in their latent space Zhang et al. (2024). Although there may be a trade-off, we argue that utilizing the spatial configurations and motion dynamics with proper alignment and training, offers a more efficient and effective solution.

As shown in Figure 2, SE and ME extract two distinct features from the sign video \(X\): Spatial features \(Z_{s}\) capture the spatial configurations Emmorey and Casey (1995), and motion features \(Z_{m}\) represent the motion dynamics Bosworth et al. (2019). These features are then integrated into a combined sign feature \(Z_{sm}\) via SA. The combined feature is then fed to an LLM with a language prompt, guiding the LLM to generate the trans 

lation in the desired language. Additionally, we perform Visual-Text Alignment (VT-Align) to minimize the gap between the visual and textual modalities before and during training under SLT supervision. In the following sections, we provide a detailed explanation of SE, ME, SA, and VT-Align.

### Spatial Encoder

SE extracts spatial features \(Z_{s}\) from the sign video \(X\). We utilize a pre-trained image encoder (e.g., ViT), which is kept frozen, and enhances its capability to capture finer spatial information by applying Scaling on Scales (\(S^{2}\)) Shi et al. (2024). \(S^{2}\) is parameter-free and enables the extraction of multi-scale features without altering the original pre-trained encoder. By processing sign images at multiple resolutions, \(S^{2}\) provides a more comprehensive spatial understanding, ensuring that SE captures both fine-grained and broad spatial details for accurate sign language interpretation. The resulting spatial features can be represented as \(Z_{s}\in\mathbb{R}^{T\times 2d}\), where \(T\) is the number of frames, and \(2d\) is the increased embedding dimension, reflecting the integration of multi-scale features.

### Motion Encoder

ME derives motion features from the sign video \(X\). Similar to SE, we employ a pre-trained video encoder (e.g., VideoMAE), which remains frozen, to process sign clips segmented from the video. However, accurately segmenting the sign video into distinct gloss-level clips is challenging without the support of pre-trained Continuous Sign Language Recognition (CSLR) models Wei and Chen (2023). To address this limitation, we use a sliding window approach to capture implicit gloss-level representations Cheng et al. (2023); Hwang et al. (2024). Specifically, we divide the sign video into short, overlapping clips, then feed each clip into ME to extract the implicit gloss-level motion features \(Z_{m}\in\mathbb{R}^{N\times d}\), where \(N\) is the number of segments. The number of segments \(N\) is calculated as \(N=\bigsqcup\frac{T-w}{s}\Bigrbrvert+1\), where \(T\) is the total number of frames, and \(w\) and \(s\) are the window size and stride, respectively. Since \(Z_{m}\) is generated by processing \(N\) short clips, it can also be interpreted as a sequence of \(N\) clip-wise features.

### Sign Adapter

In the previous sections, we extracted two distinct visual features: the spatial features \(Z_{s}\) and the motion features \(Z_{m}\). These features differ in both their

Figure 3: An overview of Sign Adapter.

Figure 2: An overview of the SpaMo framework, which consists of three parts: (i) **Sign Feature Extraction**: Spatial and motion features are extracted using SE and ME, using the \(S^{2}\) and sliding window approaches to capture detailed spatial configurations and motion dynamics. (ii) **VT-Align**: The extracted features are combined within SA to form a unified sign feature. During training, a warm-up process is employed to ensure that SA has well-initialized weights, effectively bridging the modality gap between the sign video and text. (iii) **LLM**: the LLM processes the sign feature along with a language-instructive prompt and is trained using LoRA.

 

dimensions and representation, as depicted in Figure 2. To effectively integrate these features, we introduce an additional module called Sign Adaptor (SA). As shown in Figure 3, SA includes linear projection layers, a 1D TCN, and a Multi-Layer Perceptron (MLP). These components work together to integrate the spatial and motion features into a unified sign representation, denoted as \(Z_{sm}\). First, the spatial and motion features are passed through linear projection layers to transform them into features with matching dimensions. Next, the 1D TCN is applied for short-term modeling of the combined features. Finally, a cross-modal MLP Liu et al. (2024) is employed to bridge the visual and textual modalities. The resulting outputs are represented as \(Z_{sm}\in\mathbb{R}^{M\times d^{\prime}}\), where \(M\) represents the reduced number of frames after convolution, and \(d^{\prime}\) is the dimension aligned with that of the LLM. Although SA aids in bridging the modality gap between visual and textual features during training under the SLT supervision, the gap remains. To tackle this issue, we introduce VT-Align, which will be detailed in the next section.

### Visual-Text Alignment

VT-Align is a _warm-up and go_ process designed to provide the SA module with well-initialized weights before the SLT supervision begins. This initial alignment is crucial, as it helps the model more effectively bridge the modality gap during training. To achieve this alignment, we employ a widely-used softmax-based contrastive learning approach Radford et al. (2021); Jia et al. (2021).

Specifically, given a mini-batch \(\mathcal{B}=\{(S_{1},Y_{1}),(S_{2},Y_{2}),...\}\) of sign-text pairs, the contrastive learning objective encourages the embeddings of matching pairs \((S_{i},Y_{i})\) to align closely while pushing apart the embeddings of mismatched pairs \((S_{i},Y_{j\neq i})\). Text features \(Z_{t}\) are extracted from the target translation \(Y_{i}\) using the LLM's embedding layer \(E_{llm}(\cdot)\). Note that only the SA module \(f_{sa}(\cdot)\) is updated during this process, while \(E_{llm}(\cdot)\) remains fixed to preserve the LLM's language capabilities. The VT-Align loss function \(\mathcal{L}_{vt}\) is represented as follows:

\[-\frac{1}{2|\mathcal{B}|}\sum\limits_{i=1}^{|\mathcal{B}|}\left(\frac{\overbrace {e^{xZ_{sm}^{(i)}\cdot z_{t}^{(i)}}z_{t}^{(i)}}^{\text{sign}-\text{test~{} softmax}}}{\sum\nolimits_{j=1}^{|\mathcal{B}|}e^{xZ_{sm}^{(i)}\cdot Z_{t}^{(i)}}^{ \text{sign}}+\overbrace{\log\frac{e^{xZ_{sm}^{(i)}\cdot Z_{t}^{(i)}}}{\sum \nolimits_{j=1}^{|\mathcal{B}|}e^{xZ_{sm}^{(j)}\cdot Z_{t}^{(i)}}}^{\text{ext} -\text{sign~{}softmax}}}{\sum\nolimits_{j=1}^{|\mathcal{B}|}e^{xZ_{sm}^{(j)} \cdot Z_{t}^{(i)}}}^{\text{ext}-\text{sign~{}softmax}}}\right), \tag{1}\]

where \(Z_{sm}^{(i)}=\frac{f_{sa}(S_{i})}{\left\lVert f_{sa}(S_{i})\right\rVert_{2}}\), \(Z_{t}^{(i)}=\frac{E_{llm}(T_{i})}{\left\lVert E_{llm}(T_{l})\right\rVert_{2}}\), and \(\tau\) denotes a learnable temperature parameter used to scale the logits.

### Training Details

Our framework is optimized in two phases: an initial warm-up phase followed by training with the SLT supervision. In the warm-up phase, we begin with a training phase the SA module using VT-Align for a designated number of steps (e.g., 4K steps). After completing the warm-up phase, we proceed to the joint training for both SA and the LLM. For fine-tuning the LLM, we utilize LoRA Hu et al. (2021), a lightweight and efficient method specifically designed for this purpose. Overall, our method is trained with a combined loss function:

\[\mathcal{L}_{\textit{SpaMo}}=\mathcal{L}_{ce}+\mathcal{L}_{vt}, \tag{2}\]

where \(\mathcal{L}_{ce}\) represents cross-entropy loss, and \(\mathcal{L}_{vt}\) continuously manages the alignment.

## 4 Experiments

### Implementation Details

For SE and ME, we use CLIP ViT-L/14Radford et al. (2021) and VideoMAE-L/16 Tong et al. (2022), respectively. To extract the spatial features, the sign images are interpolated to multiple scales, such as \(224\times 224\) and \(448\times 448\). For each scale, larger images are split into sub-images of regular size (\(224\times 224\)) and processed individually. These features from the sub-images are then pooled and concatenated with features from the original representation. For the motion features, each clip consists of 16 frames, based on the findings from Wilbur (2009), which suggests that this frame interval captures a single sign. We set the stride \(s\) between consecutive clips to 8. We use FlanT5-XL Chung et al. (2024) as the LLM for PHOENIX14T and How2Sign, while mT0-XL Muennighoff et al. (2022) is used for the CSL-Daily. During the warm-up phase with VT-Align, we use 4K steps on PHOENIX14T and CSL-Daily, and 15K steps on How2Sign. Additional implementation details can be found in Appendix A.

### Experimental Settings

**Datasets.** We evaluated our method on three sign language datasets: PHOENIX14T Camgoz et al. (2018), CSL-Daily Zhou et al. (2021), and How2Sign Duarte et al. (2021). **PHOENIX14T** is a German Sign Language dataset comprising 

8,257 samples and a vocabulary of 2,887 German words. **CSL-Daily** is a Chinese Sign Language dataset with 20,654 samples and a 2,343 Chinese characters. **HowZSign** focuses on American Sign Language and includes 35,191 samples with a vocabulary of 16K English words. Detailed dataset statistics are provided in Appendix B.

Evaluation Metrics.We report BLEU via Sacre-BLEU Papineni et al. (2002); Post (2018)2 and ROUGE-L Lin and Och (2004). BLEU-n assesses translation precision by evaluating n-grams. ROUGE-L measures text similarity by calculating the F1 score based on the longest common subsequences between predicted and reference texts. We also report BLEURT Sellam et al. (2020) from the BLEURT-20 checkpoint3, which has been shown to correlate well with human judgments.

Footnote 2: [https://lensex.nl/lexsae:mixed](https://lensex.nl/lexsae:mixed)|effr=[no]tok:13a/hwsmooth:expversion:2.2.1

Footnote 3: [https://huggingface.co/lucadiliello/BLURT-20](https://huggingface.co/lucadiliello/BLURT-20)

A Taxonomy of SLLT.In Section 2, we explored gloss-free methods, including those that incorporate gloss-supervised visual encoders. Although these approaches have traditionally been categorized as gloss-free, we argue that they should more accurately be described as _weakly gloss-free_ due to their dependence on gloss-annotated data. This classification is detailed in Table 1. Specifically, methods such as TSPNet Li et al. (2020b), GASLT Yin et al. (2023), ConSLT Fu et al. (2023), GloFE-VN Lin et al. (2023), and OpenSLT Tarres et al. (2023) rely on sign features extracted by visual encoders trained on continuous or isolated sign language recognition (SLR) datasets.

### Comparison with State-of-the-Art Results on PHOENIX14T and CSL-Daily.

We first compared our method with both gloss-based and gloss-free methods on PHOENIX14T. As shown in Table 1, most previous methods rely on the domain-specific fine-tuning of their visual encoders. By contrast, our method demonstrates consistent improvements across all reported metrics on PHOENIX14T without such fine-tuning. The only exception is ROUGE, where we achieved the second-best result. Specifically, the improvement on BLEU-4 is by a margin of 0.92, representing a 3.93% increase over SignLLM Gong et al. (2024). On CSL-Daily, which covers a broader range of topics than PHOENIX14T, the performance gains are even more pronounced. Our method achieved a margin increase of 4.8 in BLEU-4, reflecting a 30.41% improvement over SignLLM.

Results on How2Sign.Next, we evaluated our method on How2Sign, which poses greater challenges than PHOENIX14T due to its broader open-domain nature, longer sign videos, and larger vocabulary. As shown in Table 2, our method outperformed previous methods across all reported

\begin{table}
\begin{tabular}{c l c c c c c c c c c} \hline \hline  & & \multicolumn{4}{c}{PHOENIX14T} & \multicolumn{4}{c}{CSL-Daily} \\ \cline{2-11}
* & 37.38 & 24.36 & 16.55 & 11.79 & 36.74 \\  & BN-TN-Transf-SegNT Zhou et al. (2021) & ✗ & 50.30 & 37.55 & 29.72 & 24.32 & 49.54 & 51.42 & 37.26 & 27.26 & 27.16 & 23.44 \\  & MHTLS Chen et al. (2022) & 33.22m & 53.97 & 41.75 & 33.84 & 28.39 & 52.65 & 53.31 &
- & 30.81 & 32.39 & 53.25 \\  & TS-SLT Chen et al. (2022) & ✓ & 54.90 & 42.43 & 44.86 & 28.95 & 54.83 & 55.44 & 42.59 & 32.87 & 25.79 & 55.72 \\  & STUNET Zhang et al. (2028) & ✓ & 52.92 & 41.76 & 33.99 & 28.47 & 52.11 & 54.98 & 41.44 & 51.84 & 25.01 & 54.08 \\ \hline \multirow{3}{*}{Weddy Glass-free} & TFNet Liu et al. (2020)+ & ✓ & 36.10 & 32.12 & 16.88 & 13.44 & 39.76 & 17.09 & 39.88 & 5.07 & 2.97 & 18.38 \\  & GASHY Liu et al. (2023) & ✓ & 39.07 & 26.74 & 28.15 & 15.74 & 39.86 & 19.90 & 9.94 & 5.98 & 4.07 & 20.35 \\  & CNNST Yu et al. (2023) & ✓ &
- & 21.59 & 47.69 &
- \\ \hline \multirow{6}{*}{Gloss-free} & SGCRox Zhou et al. (2021) & ✗ & 36.71 & 25.40 & 18.86 & 15.83 & 15.83 & 15.83 & 15.83 & \\  & GSELV Zhu et al. (2022) & ✓ & 43.71 & 33.58 & 26.11 & 21.44 & 22.99 & 39.37 & 24.53 & 16.26 & 11.00 & 36.44 \\ \cline{1-1}  & FL-LMM Chen et al. (2024) & ✓ & 46.25 & 33.53 & 28.03 & 25.09 & 45.27 & 37.13 & 25.12 & 18.38 & 14.20 & 37.25 \\ \cline{1-1}  & SPoxFiner Wang et al. (2024) & ✓ & 49.55 & 35.96 & 28.83 & 22.52 & **48.90** & 41.75 & 28.22 & 30.60 & 15.40 & 42.36 \\ \cline{1-1}  & SpaiLM Gong et al. (2024) & ✓ & 45.21 & 34.78 & 28.05 & 22.40 & 44.49 & 39.55 & 28.13 & 20.07 & 15.72 & 39.91 \\ \cline{1-1}  & **SpaiMax Guers** & ✗ & **68.80** & **37.32** & **29.50** & **24.32** & 46.57 & **48.90** & **36.50** & **26.75** & **20.65** & **47.46** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance comparison on the PHOENIX14T and CSL-Daily datasets. “Vis. Fit.” denotes the visually fine-tuned on sign language datasets. ‡ denotes results reproduced by Yin et al. for CSL-Daily. The best results are highlighted as **bold**, and the second-best are underlined.

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline

- \\ \hline \multirow{6}{*}{Glass-free} & YAT-SALSTM Liu et al. (2024) & Lambmark & ✗ & 14.96 & 5.11 & 2.26 & 1.22 &
- & 29.98 \\  & SSVP-SLT Fan et al. (2024)† & RGB & ✓ & 30.20 & 16.70 & 10.50 & 7.00 & 25.70 & 39.30 \\ \cline{1-1}  & FA+LM Chun et al. (2024) & RGB & ✓ & 29.81 & 18.90 &
- & 13.27 & 9.66 &
- & 27.81 \\ \cline{1-1}  & **SpaiMo (Ours)** & RGB & ✗ & **33.41** & **20.28** & **13.96** & **10.11** & **30.56** & **42.23** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison of translation results on the How2Sign dataset. YTI-ASL-SLT and SSPV-SLT (marked with †) are reported without dataset scaling to ensure a fair comparison.
 

metrics. Specifically, we achieved a 0.45 margin in BLEU-4 which represents a 4.66% improvement over Fla-LLM Chen et al. (2024). We see a performance gain in BLEURT, reaching 2.93, which is 7.46% higher than SSVP-SLT Rust et al. (2024) under the non-scaled dataset setting.

Kernel Density Estimation.To assess the quality of sign representations, following Ye et al. (2023), we employed the Kernel Density Estimation (KDE) to estimate the probability density functions of embeddings from GFSLT-VLP and our method on PHOENIX14T. Note that we reproduced GFSLT-VLP using the official code4. As shown in Table 5, our method produced more compact and confident representations than GFSLT-VLP. More details on the KDE process are provided in Appendix A.4.

Footnote 4: [https://github.com/zhoubenjia/GFSLT-VLP](https://github.com/zhoubenjia/GFSLT-VLP)

### Ablation Study

To further evaluate our method, we conducted extensive ablation experiments on PHOENIX14T, the most widely used sign language dataset. Additional results can be found in Appendix C.

Effect of Main Components.We begin by evaluating the impact of the key components in our framework: SE, ME, and VT-Align. As shown in Table 3, using SE or ME individually results in lower performance, with ME performing the worst in terms of BLEU-4. However, combining SE and ME leads to an overall improvement. Notably, when VT-Align is integrated with SE, the performance rises, nearly matching Sign2GPT (22.52 vs. 22.49). The best results are achieved when all components (SE, ME, and VT-Align) are used together, yielding the highest scores across all metrics. This highlights the importance of each component in enhancing overall performance of SpaMo.

Effect of LLM.Next, we explored the impact of different types of LLMs by replacing our model, as shown in Table 4. We compared five models, each with a different number of parameters: our method without pre-trained weights, mBART-L, mT0-XL, Flan-T5-XL, and Llama-2. Among these, Flan-T5-XL achieves the best performance, though it nearly ties with mT0-XL. Interestingly, despite its larger parameter count, Llama-2, which is also employed in SignLLM, does not outperform the others. This finding aligns with the observations of Zhang et al., suggesting that scaling up LMs does not always lead to better performance. In our case, the main reason likely lies in the fact that larger models generally demand more extensive and higher-quality data to unlock their full potential Kaplan et al. (2020); Hoffmann et al. (2022). Since PHOENIX14T is constrained in both scale and diversity, merely increasing the model size does not necessarily yield substantial performance gains.

Effect of \(S^{2}\) and Neighboring Gap.Finally, we evaluated the effect of \(S^{2}\) and the gap between neighboring clips on SE and ME, respectively. As shown in Figure 4, \(S^{2}\) substantially improves translation performance, highlighting its effectiveness in helping SE capture more spatial details. Additionally, our analysis reveals that a stride size of 8 between neighboring clips yields the best results, suggesting that this stride size optimally aids ME in extracting the motion dynamics.

Figure 4: Ablation study for SE and ME. \(S^{2}\) represents Scaling on Scales, and \(s\) denotes stride size. Note that the presented results do not include VT-Align.

\begin{table}
\begin{tabular}{l c r r r} \hline \hline
**Models** & **FTraining Pairs** & **Flotal Params** & **B4** \\ \hline wv LMM & 60.5M & 60.5M & 6.35 \\ hARFL Ma et al. (2020) & 66MM & 66MM & 66MM & 10.84 \\ nrR-X. Liu et al. (2020) & 23.5M & 3.5M & 24.25 \\ llama-2. Qiu et al. (2020) & 32.6M & 78 & 13.86 \\ Plau-T5-XL Cheng et al. (2024) & 21.7M & 38 & **24.32** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation study of the impact of LLM.

 



### Qualitative Analysis

Translation Results.Table 6 presents two example translations on PHOENIX14T, comparing our method with GFSLT-VLP, the only other publicly available baseline. In the first example (top), our method provides an accurate translation, whereas GFSLT-VLP fails to capture the correct semantic meaning. In the second example (bottom), our method again produces a precise translation, while GFSLT-VLP introduces errors, resulting in incorrect information. These examples demonstrate the superior accuracy of our method in generating reliable translations. More translation examples are shown in Appendix C.4.

Visual Token Analysis.We performed an additional analysis to explore how the LLM interprets the sign videos. For each visual feature, we identified the word with the shortest distance in the LLM's embedding space, representing the closest match. Further details can be found in Appendix A.5. Figure 5 shows the t-SNE visualization of each sign feature mapped to the corresponding word. We observed that certain visual features align closely with specific words, which likely represents the semantic concepts that the LLM associates with these features. In other words, these words represent the LLM's interpretation or labeling of the visual content. We refer to these mapped words as "visual tokens". We further compared these visual tokens with the ground-truth glosses as shown in Table 7. To ensure a clearer and more accurate semantic comparison, repetitive words were removed from the visual tokens. Surprisingly, the LLM's interpretation of the sign videos is similar to the glosses, though not perfectly aligned. This suggests that the LLM has learned to link particular video patterns with specific textual concepts, explaining why those words cluster near the visual features in the embedding space. Additionally, we found that the visual tokens capture words that are present in the translation but not in the glosses. This finding suggests that visual tokens may provide a more comprehensive representation than current glosses, potentially broadening their scope beyond what has been traditionally documented.

## 5 Conclusion

In this paper, we introduced SpaMo, a novel gloss-free SLT framework based on LLMs. Apart from the previous methods that rely on domain-specific fine-tuning of their visual encoders, SpaMo focuses on capturing the spatial configurations and motion dynamics, eliminating the need for resource-intensive fine-tuning. We also proposed VT-Align,

\begin{table}
\begin{tabular}{c p{280.9pt}} \hline \hline Ref. & die new woche begint noch wechselhaft und evwas kühler \\  & _the new week beginst smalten and somewhat cooler)_ \\ GFSLT-VLP & am unong jovier wechselhaft und kehiller. \\ Ours: & _on Monday_ gotuin duggerbaud _and vociery_ \\ Ours: & den woche begint suchsenhaft und väder isthier. \\ \cline{3-3}  & _the new week beginst chungköeit card again couder)_ \\ \hline Ref. & senat veil somnechen. \\ GFSLT-VLP & en differenuie, a lot of sunne. \\ GPSLT-VLP & in threnglen und viel sounne. \\ Ours: & sonut veil somnechen. \\ Ours: & osterhuie, a lot of sunne. \\ \hline \hline \end{tabular}
\end{table}
Table 6: Translation results on the test set compared to GFSLT-VLP on PHOENIX14T. Correctly translated l-grams are highlighted in blue, while incorrect translations are marked in red.

Figure 5: The t-SNE visualization of sign features. Different colors represent features with distinct semantics, while gray points are other categories not listed.



a training strategy that effectively aligns and narrows the modality gap between the sign videos and target texts, enabling the transformation of the sign videos into inputs interpretable by the LLM. Our approach achieved state-of-the-art results on three popular datasets. Furthermore, we provided the first comprehensive analysis of how the LLM interprets the sign videos within its embedding space and translates them into corresponding text.

### Limitations

Recently, scaling datasets (Uthus et al., 2024; Rust et al., 2024) has consistently led to performance improvements, as seen with larger sign language datasets, such as Youtube-ASL (Uthus et al., 2024) and BOBSL (Albanie et al., 2021). While dataset scaling could also enhance our method, in this work, we focus on a constrained setting. Specifically, we use limited sign language datasets to evaluate and compare results, demonstrating the effectiveness of our method in resource-limited scenarios. Future work will involve expanding the dataset size to explore the full potential of our method and to assess its scalability and performance across more extensive and diverse datasets.

In this paper, we highlight that domain-specific fine-tuning of visual encoders is not essential for our method. However, this implies that our method relies on visual encoders pre-trained on general tasks such as action recognition and image captioning. To bridge this gap, we introduce a pre-alignment process and apply LoRA fine-tuning to the LLM. While this might appear to be a compromise, it significantly reduces the resource requirements compared to fine-tuning both the visual encoders and the LLM. Additionally, as we discussed in the previous paragraph, this limitation can be addressed as more data becomes available, allowing for improved scalability and performance over time.

## Ethics Statement

Our work focuses on developing a practical framework for SLT with the goal of overcoming communication barriers faced by the Deaf and hard-of-hearing communities. Although our approach utilizes off-the-shelf visual encoders and LLMs, there is a possibility that the framework could produce unexpected or biased outputs due to the inherent limitations in the pre-trained models. However, we are optimistic that future advancements in LLMs will help mitigate these issues. We rely on open datasets such as PHOENIX14T (Camgoz et al., 2018), CSL-Daily (Zhou et al., 2021), and How2Sign (Duarte et al., 2021), which, while containing potentially identifiable information, present minimal concerns regarding personal privacy. Additionally, our method has been validated only on German, Chinese, and American sign languages, limiting its applicability to other sign languages. We call for future research in SLT to expand on a broader range of sign languages, promoting greater equity for the Deaf community.

## Acknowledgment

This work was supported by the Institute for Information and communications Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00010, Development of Korean sign language translation service technology for the deaf in medical environment).

## References

* Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. _Advances in neural information processing systems_, 35:23716-23736.
* Albanie et al. [2021] Samuel Albanie, Gul Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencic Woll, Rob Cooper, Andrew McParland, and Andrew Zisserman. 2021. BOBSL: BBC-Oxford British Sign Language Dataset. _arXiv_.
* Bardes et al. [2024] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. 2024. Revisiting feature prediction for learning visual representations from video. _arXiv preprint arXiv:2404.08471_.
* Bosworth et al. [2019] Rain G Bosworth, Charles E Wright, and Karen R Dobkins. 2019. Analysis of the visual spatiotemporal properties of american sign language. _Vision research_, 164:34-43.
* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Camgoz et al. [2018] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 7784-7793 

Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign language transformers: Joint end-to-end sign language recognition and translation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 10023-10033.
* Chen et al. (2022a) Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022a. A simple multi-modality transfer learning baseline for sign language translation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 5120-5130.
* Chen et al. (2022b) Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. 2022b. Two-stream network for sign language recognition and translation. _Advances in Neural Information Processing Systems_, 35:17043-17056.
* Chen et al. (2024) Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. 2024. Factorized learning assisted with large language model for gloss-free sign language translation. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_, pages 7071-7081. ELRA and ICCL.
* Cheng et al. (2023) Yiting Cheng, Fangyun Wei, Jianmin Bao, Dong Chen, and Wenqiang Zhang. 2023. Cico: Domain-aware sign language retrieval via cross-lingual contrastive learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 19016-19026.
* Cheng et al. (2024) Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. _arXiv preprint arXiv:2406.07476_.
* Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianming Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgtp quality. 2(3):6.
* Chung et al. (2024) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. _Journal of Machine Learning Research_, 25(70):1-53.
* Duarte et al. (2021) Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2sign: a large-scale multimodal dataset for continuous american sign language. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 2735-2744.
* Emmorey and Casey (1995) Karen Emmorey and Shannon Casey. 1995. A comparison of spatial language in english & american sign language. _Sign Language Studies_, 88(1):255-288.
* Feng et al. (2024) Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyamka Patel, and Michael J Black. 2024. Chatting pose: Chating about 3d human pose. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 2093-2103.
* Fu et al. (2023) Biao Fu, Peigen Ye, Liang Zhang, Pei Yu, Cong Hu, Xiaodong Shi, and Vidong Chen. 2023. A token-level contrastive framework for sign language translation. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 1-5. IEEE.
* Gao et al. (2024) Pengzhi Gao, Zhongjun He, Hua Wu, and Haifeng Wang. 2024. Towards boosting many-to-many multilingual machine translation with large language models. _arXiv preprint arXiv:2401.05861_.
* Gong et al. (2024) Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. 2024. LLMs are good sign language translators. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 18362-18372.
* Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. _arXiv preprint arXiv:2203.15556_.
* Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Hu et al. (2023) Lianyu Hu, Liqing Gao, Zekang Liu, and Wei Feng. 2023. Continuous sign language recognition with correlation network. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 2529-2539.
* Huang et al. (2024) Xiahou Huang, Hao Zhou, Kun Yao, and Kai Han. 2024. FROSTER: frozen CLIP is A strong teacher for opensocabulary action recognition. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Hwang et al. (2024) Eui Jun Hwang, Sukmin Cho, Huije Lee, Youngwoo Yoon, and Jong C Park. 2024. Universal gloss-level representation for gloss-free sign language translation and production. _arXiv preprint arXiv:2407.02854_.
* Jia et al. (2021) Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerg. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In _International conference on machine learning_, pages 4904-4916. PMLR.
* Jing et al. (2024) Liqiang Jing, Xuemeng Song, Xinxing Zu, Na Zheng, Zhongzhou Zhao, and Liqiang Nie. 2024. Vk-g2t: Vision and context knowledge enhanced gloss2text.

 

In _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7860-7864. IEEE.
* Ju et al. [2023] Tianjie Ju, Yubin Zheng, Hanyi Wang, Haodong Zhao, and Gongshen Liu. 2023. Is continuous prompt a combination of discrete prompts? towards a novel view for interpreting continuous prompts. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 7804-7819.
* Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _ArXiv_abs/2001.08361.
* Li et al. [2020a] Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. 2020a. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_, pages 1459-1469.
* Li et al. [2020b] Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. 2020b. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. _Advances in Neural Information Processing Systems_, 33:12034-12045.
* Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstraping language-image pretraining with frozen image encoders and large language models. In _International conference on machine learning_, pages 19730-19742. PMLR.
* Lin and Och [2004] Chin-Yew Lin and Franz Josef Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In _Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04)_, pages 605-612.
* Lin et al. [2023] Kezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang. 2023. Gloss-free end-to-end sign language translation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_.
* Liu et al. [2024a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 26296-26306.
* Liu et al. [2024b] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. _Advances in neural information processing systems_, 36.
* Liu et al. [2020] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. _Trans. Assoc. Comput. Linguistics_, 8:726-742.
* Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. _arXiv preprint arXiv:1711.05101_.
* Muennighoff et al. [2022] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. In _Annual Meeting of the Association for Computational Linguistics_, pages 15991-16111.
* Oquab et al. [2023] Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. _arXiv preprint arXiv:2304.07193_.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_, pages 311-318.
* Post [2018] Matt Post. 2018. A call for clarity in reporting bleu scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_.
* Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Rusti et al. [2024] Phillip Rusti, Bowen Shi, Skyler Wang, Necati Cihan Camgoz, and Jean Maillard. 2024. Towards privacy-aware sign language translation at scale. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892.
* Shi et al. [2024] Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. 2024. When do we not need larger vision models? _arXiv preprint arXiv:2403.13043_.
* Shi et al. [2022] Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2022. Open-domain sign language translation learned from online video. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ 

Zixuan Tang, Youjun Zhao, Yuhang Wen, and Mengyuan Liu. 2024. A survey on backbones for deep video action recognition. _arXiv preprint arXiv:2405.05584_.
* Tarres et al. (2023) Laia Tarres, Gerard I Gallego, Amanda Duarte, Jordi Torres, and Xavier Giro-i Nieto. 2023. Sign language translation from instructional videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_, pages 5625-5635.
* Tong et al. (2022) Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. 2022. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pretraining. _Advances in neural information processing systems_, 35:10078-10093.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Uthus et al. (2024) Dave Uthus, Garrett Tanzer, and Manfred Georg. 2024. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. _Advances in Neural Information Processing Systems_, 36.
* Wei and Chen (2023) Fangyun Wei and Yutong Chen. 2023. Improving continuous sign language recognition with cross-lingual signs. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 23612-23621.
* Wilbur (2009) Ronnie B Wilbur. 2009. Effects of varying rate of signing on asl manual signs and nonmanual markers. _Language and speech_, 52(2-3):245-285.
* Wong et al. (2024) Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. 2024. Sign2gpt: Leveraging large language models for gloss-free sign language translation. In _Proceeding of the Eleventh International Conference on Learning Representations_.
* Yang et al. (2023) Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 10714-10726.
* Ye et al. (2023) Jinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Hui Xiong. 2023. Cross-modality data augmentation for end-to-end sign language translation. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 13558-13571.
* Yin et al. (2021) Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. 2021. Simulstlr: End-to-end simultaneous sign language translation. In _Proceedings of the 29th ACM International Conference on Multimedia_, pages 4118-4127.
* Yin et al. (2023) Aoxiong Yin, Tianyan Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. 2023. Gloss attention for gloss-free sign language translation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 2551-2562.
* Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In _International Conference on Machine Learning_, pages 41092-41110. PMLR.
* Zhang et al. (2023b) Biao Zhang, Mathias Muller, and Rico Sennrich. 2023b. SLtUNET: A simple unified model for sign language translation. In _The Eleventh International Conference on Learning Representations_.
* Zhang et al. (2024a) Biao Zhang, Garrett Tanzer, and Orhan Firat. 2024a. Scaling sign language translation. _Advances in neural information processing systems_.
* Zhang et al. (2024b) Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. 2024b. Motiongrptt: Finetuned lms are general-purpose motion generators. In _Proceedings of the AAAI Conference on Artificial Intelligence_, 7, pages 7368-7376.
* Zhang et al. (2024c) Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yucheng Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024c. Why are visually-grounded language models bad at image classification? _arXiv preprint arXiv:2405.18415_.
* Zhao et al. (2021) Jian Zhao, Weizhen Qi, Wengang Zhou, Nan Duan, Ming Zhou, and Houqiang Li. 2021. Conditional sentence generation and cross-modal reranking for sign language translation. _IEEE Transactions on Multimedia_, 24:2662-2672.
* Zhao et al. (2024) Rui Zhao, Liang Zhang, Biao Fu, Cong Hu, Jinsong Su, and Yidong Chen. 2024. Conditional variational autoencoder for sign language translation with cross-modal alignment. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 19643-19651.
* Zhou et al. (2023) Benjia Zhou, Zhigang Chen, Albert Clapes, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. 2023. Gloss-free sign language translation: Improving from visual-language pretraining. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 20871-20881.
* Zhou et al. (2021a) Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021a. Improving sign language translation with monolingual data by sign back-translation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 1316-1325.
* Zhou et al. (2021b) Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. 2021b. Spatial-temporal multi-cue network for sign language recognition and translation. _IEEE Transactions on Multimedia_, 24:768-779 

Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. 2024. Streaming dense video captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 18243-18252.
* Zhu et al. (2023) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis. _arXiv preprint arXiv:2304.04675_.

 



## Appendix

In this Appendix, we first provide additional implementation details in Section A. Then, Section B provides more details about the sign language dataset used in this study, including its statistics. In Section C, we present further experimental results. Finally, in Section D, we discuss the feasibility of existing Vision-Language Models (VLMs) in the SLT domain.

## Appendix A More Implementation Details

### Components of SpaMo.

In the SA module, we utilize two distinct linear projection layers tailored for the output feature of ME and SE. For short-term modeling, we employ a 1D TCN configured with a specific sequence of layers: \(\{K5,P2,K5,P2\}\), where \(K_{\sigma}\) represents a kernel size of \(\sigma\), and \(P_{\sigma}\) indicates a pooling layer with a kernel size of \(\sigma\)Hu et al. (2023). To integrate features into the LLM's embedding space, we leverage an MLP cross-modal connector Liu et al. (2024), projecting the features into a 2048-dimensional space.

### Prompt Template.

To focus the LLM on the SLT task, we employ a specific prompting strategy. Our prompt includes a clear instructive prompt: "Translate the given sentence into German." Following this, we incorporate multilingual translations via a translation engine such as Google Translate5, which are sampled from the training set. These translations are included to facilitate In-Context Learning (ICL) Brown et al. (2020). The prompt is structured as follows: "Translate the given sentence into German. [SRC] = [TRG]." Here, the source input (e.g., a sentence in French) serves as the foreign language example, and the corresponding response is the translation into the target language (e.g., German, as used in PHOENIX14T). An example of this prompt structure is provided in Table 8. To ensure that the LLM does not directly access the target translations during training, we shuffle the translation samples so that they do not match the target translation. At test time, we select a translation pair from the training set to use as a reference.

Footnote 5: [https://cloud.google.com/translate](https://cloud.google.com/translate)

### Training.

For training, we use the AdamW optimizer Loshchilov and Hutter (2017), with \(\beta_{1}=0.9\), \(\beta_{2}=0.98\), and a weight decay of \(0.01\). The learning rate schedule includes a cosine decay with a peak learning rate of 1e-4 and a linear warmup of over 10K steps, with a minimum learning rate of 5e-5. We train our model for 40 epochs, using a single NVIDIA A100 GPU, completing the entire process within 24 hours.

### Evaluating Process with KDEs.

To evaluate the quality of the learned representations, we utilize Kernel Density Estimation (KDE) to estimate the probability density functions of the embeddings from GFSLT-VLP and ours. Due to different dimensionality between these methods (1,024 vs. 2,048), we run Principal Component Analysis (PCA) to reduce the number of dimensions while retaining the most significant variance components. This dimensionality reduction facilitates more efficient and stable KDE fitting. KDE can be expressed as:

\[f_{\text{kde}}(\mathbf{z})=\frac{1}{nh^{d}}\sum_{i=1}^{n}K\left(\frac{\mathbf{ z}-\mathbf{z_{i}}}{h}\right), \tag{3}\]

where \(\mathbf{z_{i}}\) denotes the representation points, \(K\) denotes the kernel function, \(h\) is the bandwidth parameter, \(d\) is the dimensionality of the data, and \(n\) is the number of data points.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Methods** & **B1** & **B2** & **B3** & **B4** & **RG** \\ \hline Ours (w/o LoRA) & 46.11 & 32.65 & 24.69 & 19.67 & 42.91 \\ Ours (w/ LoRA) & **49.80** & **37.32** & **29.50** & **24.32** & **46.57** \\ \hline \end{tabular}
\end{table}
Table 10: Ablation on our method with and without LoRA.

\begin{table}
\begin{tabular}{c l c c c} \hline \hline Sign Video Input: & [Characterted Sign Feature] \\ \hline Instruction: & Translate the given sentence into German. \\  & Soft it’s possible filter and in the switchmen \\  & low momentum ranges-after route in data collection \\  & midstepwise at boundaries. \\ \multirow{4}{*}{In Context Examples:} & La helpful eduate code capable all up has no consensus \\  & modulations don’t ask-short swine in den satisfaction \\  & multilegrapeters in bootstrapping. \\ \cline{2-4}  & Le gid and scul spacible havs or dans les chaires \\  & de moangnees, places du aub-and-adieu ia in den satisfaction \\  & mintegeborgen stu bedontront maglius. \\ \hline \hline \end{tabular}
\end{table}
Table 8: An example of prompt used in this paper.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Visual Encoders (SE + MF)** & **B1** & **B2** & **B3** & **B4** & **RG** \\ \hline DINOv2 + VJEQA & 45.67 & 32.94 & 25.27 & 20.35 & 41.32 \\ DINOv2 + VJEQMAE & 47.31 & 34.60 & 26.90 & 21.86 & 42.50 \\ CLIP + VJEQA & 47.82 & 47.17 & 26.76 & 21.66 & 43.68 \\ CLIP + VideoMAE & **49.80** & **37.32** & **29.50** & **24.32** & **46.57** \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation study on various combinations of visual encoders. The results are with VT-Align.



The entropy of KDE is then calculated as:

\[H=-\sum_{i=1}^{n}f_{\text{kde}}(\mathbf{z_{i}})\log f_{\text{kde}}( \mathbf{z_{i}}), \tag{4}\]

where \(H\) represents the entropy, and \(f(\mathbf{z_{i}})\) are the estimated density values at the representation points.

### Generating Visual Tokens

Inspired by the reverse engineering Ju et al. (2023), we first compute the Euclidean distance between the sign feature \(Z_{sm}\) and the LLM's embedding table \(E_{llm}\in\mathbb{R}^{V\times d^{\prime}}\), where \(V\) represents the vocabulary size. Each sign feature is then mapped to the word associated with the shortest distance in this space. This process can be expressed as dist\((Z_{sm},E_{llm})\leq\Delta\), where dist\((\cdot)\) denotes the Euclidean distance function, and \(\Delta\) represents the shortest distance to \(E_{llm}\) across all sign features.

## Appendix B Statistics of Sign Language Datasets

Table 11 presents a comparative overview of three popular sign language datasets: PHOENIX14T, CSL-Daily, and How2Sign, each with distinct statistics and domain.

PHOENIX14T focuses on German Sign Language (DGS) within the specific domain of weather forecasting, featuring a relatively small vocabulary of 3K words and a concise average video length of 116 frames. It includes 7,096 training samples, 519 validation samples, and 642 test samples, with gloss annotations available. This dataset is tailored for domain-specific tasks, offering clear and repetitive patterns ideal for translation and recognition within weather-related contexts.

In comparison, CLS-Daily, a dataset for Chinese Sign Language (CSL), covers a broader range of topics than PHOENIX14T, spanning areas such as family life, medical care, school, banking, shopping, and social interactions. It features a vocabulary of 2K words and an average video length of 119 frames. The dataset includes 18,401 training samples, 1,077 validation samples, and 1,176 test samples, also with gloss annotations.

On the other hand, How2Sign focuses on American Sign Language (ASL) in the instructional domain. It offers a significantly larger and more diverse dataset, with a vocabulary of 16K words and an average video length of 173 frames. The dataset consists of 31,128 training samples, 1,741 validation samples, and 2,322 test samples, but lacks gloss annotations. The diversity and complexity of How2Sign make it particularly suitable for general sign language related tasks, especially those that involve understanding varied and intricate sign sequences.

## Appendix C More Experiments

### Effect of Visual Encoders.

We assess the effect of various combinations of visual encoders (SE & ME). Table 9 shows four different encoders: DINOv2 Oquab et al. (2023), CLIP Radford et al. (2021), V-JAPA Bardes et al. (2024), and VideoMAE Tong et al. (2022). The results demonstrate that the combination of CLIP and VideoMAE delivers the highest performance, suggesting potential for further improvement as visual encoders continue to advance.

### Effect of LoRA.

We evaluate the effect of LoRA on the LLM. As illustrated in Table 10, the LLM with LoRA demonstrates superior performance.

### Parameter Comparisons

We present a comparison of various SLT methods, focusing on the presence of visual fine-tuning ("Vis. Fit."), the number of trainable and total parameters, and their performance as measured by the BLEU-4

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} c c c l} \hline \hline
**Dataset** & **Language** & **\#Vocah** & **Train / Valid / Test** & **Avg. No. Frame** & **Gloss** & **Domain** \\ \hline PHOENIX14T Camgoar et al. (2018) & DGS & 3K & 7,096 / 519 / 642 & 116 & ✓ & Weather Forecast \\ CSL-Daily Zhou et al. (2021a) & CSL & 2K & 18,401 / 1,077 / 1,176 & 119 & ✓ & Daily-life \\ How2Sign Duarte et al. (2021) & ASL & 16K & 31,128 / 1,741 / 2,322 & 173 & ✗ & Instructional \\ \hline \hline  & & & & & & \\ \end{tabular}
\end{table}
Table 11: Statistics of three sign language datasets used in this work. DGS: German Sign Language; CSL: Chinese Sign Language; ASL: American Sign Language; Avg. No. Frame: average number of video frames.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
* & **Vis. Fit.*
* & **Triaxialities Params*
* & **Fluvial Paramism*
* & **Bd*
* \\ \hline GSVLP’s Openov et al. (2023) & ✓ & 215,604 & 215,604 & 215,604 & 21,44 \\ HILMN Chen et al. (2020) & ✓ & 215,604 & 215,604 & 215,604 & 21,604 \\ SMPL Mall Hong et al. (2020) & ✓ & 2706,604 & 215,604 & 215,604 & 21,604 \\ Spelling+M Gong et al. (2020) & ✓ & 2706 & 215,604 & 215,604 & 215,40 \\ Spelling+Mong et al. (2020) & ✓ & 270 & 215,604 & 215,604 & 215,604 \\ Spelling+Mong et al. (2020) & ✓ & 270 & 215,604 & 215,604 & 215,604 \\ Spelling+Mong et al. (2020) & ✓ & 270 & 215,604 & 215,604 & 215,604 \\ \hline \hline  & & & & & \\ \end{tabular}
\end{table}
Table 12: Model parameter comparison.
* denotes an estimated value due to the unavailability of public code. “Vis. Fit.” denotes to the visually fine-tuned on sign language datasets.
 

score. As shown in Table 12, our method, SpaMo, achieves the highest BLEU-4 score of 24.32 without the need for the visual fine-tuning. SpaMo requires 22.7M trainable parameters, which is relatively efficient compared to other methods such as GFSLT-VLP (215.6M), Sign2GPT (16M), and FlALLM (>705.6M). This demonstrates that SpaMo effectively balances model complexity and training efficiency to achieve superior performance without the additional step of the visual fine-tuning.

### More Qualitative Results

We provide additional translation examples for PHOENIX14T, CSL-Daily, and How2Sign. As shown in Table 14, in PHOENIX14T, our method consistently delivers accurate translations, while GFSLT-VLP struggles to capture the correct semantic meaning.

In CSL-Daily, we present a comparison between glosses and visual tokens, as well as between reference translations and generated translations. As shown in Table 15, most visual tokens are matched to the glosses, though they are not perfectly aligned. Notably, in the last three examples, the visual tokens include words that are missing from the glosses but appear in the reference translations.

For How2Sign, Table 16 presents translation results along with their corresponding visual tokens. Since How2Sign lacks gloss annotations, we include actual sign frames for qualitative comparison. Similar to the results on PHOENIX14T and CSL-Daily, many visual tokens in How2Sign are closely aligned with the translations. Note that although OpenSLT Tarres et al. (2023) is the only publicly available baseline6, we were unable to reproduce their results due to a broken link to the fine-tuned I3D features at the time of drafting.

Footnote 6: [https://github.com/imstge-upc/slt_bow2sign_vicx9k23](https://github.com/imstge-upc/slt_bow2sign_vicx9k23)

### Cosine Similarity Between Visual Tokens and Glosses

We use Sentence-BERT Reimers and Gurevych (2019) to evaluate the similarity between the generated visual tokens and the ground-truth glosses from PHOENIX14T, using cosine similarity. Additionally, we assess the similarity between the visual tokens and the target translation, as well as the alignment between the glosses and the translation, which highlights varying degrees of correspondence.

As shown in Table 13, and as we expected, the highest similarity occurs between glosses and target translations, indicating a strong semantic correspondence. By contrast, the visual tokens show lower similarity to both target translations and glosses, suggesting that they are not random. The reason for the score not being higher is likely the inclusion of unrelated words in the visual tokens compared to the actual glosses, as illustrated in Table 7.

### Performance Curve Across Epochs

Figure 6 shows the performance curves of SpaMo using BLEU-4 and ROUGE over 40 training epochs on the PHOENIX14T dataset. For comparison, we note that other models such as SignLLM, Sign2GPT, and Fla-LLM are trained for 20, 100, and 75 epochs, respectively. These results highlight the progressive improvements in SpaMo's performance as training advances, offering a detailed look at its efficiency relative to other models.

## Appendix D Feasibility of Existing VLMs in SLT

Recent advancements in Vision-Language Models (VLMs) Alayrac et al. (2022); Li et al. (2023); Liu et al. (2024b); Cheng et al. (2024) have enabled LLMs to comprehend various modalities including images and videos, beyond just text. However, in the SLT domain, current VLM designs are not well-suited for processing long sequences of sign videos. As shown in Table 11, the average sign video length exceeds 116 frames, which is significantly longer than typical action recognition or video-text datasets, where sample lengths are often under 16 frames. For example, Flamingo Alayrac

\begin{table}
\begin{tabular}{l c} \hline \hline
**Pairs** & **Cosine Similarity** \\ \hline   Gloses \& Target Translations & 0.8400 \\   Visual Tokens \& Target Translations & 0.6781 \\   Visual Tokens \& Glasses & 0.6779 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Cosine similarity between the visual tokens and the glosses.

Figure 6: Performance curves across epochs.

 

et al., 2022), a widely recognized vision-language model, uses 8 frames during training and only 32 frames during inference--far fewer than what is required for SLT. Similarly, VideoLlama2 Cheng et al., 2024) employs 8 frames for training. Moreover, recent LLM-based SLT methods Wong et al., 2024; Gong et al., 2024), including our method, can be classified within the VLM family but are specialized to process and capture long sign video sequences.

 

* [150] M. C. Bader, A. D. Kessler, and J. S. H. Friedman, _Theoretical Population of Population Biology: An Introduction to the Theory of Population Biology_. Springer Science Publishing Company, New York, NY, USA, 2007.
* [151] R. G. Feller, _A Survey of Population Biology_. Oxford University Press, Oxford, UK, 2008.
* [152] P. W. Anderson, _The Population Biology_. Cambridge University Press, Cambridge, MA, 1964.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{r r l} \hline \hline  & Gloss: & \\ Vis. Token: & \\ Vis. Token: & \\ Ref: & \\ Ours: & \\ \hline \multirow{2}{*}{Vis. Token:} & \\ Vis. Token: & \\ Res. & \\ Ours: & \\ \hline \multirow{3}{*}{Vis. Token:} & \\ Vis. Token: & \\ Vis. Token: & \\ Ours: & \\ \hline \multirow{4}{*}{Vis. Token:} & \\ Vis. Token: & \\ Vis. Token: & \\ Vis. Token: & \\ Ours: & \\ \hline \end{tabular} 
\begin{tabular}{r r l} \hline \hline  & Gloss: & \\ Vis. Token: & \\ Vis. Token: & \\ Ours: & \\ \hline \multirow{4}{*}{Vis. Token:} & \\ Vis. Token: & \\ Ask.





\begin{table}
\begin{tabular}{r l|l} \hline  & Image: \\ Vis. Token: & **AGAIN** & SOMEONE & ONE \\ Ref: & again, one more time we’ll show it for you. \\ Ours: & again, one more time. \\ \hline \end{tabular}

Image: \\ Vis. Token: & **LITTLE** & MORE HOW \\ Ref: & a little bit more then this maybe. \\ Ours: & a little bit more about it. \\ \hline \end{tabular}

Image: \\ Vis. Token: & **NON** GO TODAY to TAKE LITTLE THREE Seven Four WEDI OUTILITIESHERE JULlet VORSCHRITEN and we’re going to take a little weed out here. \\ Ours: & now we’re going to take a little bit of the weed out here. \\ \hline \end{tabular}

Image: \\ Vis. Token: & **MANY** PEOPLE & NOT other & UNDERAND thought \\ Ref: & many people don’t understand. \\ Ours: & many people don’t understand that. \\ \hline \end{tabular}

Image: \\ Vis. Token: & **LTRACTICE** WHEN WITH B FOAMERS CAST was so OROU CAN know IF OR GROUP

Ref: & i practice with the barton oaks dental group. \\ Ours: & i practice with the barton oaks tennis team. \\ \hline \end{tabular}

Ref: & so, let’s keep doing the same thing with the arms. \\ Ours: & so, let’s keep doing the same thing with the arms. \\ \hline \end{tabular}

Ref: & here, two, three, four, elbow and follow wherever you're going to go, like the knee to the point and your elbow. \\ Ours: & here, two, three, four, fellow through where you’re going to want to squeeze, woo, woo, your elbow. \\ \hline \end{tabular}

\begin{tabular}{c c} Ref: & nruxnt our next segment to learn more about natural beauty products. \\ Ours: & watch our next segment and we’ll talk a little bit more about natural beauty products. \\ \hline \end{tabular}

Ref: & remember, be careful when doing your home remedies, and if you’re not sure, check with your local professional. \\ Ours: & remember very carefully when doing your home remedies if you have a cell phone. \\ \hline \end{tabular}

\begin{tabular}{c c} Ref: & you can start to rotate your shoulders and start to get more comfortable with your feet by turning. \\ Ours: & you can start rotating your shoulders and start getting comfortable with your five by rotating. \\ \hline \end{tabular}

\begin{tabular}{c c} Ref: & ii, rn johanna krynytzky with hip expressions belly dance studio in st. Petersburg, florida. \\ Ours: & hi, rjohanna krynytzky with hip expressions helps dance studio in st. Petersburg, florida. \\ \hline \end{tabular}

\begin{tabular}{c c} Ref: & i’m going to show you how to do some step-touch side foot work for belly dancing. \\ Ours: & i’m going to show you some step touch side and medium rock for help dancing. \\ \hline \end{tabular}
\end{table}
Table 16: Translation results on the How2Sign test set. Correctly translated 1-gram matches are highlighted in blue. Exact visual token matches within the translation are highlighted in green.