Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
Authors: Ryan Wong
Necati Cihan Camgoz
Richard Bowden
Venue: ICLR 2024 poster (2024)
Keywords: sign language translation
sign recognition
large language models
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Automatic Sign Language Translation requires the integration of both computer vision and natural language processing to effectively bridge the communication gap between sign and spoken languages. However, the deficiency in large-scale training data to support sign language translation means we need to leverage resources from spoken language. We introduce, Sign2GPT, a novel framework for sign language translation that utilizes large-scale pretrained vision and language models via lightweight adapters for gloss-free sign language translation. The lightweight adapters are crucial for sign language translation, due to the constraints imposed by limited dataset sizes and the computational requirements when training with long sign videos.
We also propose a novel pretraining strategy that directs our encoder to learn sign representations from automatically extracted pseudo-glosses without requiring gloss order information or annotations.
We evaluate our approach on two public benchmark sign language translation datasets, namely RWTH-PHOENIX-Weather 2014T and CSL-Daily, and improve on state-of-the-art gloss-free translation performance with a significant margin.

Published as a conference paper at ICLR 2024
SIGN2GPT: LEVERAGING LARGE LANGUAGE MOD- ELS FOR -LOSS-FREE SIGN LANGUAGE TRANSLA- TION
Ryan Wong', Necati Cihan Camgoz", Richard Bowden' University of Surrey, Meta Reality Labs fr. wong, r. 2.bowdenJesurrey.c ac. uk, neccamêneta.com
ABSTRACT
Automatic Sign Language Translation requires the integration of both computer vision and natural language processing effectively bridge the communication gap between sign and spoken languages. However, the deficiency in large-scale training data to support sign language translation means we need to leverage re- sources from spoken language. We introduce, Sign2GPT, a novel framework for sign language translation that utilizes large-scale pretrained vision and lan- guage models via lightweight adapters for gloss-free sign language translation. The lightweight adapters are crucial for sign language translation, due to the constraints imposed by limited dataset sizes and the computational requirements when training with long sign videos. We also propose novel pretraining strategy that directs our encoder to learn sign representations from automatically extracted pseudo-ge -ooioo-gggsssssss without requiring gloss order information or annotations. We eval- uate our approach two public benchmark sign language translation datasets, namely RWTH TWTHH-PHOEIXX-- Weather 2014T and CSL-Daily, and improve on state- of-the-art gloss-free frese translation performance with significant margin.
INTRODUCTION
Sign languages are the primary form of communication for millions of Deaf individuals. Sign lan- guages make use of complex visual gestures as form of communication (Braem & Sutton-Spence/ 2001]. Automatic sign language translation is a challenging task for both natural language process- ing and computer vision, as it requires understanding of both sign and spoken language semantics.
Many prior studies have heavily relied gloss annotations means to achieve better translation performance (Camgoz et al., 2018) 2020b, Ye et al., 2023 Yao et al. 2023 Zhang et al. (023a), where gloss annotations represent written descriptions of signs. These glosses provided in sign order, aiding in the regularization of models to learn valuable sign features necessary for continu- sign language recognition and translation. However, creating datasets with gloss annotations is resource-ing interrvve and time-consuming, Hence, recent trend is to shift towards gloss-free sign language translation (Camgoz et al., 2020aa Yin al. 2023 Zhou et 2023), which the pri- mary focus of our paper. Gloss-free sign language translation poses significant challenges due to the unique grammatical structures and vocabulary of sign languages. Limited data availability, individ- ual signing style variation and long sign videos add to the complexity of the task.
In this paper, we propose Sign2GPT for sign language translation to address the aforementioned challenges. As depicted in Figure T Sign2GPT utilizes pretrained large vision (Oquab et al. 2023) and language models (Lin et al. (021), benefiting from their extensive training on large-scale datasets to improve sign language translation performance. We also present novel pretraining strategy before the translation task, which encompasses two key components. Firstly, we develop an introduce prototype
algorithm designed to automatically generate pseudo-glosses. Secondly, driven method for pretraining the sign encoder using the pseudo-glossee Notably, our approach eliminates the necessity of manual gloss annotations as well as glosses to be in sign order.
To combat over-fitting and address memory constraints resulting from long video sequences dur- ing training, adopt a strategy of freezing the external pretrained models and utilize specialized

---

Published as a conference paper at ICLR 2024
Pretraining Stage
Pseudo Sreudo-Closs -loos Pretraining
Downstream Translation Stage
Sign Features
und nun die wettenvorhensage.
pseudo glosees
Run
wettervorhersage für
Sign Encoder
morgen samslag
Spatial Features
GPT
zwelten
(XGLM)
oklober
Spatial Model
und die wateervorhersage
morgen samstag den zweiten oktober
(Dino-V2 VIT)
<bos> und nun die
Key:
frozen weights
trainablle weights
Overview of Sign2GPT, which consists of a pretraining stage that makes of pseudo- glosses and downstream translation that leverages a frozen GPT model.
Figure 1:
low-rank adapters. Inspired by Hu al. [2021) we employ low-rank adapters method for fa-
cilitating the adaptation of frozen Generative Pretrained Transformers (GPT) (Radford et al. 2018
Lin et al. 2021) as well Vision Transformer (ViT) (Dosovitskiy et 2020 Oquab al. 2023
to the specialized domain of sign language. Our contributions can be summarized as follows: (1) We introduce an end-to-end gloss-free sign language model, Sign2GPT, designed for sign language translation, leveraging frozen GPT language model, (2) We propose novel pscudo-gloss pretrain- ing strategy, utilizing automatically extracted pseudo-glosse from sentences to pretrain the sign en- coder. (3) Sign2GPT demonstrates significant performance improvements over previous gloss-free sign language translation approaches, offering a promising pathway for adapting frozen language and vision models to the domain of sign language translation.
RELATED WORK
Isolated Sign Recognition (ISR), the task of identifying individual signs within sign video, has been extensively studied in the literature. Many ISR approaches leverage deep learning, adapting action recognition models (Carreira & Zisserman [017 Tran al., 2018 Yan et al.) 2018 Diang
et al. 2021 Albanie et al.) 2020) on datasets such as WLASL (Li et al. (020a). MSASL (Joze & Koller 2018) and AUTSIL (Sinean Keles, 2020). New approaches have explored the integration of linguistic priors to enhance the accuracy of ISR models (Zuo et al. 2023 Wong et al. 2023). This is achieved through the utilization of pretrained word embeddings (Joulin et al. 2016) as features to regularize the models during training.
While these methodologies have been successful for ISR, they address only a portion of the sign lan- guage translation challenge. Consequently, new datasets for Continuous Sign Language Recognition have been developed, such as RWTH--PHOENIX - 2014 (Koller et al. 2015) and CSL-Daily (Zhou et al. 2021). These datasets consist of sequences of coarticulated signs continuous video, accompanied by target labels known as glosses. These glosses are arranged in sign order and serve as an intermediary representation bridging the gap between sign videos and spoken language trans- lation. Many approaches use Connectionist Temporal Classification (CTC) loss GGrves et al. 2006
localize and recognize glosses within sign videos (Camgoz et al. 2018 2020b Zhou al. 2021
Cui et al. 2017 Zheng et al. (023) but this relies on the monotonicity of labels and content.
Automatic Sign Language Translation (SLT) is one the main goals of computational sign language research, aimed at bridging the communication gap between sign languages and spoken languages. SLT datasets, such as RWTH-PHOENIX-WE -eether 2014T (Camgoz et al.) 2018) and CSL--Daily (Zhou et al.) 2021I, contain video sequences of signers and corresponding spoken language sen- tences. These videos are often much longer than action recognition datascts, which typically use a fixed-length short sequence as input (Soomro al. 2012 Carreira & Zisserman, 2017).
SLT is generally divided into gloss-based based and gloss-free translation. Gloss-based SLT involves translating sign language into spoken language with the aid of gloss annotations via the utilization of the aforementioned Continuous Sign Language Recognition methods. The gloss annotations are manually annotated by expert annotators and depict sign type and order. Initial approaches used
2

---

Published as a conference paper at ICLR 2024
CNN and RNN-based models (Camgoz al., 2018). but transformers have gained prominence in neural machine translation (Vaswani et al. 2017] ) leading to the development of Transformer based
approaches for SLT (Camgoz et al. 20200 at Chen et al. 2022b Gan et al.) Zhang al. 2023a).
Further improvements by progressive pretraining of large language models are also achieved by using manually annotated sign-ordered gloss as supervision (Chen et al. 2022a)..
Recently, researchers have explored gloss-free sign language translation to avoid the dependencies of manually created gloss annotations. Approaches of finding glosses include using sign spotters (Sincan et al., 2023) Shi et al., 2022] Momeni et al., 2022), but these models are limited to the domain of the sign datasets and the vocabulary was trained on. We take an alternative approach by automatically creating aseudo-glosses from the spoken language sentences. By using sentences and sign language priors, we automatically generate pseudo-glosses overcoming limitations associated with pretrained sign spotting models and manually annotated labels. While techniques for verifying the existence of words exist (Zhao et al. 202T), they utilize separate models for each vocabulary item and two-stage translation system, incurring high training costs. In contrast, we propose singular model that efficiently determines both the localization and existence pseudo-glosses. This model serves the foundation for end-to-end solution in sign language translation.
Other methods of gloss-free free SLT make use of domain transfer through the integration of linguisti- cally pretrained models to enhance SLT performance by using pretrained encoder-decoder models like mBart (Liu et al., 2020) and T5 (Raffel et al., 2020) models to improve translation quality (Zhou eet al.] 2023] Uthus al. (023). We instead use frozen Generative Pretrained Transformer (GPT) (Lin et al.) 2021| Radford et al. 2018) as the decoder to enhance translation performance, rather than fine-tuning or progressively pretraining encoder-decoder language models used in previous ap- proaches.
Recently Low-Rank Adapters (Hu al. 2021). which update only small number of weights, have been proposed to adapt frozen LLMs to spoken language tasks. They have also been applied to multi-modal learning for general visual understanding related tasks (Zhang et al. 2023c Gao
et al. 2023 Zhang et al., 2023b). These approaches use features extracted from CLIP (Radford ett al. 2021) which are not suitable as sign language representation due being trained on image
captions. Our proposed pretraining approach focuses visual-linguisti sign language features and subsequently adapting them to spoken language models, harnessing the capabilities of LLMs.
METHOD
Our approach aims to leverage the linguistic knowledge inherent in LLMs to enhance Sign Lan- guage Translation (SLT). We address two main challenges: handling memory-intensive sign lan- guage videos with numerous frames and creating sign representations that seamlessly integrate with LLMs. In the following subsections, we outline our model architecture (Figure [ and training strategy to tackle these challenges and enhance SLT using L.LMs.
3.1 MODEL ARCHITECTURE
Spatial Backbone. Our video-based sign language translation framework relies on the spa- tial model, designed to extract spatial features, Z", from input video frames denoted as X F1,.,,,, with T frames. These features of dimension C, represented as 7' TC are subsequently used by the sign encoder. We employ the Dino-V2 Vision Transformer (Oquab) jet al., 2023), specifically the ViT-S/14 variant. Dino- V2 a self-supervised vision model known for its robust feature extraction capabilities across various visual tasks, making it suitable choice for our framework. Fine-tuning Dino-V V2 is essential for adapting it to the unique characteristics of sign language translation datasets.
The spatial model often involves a significant number of parameters, posing memory and computa- tional challenges. To address this, adopt LoRA (Low-Rank Adapters), lightweight adaptation technique. LoRA has been shown to be effective in finetuning LLMS (Hu et al. (022) and Diffusion models (Roich al. 2022 Ruiz al. 2022 Gal al. 2022). LoRÁ is applied to the top encoder layers, targeting fully connected layers in the MLP and Multi-Head Attention as shown in Figure 2 (left). We utilize the class token's 's output as a feature vector, undergoing linear transformation and
batch normalization to produce 7"
RTT xC feeding it into sign encoder.
3

---

Published as a conference paper at ICLR 2024
Adapted
MLP 4
Norm
Multi-Head
Adapted
Adapted Masked
Zero-Gatad
Attention
Attention
Cross Attention
Norm
Norm
Norm
Sign
Features
Adapted Spatial Layer
Gated
Attention Decoder Layer
Figure 2: Overview of adapting layers in the spatial model layers (left) and decoder layer (right). We make use of adapters that introduce new low-rank weights to blocks shown by the dashed lines while keeping the original pretrained weights frozen.
Sign Encoder. We utilize spatial features, Z", as input to our sign encoder, aiming to learn spatio- temporal sign representations. Our translation model must handle sequences often comprising hun- dreds of frames. To meet this requirement, we designed a spatio-temporal transformer model, draw- ing inspiration from prior sign language translation approaches (Camgoz et al.j [2020b] Yin et al.] 2023). This model incorporates two crucial modifications to enhance efficiency and effectiveness.
Firstly, to address the challenge of processing many frames, we employ temporal downsampling after specific layers within our encoder. This downsampling reduces the temporal dimension from T* to T using strided averaging with kernel size of three and stride of two. This design was chosen to balance between computational efficiency and the preservation temporal information.
Secondly, we use local self-attention with window size of seven, a technique proven to be highly effective in SLT tasks (Yin et al.] 2023). This local attention mechanism is integrated with the temporal downsampling, extending the model's temporal receptive field deeper into the network. This streamlined approach not only conserves memory but also minimizes redundancy, making highly compatible with the subsequent decoder model. The output sign representation is denoted as E T'C, with 7 representing the output temporal dimension which has been downsampled where T = T
Language Decoder. In the decoder, adapt the XGLM model (Lin al. 2021). multilingual GPT Language Model known for its versatility in few-shot learning on text data. We chose the 1.7B parameter variant XGLM balance performance and memory utilization. We draw inspiration
from successful language model adaptation techniques like zero-gated cross-altention and LoRA
which have been used adapt LLMs different textual and multi-model tasks (Goo et al. 2023
Zhang et al. 2023c b
Alayrac et al. 2022).
Before passing the sign features to the decoder, we map our sign features the decoder's dimension with linear layer FCm To adapt the XGLM decoder for sign language translation, we employ the zero-gated multi-head cross-attention hancement shares weights from the pretrained masked multi-head attention and integrates separate LoRA for masked multi-head attention (A (Adapted Masked Attention) and cross-attention (Zero-Gated Cross Attention). The sign features are first passed through the frozen decoder's 's layer normaliza- tion and then used as keys for cross-attention. To integrate sign features without overshadowing linguistic features, employ gated scaled dot-product attention into the cross- cross-attention.
the decoder layer as shown in Figure 2 (right). This en-
QKT VoT
GatedAAtennion((,,KK
g x softmax
K and represent the inputs from the key and value derived from the sign features, while Q orig- inates from the textual features. g is learnable gate parameter for each attention head which is clamped between and I and initialized to zero to preserve linguistic knowledge at the start of train- ing. This modification allows XGLM to adapt seamlessly sign language. The gate parameter and
4

---

Published as a conference paper at ICLR 2024
LoRA parameters drive this adaptability, gradually incorporating sign features while leveraging lin- guistic knowledge. As depicted in Figure (right, the outputs from the Adapted Masked Attention and Zero- Pated Cross Attention summed together.
(right),
3.2 TRAINING STRATEGY
Our model architecture (Figure I allows direct video-to-text training for SLT. We prioritize high- quality sign features by concentrating most the trainable parameters in the sign encoder. We freeze the pretrained vision and language model and use the adapters for sign language domain transfer. This approach ensures our model's primary focus on capturing and utilizing sign language features and then leverages the language model's linguistic ability adapt the features to spoken language translation. The result is an architecture tailored to SLT, enabling effective translation from video input to textual output. our framework, the spatial model and decoder are pretrained from large-scale datasets while the sign encoder is randomly initialized. We therefore develop gloss-free free strategy to pretrain the sign encoder.
3.3 PRETRAINING STAGE
Pseudo-gloss generation. To perform pseudo-gloss generation, we extract pseudo-gloss from each spoken language sentence using the spaCy natural language processing library (Honnibal et al.) 2020 In the case of German (Phoenix 14T), this involves lemmatization, while for Chinese (CSL- Daily), it encompasses word segmentation. The preprocessing step enables the extraction of the base forms of words.
Following this step, we apply Parts-of-Speech (POS) tagging, retaining only words categorized as ("NOUN", "NUM", "ADV", "PRON", "PROPN", "ADJ", "VERB"). This filtering process pri- oritizes words that most likely to convey meaningful information, ensuring that extracted useudo-glosses are semantically relevant, retain most the sentence context and have potential gloss correspondence. The lemmatized words (in German) segmented words (in Chinese) that satisfy this filter are considered as pseudo-glosses. It's important to highlight that our pseudo- glosses are in spoken language order, unlike manually annotated glosses which are in sign order. Consequently, conventional approaches such as the CTC loss (Graves et al. 2006) are not suitable for our pseudo-glosses.
P P.O
Prototypas
TV
softmax!!
g
Fealures
Sign
Lineaz
Feat
Mapped
8000s wis
B
Z Txc
Z' TXD
e zhx
softnax?
E TxU
B n
Figure 3: Overview of pretraining process, which takes the sign features input and predicts the existence of pseudo-glosses.
Pseudo-gloss pretraining, Our goal is to enable the sign encoder to learn visual-linguisee rep- resentations. CLIP's ability to unify visual and linguistic domains (Radford al. 2021 motivates the design of our sign encoder to develop sign features using pseudo-glosses for Sign Language Translation (SLT), as illustrated Figure
We generate prototypes for each pseudo-gloss. The aim is to ensure that the sign encoder generates representations that closely align with these prototypes when they are present within the sign video. These prototypes initialized with word embeddings obtained from fastText (Joulin et al. (016)), each of dimension D 300. Additionally, we include an extra prototype initialized with zeros, which serves the prototype for sign transitions non-sign-related components. As result, the prototype matrix takes the form P DxU
The sign features Z {20, 21, ...: 24 .... 2r) are learned to be aligned with the prototypes if the associated pseudo-gloss exists within the sign video. We therefore project Z to E RT* through linear layer and compute the cosine similarity between the prototypes and each of the projected
5

---

Published as a conference paper at ICLR 2024
sign features such that:
2/. |||
= sim(si..))))) P)
(2)
which is the cosine similarity score for the 7th feature index in I and S = (80 8 81,..., 8T) E _Txx -1 f 8, Σ 1. High scores indicate similarity, and low scores indicate dissimilarity. We then introduce temporal probability (5T) and prototype probability (SU) scores using temperature-scaled
softmax operations across the time and prototype axis:
ST = (S/T)) Se = softmax:(S/TT)
(3)
(4)
These scores emphasize temporal and class-related aspects of similarity and ensure they fall within the range of to 1. Learnable scaling factors, TT and TV. modulate the extent of temporal and class similarity, allowing control of their influence on prototype creation. Element-wise multiplication of S and S yields E' E TT' forming the basis for prototype localization. To discern the presence or absence of prototypes, we aggregate E values over the temporal dimension, resulting in E E R such that:
T E.J
T
E
Σ
(S x S%
(5)
1=0
where 0 S E., S. 1. High E, value for the jth prototype indicates the presence of the prototype, while low values signify the absence. We employ binary cross-entropy loss to train the model, optimizing the presence or absence of prototypes. For our case each prototype is assigned as if the pseudo-gloss exists within the sign video and 0 if not.
Note that during pretraining the learned sign representations are temporally invariant, therefore we also add sinusoidal positional encoding before FC, for the translation task. The resulting pretraining allows the weights to be initialized with vision priors (spatial model), sign priors (sign encoder) and linguistic priors (decoder). This enables to then these models for the downstream translation task.
4 RESULTS
4.1 DATASETS AND EVALUATION PROTOCOL.
We evaluate our approach using two distinct sign language translation datasets. RWTH- PHOENIX-WEATHER-220
(Phoenix14T) (Camgoz et al. 2018) is a German Sign Language dataset for sign to spoken language translation tasks. encompasses translations from German Sign Language to the German language, derived from German weather broadcast videos. CSL-Daily (Zhou et al. 2021 is translation dataset focusing on Chinese Sign Language to Chinese. It com- prises of lab recorded videos that cover various daily interaction topics, including travel, family life, bank service and shopping.
We evaluate the translation performance using standard metrics commonly employed in sign lan- guage translation. These metrics include BLEU (Bilingual Evaluation Understudy) (Papineni et al. 2002) and ROUGE-L (Lin 2004) scores. BLEU measures the similarity between machine-gent generated and reference translations based on n-gram overlap. ROUGE-L assesses translation quality by cal- culating the length of the longest common sub-sequence between generated and reference texts.
4.2 TRAINING SETTINGS
The model is trained end-to-end with batch size of 8 on two A100 GPUs, subsampling every second frame. Due to memory constraints we apply spatial adapters to the top 3 layers of the spatial model. The sign encoder is 4 layer transformer with hidden dimension of 512, 8 attention heads and intermediate size of 2048. The temporal downsampling is applied after the 2nd layer. Training is conducted in bfloat16 and Flash attention v2 (Dao 2023) is used to optimise memory usage. We explain further details of the libraries and models in Appendix A.I. We employ the Adam optimizer Kingma & Ba 2014) with learning rate of x 10 and weight decay of 0.001. Training spans

---

Published as a conference paper at ICLR 2024
100 epochs with gradient clipping of 1.0 and includes one-cycle cosine learning rate scheduler [Loshchilov & Hutter 2016) with warmup for the initial epochs. Data augmentation techniques, such color jitter, random resized cropping from 256 x 256 to 224 224 pixels, frame rotations and horizontal flips are consistently applied across all frames within video sequences. During evaluation, center cropping to 224 x 224 pixels.
Pretraining. We initialize the prototype (Tc) and time temperature (Ty) to 0.1. In Table Ta), present the counts of extracted pseudo-glosses for each dataset using the approach described in Section 3.3 For the CSL-Daily dataset, the number of pseudo-glosses significantly exceeds the vocabulary size. This discrepancy arises because the vocabulary is based on Chinese characters, while pseudo-glossee represent word segmentation that convey meaning.
Downstream Translation. We utilize cross-entropy loss with label smoothing training. The LoRA rank and alpha values are both set to 4. During inference, we employ a beam search with a width of 4. In Table [b). we present detailed information regarding the number of trainable parameters for each component of our network during the downstream training phase. No- tably, the sign encoder contains a significantly larger number of trainable parameters, emphasizing its role in learning sign features. For the CSL.-Daily dataset, we implement additional tokenization processing to handle unknown tokens, as elaborated in Appendix A.2
0.1 during
Table I:
Training setting with (a) the number of pseudo-glosses glooses during pretraining and (b) parameter counts during downstream translation.
Component
# Params
# Trainable
Spatial
228328,448
271,872
Dataset
# Vocab
# p-glosses
Sign Encoder
12,613,632
12,613,632
Decoder
1,736,710,528
3,803,520
14T
2, 887
2, 2,533
CSL-Daily
2,343
7,918
Total
1,771,652,608
16,689,024
(a) Vocabulary versus gooussssssssss per dataset
(b) Number model parameters during translation
4.3 COMPARISONS WITH STATE-OF-THE-ART METHODS
Results on Phoenix14T. In Table we present comparative analysis between approach and ofate-of-the-art methods for sign language translation on Phoenix141] We observe an approximate improvement of 1.1 BLEU4 on the test when doing pseudo-gloss pretraining (PGP) followed by downstream translation (SignGGPT(W/GP)). Our gloss-free approach demonstrates substantial improvements in BLEU-1,2.3 scores, surpassing the performance of previous gloss-free methods and reduces the performance gap between gloss-free and gloss-based SLT. This progress can be at- tributed our approach's capability to learn representations for individual pseudo-glos in contrast to prior gloss-free free methods that primarily focused on sentence-bar based representations. Notably, even without pretraining (Sign2GPT), our approach remains competitive with previous state-of-the-an (SOTA) gloss-free results. We also find that downstream training with only the trainable decoder parameters and extract sign features from our frozen pretrained stage model, Sign( 2. (2GPT(W/PPP)), is able to achieve competitive results with under 4 million parameters showing the effectiveness of our learnt sign representations.
Results on CSL-Daily. In Table 3 we present the results of our SLT experiments on the CSL- Daily dataset. We observe a substantial performance increase in BLEU-4 compared to previous SOTA gloss-free results, with an approximate improvement of 4.4 BLEU-4 when utilizing our pre- training (SPP)) Sign2GPT(w/PPP) significantly outperforms training without PGP (Sign2GPT), with notable 3.5 BLEU-4 improvement on the CSL-Daily dataset.
4.4 QUALITATIVE RESULTS
We visually demonstrate the effectiveness of our pseudo-glas -gosss pretraining phase for sign localization in and Appendix A.4 To achieve this, we leverage the output values of F,, with dimensions
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of test set results on P4oenix14T. We present our gloss-free results for three experimental settings: (1) Without pseudo-gloss pretraining (Sign2GPT), (2) with pseudo-gloss pretraining (Sin2GGT((/PGP)) and (3) extracted features Z from the frozen spatial and sign encoder model that has been trained with pseudo-gloss pretraining S
Test Set
Method
BLEU1
BLEU2
BLEU3
BLEU4
ROUGE
Gloss-based
3L-Transformer (Camgoz al.
2020b)
46.61
33.73
26.19
21.32
MEL-TI--Trans....
Zhou et al.
2021
50.80
37.75
29.72
24.32
49.54
MMTLB Chen et al. 2022a
53.97
41.75
33.84
28.39
52.65
SLTU NET Zhang et al
2023a
52.92
41.76
33.99
28.47
52.11
TwoStream-SI.I CCen et
2022b)
54.90
42.43
34.46
28.95
53.48
Gloss-free
NSLT Camgoz et al. 2018)
29.86
17.52
11.96
9.00
30.70
TSPNet Li
2020b)
36.10
23.12
16.88
13.41
34.96
CSGCR Zhao et al.
(2221)
36.71
25.40
18.86
15.18
38.85
GASLT Yin et 2023)
39.07
26.74
21.86
15.74
39.86
GFSLT Zhou et al
2023
41.39
31.00
24.20
19.66
40.93
GFSLT-VLP (Zhou al.
2023)
43.71
33.18
26.11
21.44
42.49
Sign2GPT
45.43
32.03
24.23
19.42
45.23
Sign2GPT(w/PGP)
49.54
35.96
28.83
22.52
48.90
Sign(2)2GPI(//PP)
47.06
33.61
25.85
20.93
47.11
Table 3: Comparison of test set results on the CSL-Daily. We present gloss-free results for three experimental settings: (I) Without pseudo-gloss pretraining (Sign2GPT), (2) with pscudo-gloss pretraining (Sinn2GPT(WPPPP), and (3) extracted features Z from the frozen spatial and sign encoder model that has been trained with pseudo-gloss pretraining (Sign( Z 2)2GPT(w/PPP))
Test Set
Method
BLEU1
BLEU2
BLEU3
BLEU4
ROUGE
Gloss-based
SL-Transformer (Camgoz et al.
2020b
37.38
24.36
16.55
11.79
36.74
ON-TIN-Transt..
(Zhou et al.
2021
51.42
37.26
27.76
21.34
49.31
MMTLB
Chen et al.
2022a)
53.31
40.41
30.87
23.92
53.25
SLTU NET (Zhang
2023a
54.98
41.44
31.84
25.01
54.08
TwoStream-SDn
Chen et al.
2022b
55.44
42.59
32.87
25.79
55.72
Gloss-free
GASLT
Yin et al.
2023)
19.90
9.94
5.98
4.07
20.35
NSLT Camgoz et al.
2018
34.16
19.57
11.84
7.56
34.54
GFSLI Zhou al. 2023
37.69
23.28
14.93
9.88
35.16
GFSLT- VEP Zhou et al.
2023
39.37
24.93
16.26
11.00
36.44
Sign2GPT
34.80
24.00
17.27
12.96
41.12
Sign2GPT(w/PGP)
41.75
28.73
20.60
15.40
42.36
Sign(2)2GPT(/PPP))
32.73
20.52
13.75
9.73
33.39
T x U. The values in this matrix vary between 0 (indicating the absence of a sign) and I (indicating the presence sign), reflecting the temporal occurrence of seeudo-glosses, Our observations sug- gest that pretraining approach holds promise for sign spotting, in scenarios where precise localization not available. This aspect may warrant further exploration as a potential avenue for future research.
8

---

Published as a conference paper at ICLR 2024
und nun weltervorhereagee for morgen sametng den zweten oloober
wettervorhersage
samstag
oktober
nun
1b11
wagenvorthessgee
morgen
samolag
22
zoeetrr
30
ollocer
Figure 4: Visualizations of the localization capabilities of our pretraining stage. We visualize only the pseudo-glosses from the target sentence (y-axis) over time (x-axis), with whiter regions indicat- ing a higher probability of the pseudo-glos occurring during the time segment. We also display the localized gloss (under the video frames) based threshold of 0.2 on E.
4.5 ABLATION STUDY
We conduct our ablation studies on the Phoenix 14T dataset, evaluating the BLEU-4 score on the development set. In Table present the results of architectural modifications made to our net- work. For our initial study, we conducted experiments without pretraining Sign2GPT, We observed significant performance improvements when incorporating spatial adapter. Conversely, the use of global attention led to reduction in model performance compared to local attention. Furthermore, we investigated the impact of downsampling on the model's performance. While downsampling showed only marginal improvements in terms of the BLEU-4 score, it notably reduced computa- tional complexity. This reduction is due to only half the number of features being passed through the decoder during training.
Our pretraining generates representations for pseudo-glosses, which inherently removes temporal information from the output sign representations. Subsequently, we investigated methods to reintro- duce temporal information into the model using various approaches as described in Section 3.3 We explored learnable approaches for positional embeddings with both zero and random initialization. However, these approaches yielded minimal to no discernible benefits. In contrast, the utilization of sinusoidal positional embeddings demonstrated substantial performance improvements.
Table 4: Ablation of results the Phoenix 14T dataset showing different architecture changes with no pseudo-gla gloss pretraining (Sign2GPT) and with pseudo-gloss pretraining (Sign2GPT(w/PGP))
Architecture
BLEU4
Sign2GPT
Spatial Adapters Local Attention Downsampling
19.55
No Spatial Adapters
16.38
No Local Attention Global Attention)
18.56
No Downsampling on Sign Encoder
19.30
Sign2GPT(w/PGP)
No positional
21.68
Learnable positional (zero init)
21.89
Learnable positional (random init)
21.16
Sinusoidal positional
23.20
5 CONCLUSIONS
In this paper, we have presented a novel approach to address the challenging problem of Sign Trans- lation in gloss-free setting. Our method, Sign2GPT, demonstrates significant performance im- provements over existing state-of-the-art techniques on the Phoenix and CSL-Daily datasets. We introduce novel pretraining strategy that learns from pseudo-glosses which are generated au- tomatically to learn word-level sign features, thereby allowing our sign encoder to be effectively pretrained without the of manually annotated glosses. Moreover, the proposed Sign2GPT chitecture presents a promising direction for the exploration of fusing visual features to spoken language models for sign language recognition and translation tasks.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To facilitate reprodueibility, we have provided details the training settings in Section 4.2 with additional details of the libraries used for the pretrained models in Appendix [A.] We also give further details of the CSL tokenization and post processing to address issues with unknown tokens in Appendix A.2
ACKNOWLEDGEMENTS
This work was supported by the EPSRC project EXTOL (EP/RO3298X//1) SNSF project 'SMILE II' (CRSII5 193686), European Union's Horizon2020 programme ('EASIER' grant agreement 101016982) and the Innosuisse IICT Flagship (PFFS-21-47). This work reflects only the authors view and the Commission is not responsible for any use that may be made of the information it contains. Neither Necati Cihan Camgoz nor Meta were involved the model training, evaluation, or use of the datasets.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Samuel Albanie, Güil Varol, Liliane Momeni, Triantafyllos Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. Bsl-Ik: Scaling up co-articulated sign language recognition using mouthing cues. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 35-53. Springer, 2020.
P Boyes Braem and RL Sutton-Spence. The Hands Are The Head of The Mouth. The Mouth as Articulator in Sign Languages. Hamburg: Signum Press, 2001.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7784-7793, 2018
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel trans- formers for multi-articulatory sign language translation. Computer Vision-ECCV 2020 Work- shops: Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 301-319. Springer, 2020a.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language trans- Joint end-to-end sign language recognition and translation. In Proceedings of the JEEF/CVF conference on computer vision and pattern recognition, pp. 10023-10033, 2020b.
formers:
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 3120-5130, 2022a.
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream net- work for sign language recognition and translation. Advances in Neural Information Processing Systems, 35:17043-17056 2022b.
Runpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for continu- ous sign language recognition by staged optimization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7361-7369, 2017.
Tri Dao. FlashAttenioon-2: Faster attention with better parallelism and work partitioning. 2023.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 2022.
Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Kang Xia, Lei Xie, and Sanglu Lu. Contrastive learning for sign language recognition and translation.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. Llama- Llama-adapter v2: Parameter-efficiet visual instruction model. arXiv preprint 1.XIv:3304.10110 2023.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuher. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
John Hewitt, Initializing new word embeddings for pretrained language models. https:/nlp. expansion. .httlllloo..mmm 2021.
stanford. edw/ wwwhhw////////////eeooooe htteee/w/vocce //wcaaa-eeppaniio......
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. doi: (0.5281/2enodo.122233 1212303.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Songyao Jiang, Bin Sun, Lichen Wang. Yue Bai, Kunpeng Li, and Yun Fu. Skeleton aware multi- modal sign language recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3413-3423, 2021.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint 11X.v:16/2.03661,, 2016.
Hamid Reza Vaezi Joze and Oscar Koller. Ms-asl: large-scale data set and benchmark for under- standing american sign language. arXiv preprint arXiv: 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Oscar Koller, Jens Forster, and Hermann Ney. Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Computer Vision and Image Understanding, 141:108-1P 125, 2015.
Dongxu Li, Cristian Rodriguez, Xin Yu, and Hongdong Li. Word-level deep sign language recog- nition from video: new large-scale dataset methods comparison. In Proceedings of the IEEE/CVF winter conference applications of computer vision. pp. 1459-1469, 1469, 2020a.
Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hong- dong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign language translation. Advances in Neural Information Processing Systems, 33: 12002-120045 2020b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual language models. arXiv preprint arXiv:2 12. 10668, 2021.

---

Published as a conference paper at ICLR 2024
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Trans- actions of the Association for Computational Linguistics, 8:726-742. 2020.
Ily Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:I608. 003983 2016.
Liliane Momeni, Hannah Bull, KR Prajwal, Samuel Albanie, Gäl Varol, and Andrew Zisserman. Automatic dense annotation of crge-vocabulary sign language videos. In European Conference on Computer Vision, pp. 67 671- Springer, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szaframiec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07192; 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1):55555-5 2020.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1-13, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint ar Xrx:222008. 12242, 2022.
Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. arXiv preprint arXiv:2205. 2022.
Ozge Mercanoglu Sincan and Hacer Yalim Keles. Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access, 8:1813:0:- 11340-181355, 2020.
Ozge Mercanoglu Sincan, Necati Cihan Camgoz, and Richard Bowden. Is context all you need? scaling neural sign language translation to large domains of discourse. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1955-1965, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12. 2012.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6455 2018
David Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain ameri- can sign language-english parallel corpus. arXiv preprint 23X.::2306.11622. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Learnt contrastive concept embeddings for sign recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1945-1954, 2023.
12

---

Published as a conference paper at ICLR 2024
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-basee action recognition. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.
Huijie Yao, Wengang Zhou, Hao Feng, Hezhen Hu, Hao Zhou, and Houqiang Li. Sign language translation with iterative prototype. arXiv preprint arXiv:2308. /2/91, 2023.
Jinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Hui Xiong. Cross-modality data augmen- tation for end-to-end sign language translation. arXiv preprint arXiv:2305. 2023.
Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2551-2561 2023.
Biao Zhang, Mathias Müller, and Rico Sennrich. Sltner:: A simple unified model for sign language translation. arXiv preprint 2xXiv:2305.01111 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.022. 028558 2023b.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuming of language models with zero-init atten- tion. arXiv preprint 2rxiv:2303.16199 2023c.
Jian Zhao, Weizhen Qi, Wengang Zhou, Nan Duan, Ming Zhou, and Houqiang Li. Conditional sentence generation and cross-modal reranking for sign language translation. IEEE Transactions on Multimedia, 24:2662-2672, 2021.
Jiangbin Zheng, Yile Wang, Cheng Tan, Siyuan Li, Ge Wang. Jun Xia, Yidong Chen, and Stan Z Li. Cvt-slr: Contrastive visual-textual transformation for sign language recognition with varia- tional alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -211500, 2023.
Benjia Zhou, Zhigang Chen, Albert Clapés. Jun Wan, Yanyan Liang. Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. arXiv preprini urXiv:2307. 0ete3307.44400 14768, 2023.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign nack-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1316-1325, 2021.
Ronglai Zuo, Fangyun Wei, and Brian Mak. Natural language-assiste assisted sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14890-14900, 2023.
A APPENDIX
A.1 LIBRARIES
Spatial Backbone. The spatial backbone is VIT-S/14 distilled with the pretrained model down- loaded from github. com/ waw.bookeeeercc//chaaaaoo..
Decoder. We the HuggingFace library for the pretrained XGLM transformer model and tok- enization, specifically the pretrained weights from facebook/xging wttoo///ggimimlm
Pseudo-gloss generation. For pseudo-gloss generation for generating our pseudo-glosses for the
the SpaCy library. We use
the German pipeline loaded from de.coree.eewe Phoenix14T dataset. For the CSL-Daily dataset use zh_core wee_webbsr
Fast embeddings. We load the fastText embeddings from cc. de.300 300. bin for the German Phoenix 14T dataset and cc. zh. 300. bin for the Chinese CSL Dataset.
13

---

Published as a conference paper at ICLR 2024
A.2 CSL TOKENIZATION AND POST PROCESSING
By using the pretrained tokenization from XGLM, we encountered unknown tokens during CSL- daily training when Chinese characters were absent from the vocabulary. To address this, we per- formed vocabulary expansion, utilizing the average embedding technique by Hewitt (2021) for ini- tializing new tokens with slight noise. We maintained the consistency of our proposed architecture by keeping these new embeddings frozen throughout training, avoiding the introduction of addi- tional learnable parameters. In total, there are 252 new tokens added to the vocabulary. We also apply post-processing of replacing punctuation tokens inference time the pretrained tokenizer converts punctuation to the Unicode equivalent such that "?" "!", " and " are replaced by "? ! and ; respectively.
A.3 QUALITATIVE TRANSLATION EXAMPLES
In Table and we present randomly selected qualitative translation results on Phoenix147 and CSL-Daily respectively. We also display the extracted pseudo-glosses for each of the target sen- tences which capture the core content of the sentences. Our findings demonstrate that our transla- tion model successfully generates sentences that effectively capture the semantic content of spoken language sentences, albeit with variations in sentence structure.
Table 5: Examples of translation results on the Phoenix 14T dataset.
Hypothesis:
im übrigen land scheint häufig die sonne und es gibt nur wenig schauer. , (in the rest of the country the often shines and there are only few showers .) übrig, gebiet, sonne, nur, locker, wolke
Pseudo-glosses:
Reference:
in den übrigen gebieten viel sonne und ein paar lockere wolken (lots of sun in the remaining areas and only a few loose clouds
Hypothesis:
am tag zwölf grad an der ostsee und bis zu zwanzig grad im süden (twelve degrees a day on the baltic sea and up to twenty degrees the south
Pseudo-glossed
tag, zwölf, grad, ostsee, zwanzig, grad, niederbayer am tag zwölf grad an der ostsee und bis zwanzig grad in niederbayern (on the day twelve degrees the baltic sea and up to twenty degrees in lower bavaria.)
Reference:
Hypothesis:
ich wünsche ihnen noch einen schönen abend und machen sie es gut (i wish you nice evening and do well .)
seudo-glosses,
ich, wünschen, ihnen, schön, abend, machen, sie, gut ich wünsche ihnen einen schönen abend und machen sie es gut wish you nice evening and do well
Reference:
Hypothesis:
der wind weht schwach bis mäßig aus süd bis südost ( (the wind blows weakly to moderately from the south to southeast.)
Pseudo-glosses:
dazu, wehen, schwach, wind, südost, süd dazu weht ein schwacher bis mäßiger wind aus südost bis süd inn addition a weak to moderate wind blows from the southeast to the south .)
Reference:
A.4 QUALITATIVE PSEUDO-GLOO LOCALIZATION EXAMPLES
We visualize the localization capabilities of our pretraining on Phoenix14T in Figure and and CSL- Daily in Figurel 6 We notice that our pretraining has automatic localization capabilities irrespective of the order of the pseudo-glos when using the output E. This has the potential future avenues for automatically creating sign-ordered glosses based thresholds of E. We observe that while provide no information about the localization of the pseudo-glosee glosses, our approach is able to identify the potential glosses and the sign order of these identified glosses.
14

---

Published as a conference paper at ICLR 2024
Table 6: Examples of translation results on the CSL-Daily dataset.
Hypothesis:
a (This place is far from the hotel, just few steps to the door of the hotel.)
i 1e कं A
Pscudo-glosses:
T2 3 / .- /
# ///
Reference:
i $ i2 I2 T - - $ #1 (Okay, there
restaurant not far from here, can be reached in a few minutes" walk.)
Hypothesis:
1 A & # (The company far away, why doesn't 2 ब 1 i
he take taxi?)
:heudo-glosses:
8 / / RC/ /2/ 3 2/ /// ेτ T #
Reference:
(The company far from home, why
doesn't he take taxi?)
Hypothesis:
T .t $ $ 15
(I'm not going climb mountains, have #E
things to do.)
:heudo-glosse:
/ K// t\ f&e 44/ $ म * $ fL .t. $
Reference:
(I'm not going to climb the mountain, have some-
thing to do.)
Hypothesis:
$
(I like snow.)
Pseudo-glosse glosses:
Reference:
$$ FSAÆT.
(I like winter, the snow is so beautiful.)
B ADDITIONAL ABLATION STUDIES
B. IMPACT OF SPATIAL BACKBONE
In Table we present comparisons between the performance of a ResNet18 spatial backbone and the proposed adapted VIT model within Sign2GPT architecture. We find that the frozen ViT model trained with DinoV2 pretraining, incorporating adapters, exhibits slightly better performance compared to the ResNer18 backbone. A noteworthy advantage of the adapted ViT model lies in the application of LoRA instead of fine-tuning all parameters. Consequently, the proposed backbone has under three hundred thousand trainable parameters, significant reduction compared to the 11 million trainable parameters of the ResNet18 model.
Table 7: Ablation of different spatial backbones on CSL-Daily using our Sign2GPT architecture without Pseudo-Glos Pretraining.
Test Set
Spatial Backbone
BLEU4
ROUGE
ResNet18 DinoV2 (ViT-S/14)
12.28
38.31
12.96
41.12
B.2 PRETRAINING PERFORMANCE
In T we demonstrate the performance of our pseudo-gloss pretraining by measuring the pre- cision, recall and F1-score for Phoenix 14T and CSL--Dally using threshold of 0.2.
Table 8: Quantitative results of pseudo gloss pretraining on the sign language datasets.
Dataset
Precision
Recall
F1-Score
Phoenix 14T
0.52
0.39
0.44
CSL-Daily
0.38
0.34
0.36
15

---

Published as a conference paper at ICLR 2024
Impact of pseudo-gloss selection. In Table illustrate the impact of utilizing POS preci- sion, recall, and Fl-score. Notably, when all words are used as tokens, recall significantly decreases from 0.39 to 0.28. This result validates our assertion that not all words have corresponding signs in sign language.
Table 9: Ablation of pretraining results on Phoenix 14T using all words as tokens vs the selected pseudo-glosses with a threshold 0.2
Tokens
Precision
Recall
F1-Score
all words pseudo-glosses
0.55
0.28
0.37
0.52
0.39
0.44
Impact of Sign Encoder. In Table 10 we demonstrate the importance the sign encoder on pretraining. The role of the sign encoder forms important part of the model it learns temporal features.
Table 10: Ablation of pretraining results Phoenix147 with sign encoder and the inclusion of sign encoder.
Precision
Recall
F1-Score
no sign encoder
0.50
0.25
0.33
sign encoder
0.52
0.39
0.44
B.3 IMPACT OF MODEL SIZE.
In Table 11, we demonstrate the results of our approach with the smaller language model XGLM- 564M. The table shows marginal performance reduction the 14T dataset compared to the larger XGLM-1.7B while still outperforming GFSLT-VLP (Zhou et al. 2023) with similar size language model, mBART (Liu al. 2020).
Table 11: Ablation of XGLM backbones Phoenix 14T using Sign2GPT architecture.
Test Set
Backbone
BLEU4
ROUGE
GESLT-VLP (mBART) (Zhou et al. 2023)
21.44
42.49
Sign2GPT(w/ PGP) (XGLM-564M)
22.29
48.21
Sign2GPT(w/ PGP) (XGLM-I.7B)
22.52
48.90
16

---

Published as a conference paper at ICLR 2024
tag zwölf grad bis zwanzig grad niederbayern
tag
ostsee
zwolf
grad
zwanzig
grad
fag
grad
osisee
zwanzig
grad
niederbayer
und nun die wettervorhersage für morgen donnerstag den dritten märz
morgen
wettervorhersage
donnerstag
márz
nun
wetlervorhersage
morgen
donnerstag
dritter
märz
milder bleibt es unter regenwolken im fünffzhhn sechzehn grad
osten
mild
fünfzehn
siebzehn
grad
mild
bleiben
regenwolke
osten
fünfzehn
sechzehn
grad
der südosthälfte deutschlands bleibt morgen leichtem hochdruckeinfluss recht freundlich
deutschland
hochdruckeinfluss
bleiben recht freundlich
südosthálfte
deutschland
bleiben
morgen
leicht
hochdruckeinflsse
noch
freundlich
Figure 5: Visualizations of the localization capabilities of our pretraining stage on the pseudo- glosses from the Phoenix 14T dataset. We visualize only the pseudo-glosses from the target sentence (y-axis) over time (x-axis), with whiter regions indicating higher probability of the pscudo-gla -oosss occurring during the time segment. We also display the localized gloss (under the video frames) based on threshold of 0.2 E.
17

---

Published as a conference paper at ICLR 2024
&&H
BF
E
8
ael itt
FBR
BEN
1AIR
RA
REL TEE
Figure 6: Visualizations the localization capabilities of our pretraining stage on the pseudo- glosses from the CSL-Daily dataset. We visualize only the pseudo-glosser from the target sentence (y-axis) over time (x-axis), with whiter regions indicating a higher probability the pseudo-glas occurring during the time segment. We also display the localized gloss (under the video frames) based on a threshold of 0.2 on F.
18


Reviews:
[{"review_id": "xuUdgn26K7", "reviewer": "Reviewer_3mSZ", "summary": "This paper aims to leverage the large-scale pretrained vision and language models via lightweight adapters for gloss-free sign language translation. Besides, it also proposes a pretraining strategy which make the framework aware of important text information. The experiments are conducted on two benchmarks to validate the effectiveness of the proposed method.", "questions": "See the Weakness section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Leveraging large-scale pretrained model for SLT is sound.\n\nThe paper is well-written and well-organized.\n\nThe overall performance seems promising.", "weaknesses": "The introduction part seems inconsistent with the title. The introduction mentions the both large-scale vision and language model, while the title only mentions the language one. What do the authors want to emphasize?\n\nThe pseudo-gloss pretraining technique shares the similar core idea with CSGCR. The authors should discuss the difference.\n\nUtilization of pretrained model is not new. The author should cite the following work and discuss the difference with it.\nChen Y, Wei F, Sun X, et al. A simple multi-modality transfer learning baseline for sign language translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 5120-5130.\n\nWhat are the performance gains derived from the utilization of large-scale vision and language models? The ablation part should demonstrate it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZHaubIGcu", "reviewer": "Reviewer_DHgv", "summary": "The paper proposes an automatic sign language translation based on large language models. \nThe amount of training data for sign language is limited, but the authors present an idea to leverage the large-scale resources from spoken language. \nThe proposed framework utilises large-scale vision and language models with lightweight adapters. \nIn particular, the method leverages a fronzen GPT model for translation.\nThe method is gloss-free, so that large-scale data can be used without gloss-level annotations for supervised learning. \nInstead, the authors propose a novel encoder pretraining strategy based on pseudo-gloss that can be extracted from natural language sentences. \nThe authors experiment on the popular PHOENIX and CSL-Daily datasets, on which they demonstrate state-of-the-art performance compared to all baselines.", "questions": "Please see the last point of weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of LLM for sign language recognition is novel and effective.\n- The gloss-free framework using pseudo-gloss mitigates the challenging supervision problem in sign language recognition.\n- The results are state-of-the-art, and the ablations in Tables 2 and 3 show that the proposed pre-training helps to improve performance.\n- The writing is generally clear.", "weaknesses": "- The method is mostly based on existing models such as GPT, Dino-V2 and LoRA, so there is not much novelty from the architectural standpoint.\n- There is no ablations to demonstrate if the choice to use parts-of-speech tagging effective. How does it compare to using the words as tokens? Can the downstream translation learn to generate meaningful words that are missing in the parts-of-speech tagging?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pr5Bz9FfIG", "reviewer": "Reviewer_aj4n", "summary": "This paper aims to improve gloss-free sign language translation by exploring pretrained vison and language models, i.e., ViT and GPT models. The authors use pretrained ViT models to extract spatial features from sign frames and use pretrained GPT models to perform the translation. They design a sign encoder and a zero-gated cross-attention module to bridge these two models and pretrain the encoder with a pseudo-gloss pretraining strategy. On Phoenix14T and CSL-Daily, this method achieves new SOTA in the gloss-free setting.", "questions": "1) It's great that the proposed method outperforms GFSLT-VLP. However, GFSLT-VLP is based on MBart with ~600M parameters, and this study adopts XGLM with ~1.7B parameters, making the fairness of the comparison questionable. It's unclear whether the improvements are really from the proposed modeling and pretraining strategy. Could you please add further ablations regarding the size of GPT models? \n2) The sign encoder and pseudo-gloss pretraining take a crucial role in the proposed method, but analysis and ablation regarding them are insufficient. \n  - Do we need the sign encoder? What if dropping it? \n  - Apart from the visual analysis, what about the top-N prediction accuracy and recall?\n3) Please also add gloss-based SOTA systems in Tables 2 and 3 so that readers can understand the gap.\n4) In Eq (1), you mentioned that V represents keys from textual features while K originates from sign features. Shouldn't they both come from sign features?\n5) How did you set the rank in LoRA?\n6) While Phoenix14T and CSL-Daily are popular, they are less significant due to limited vocabulary and data size. Please consider adding results for DGS3-T (Zhang et al., 2023).\n\n\nAfter Response:\n\nThanks for the new results which address part of my concerns. I increased my score for this. Still, I believe the study of gloss-free approach should be performed on more realistic datasets, such as DGS3-T and WMT-SLT.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) Introducing a simple method leveraging pretrained vision and language models to improve SLT: SOTA performance\n2) Proposing a pretraining strategy based on pseudo glosses that induce meaningful sign representations", "weaknesses": "1) While achieving good performance, the pretrained models are significantly larger than previous approaches, raising concerns about fair comparison and what helps translation. Relevant ablation is missing.\n2) Analysis regarding the sign encoder and pseudo-gloss pretraining is insufficient.\n3) The used SLT benchmarks are somehow artificial albeit popular.\n4) Some details are confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HAS3e11BNZ", "reviewer": "Reviewer_hMSD", "summary": "The paper propose Sign2GPT, which enjoys the benefit of large pretrained language model to promote gloss-free sign language translation. The authors also propose a CLIP-styple pseudo-gloss pretraining technique to better learn visual-lingusitic representations. The overall method achieves SOTA performance on two widely adopted benchmarks, Phoenix-2014T and CSL-Daily.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea is sound. It is good to see that current large language model can be helpful for sign language understanding.\n2. The proposed pseudo-gloss pretraining is novel, which can inspire future works.\n3. SOTA gloss-free sign language translation performance on two benchmarks.", "weaknesses": "1. Using pretrained language model to boost sign language translation (SLT) is not surprisingly novel. Several works [1,2] have already verified that pretrained language model can boost (gloss-free) SLT.\n\n2. The name of pseudo-gloss pretraining is a bit confusing, although the method itself is sound. Because the pseudo-glosses are in spoken language order, it is not quite appropriate to call them \"glosses\".\n\n3. The notations of \"P\", \"F\", \"D\" in Table 2 and 3 are also confusing. For example, D denotes without pretraining and P denotes pretraining, then what does \"P+D\" mean? I suggest authors adding a \"check mark\" column to show which parts are pretrained.\n\n4. I cannot find an ablation study on the pseudo-gloss pretraining. What is the performance if removing it?\n\n5. In Table 4, removing sinusoidal positional encoding leads to the best performance. Then what poistional encoding is used by default?\n\n6. The paper uses a new spatial backbone which is under-explored in previous sign language papers. The authors need to better motivate it. For example, is it better than other vision transformers, or is it better than widely adopted 2D/3D CNNs?\n\n7. In figure 2, how to fuse the outputs of adapted masked attention (solid lines) and zero-gated cross attention (dashed lines)? Besides, what is adapted masked attention? I didn't see a clear definition of it. \n\n[1] A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation, CVPR 2022\n\n[2] Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining, ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)