Cross-modality Data Augmentation for End-to-End Sign Language Translation
Authors: Jinhui Ye
Wenxiang Jiao
Xing Wang
Zhaopeng Tu
Hui Xiong
Venue: EMNLP 2023 Findings (2023)
Keywords: Sign Language Translation
Cross Modality
Data Augmentation

Abstract: End-to-end sign language translation (SLT) aims to directly convert sign language videos into spoken language texts without intermediate representations. It has been challenging due to the data scarcity of labeled data and the modality gap between sign videos and texts. To tackle these challenges, we propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation (i.e., video-to-text). Specifically, XmDA consists of two key components: cross-modality mix-up and cross-modality knowledge distillation. The former one explicitly encourages the alignment between sign video features and gloss embeddings to bridge the modality gap. The latter one utilizes the generation knowledge from gloss-to-text teacher models to guide the spoken language text generation. Experimental results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily, demonstrate that the proposed XmDA framework significantly and consistently outperforms the baseline models. Extensive analyses confirm our claim that XmDA enhances end-to-end sign language translation by reducing the representation distance between sign videos and glosses, as well as improving the translation of low-frequency words and long sentences.

Cross-modality Data Augmentation for End-to-End Sign Language Translation
Wenxiang Jiao" Xing Wang4t Zhaopeng Tu Hui Xiong' 122.. 'Thrsst of Artificial Intelligence, HKUST (Guangzhou), Guangzhou, China Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China Guangzhou HKUST Fok Ying Tung Research Institute "Tencent Al Lab jye6248connect. ikust-gz.eduu en xionghui@ust. hk [joelwxjiao, wptptw
Jinhui Ye'
Abstract
which sign language recognition identifies gloss se- quences from continuous sign language videos, and gloss-to-text translation (Gloss2Text) converts the sign gloss sequence into written text (Yin and Read, 2020; Camgoz et al., 2020b; Moryossef et al., 2021; Kan et al., 2022; Ye et al., 2023). Nevertheless, it is well-known that the cascaded system has the significant drawback of time delay and error propa- gation (Chen et al., 2022: Zhang al., 2023a). End-to-end SLT models offer an intuitive so- lution circumvent the challenges posed by cas- caded systems. These models directly translate sign language vidcos into spoken texts without requiring an intermediary sign gloss representation (Camgoz et al., 2020a; Chen et al., 2022; Zhang et al., 2023a). However, the end-to-end approach faces significant obstacles in practice due to inadequate documenta- tion and resources for sign languages. This short- age of resources results in high cost of annotated data (NC et al., 2022), which hinders the develop- ment of end-to-end models. There have been a few recent attempts to improve the end-to-end SLT per- formance, including back-translation (Zhou et al., 2021) and pre-training (Chen et al., 2022), to miti- gate the issue of data scarcity.
End-to-end sign language translation (SLT) aims to directly convert sign language videos into spoken language texts without interme- diate representations. ing due the data scarcity of labeled data and the modality gap between sign videos and texts. To tackle these challenges, we propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to- end sign language translation (i.e., video-to- lext). Specifically, XmDA consists two key components: cross- moos-todyllt mix-up and cross- modality knowledge distillation. The former one explicitly encourages the alignment be- tween sign video features and gloss embed- dings to bridge the modality gap. The latter one utilizes the generation knowledge from gloss-to-text teacher models to guide the spo- ken language text generation. Experimental results widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily, demonstrate that the proposed XmDA framework signifi- cantly and consistently outperforms the base- line models. Extensive analyses confirm our claim that XmDA enhances end-to-end sign language translation by reducing the representa- tion distance between sign videos and glosses, well as improving the translation of low- frequency words and long sentences. Codes have been released at https: //g /i thub. com/ Atrewin/SignXIDA
has been challeng-
Along this research line, in this work, we pro- pose novel Cross- (XmDA) approach improve the end-to-end SLT performance. The main idea of XmDA to lever- age the powerful gloss-to-tex translation capabili- ties (unimode, i.e., text-to-text) to end-to-end sign language translation (cross mode, i.e., video-to- text). Specifically, XmDA integrates two tech- niques, namely Cross-modalit, Mix-up and Cross- modality Knowledge Distillation (KD) ($ 2.2). The Cross-modalite Mix-up technique combines sign language video features with gloss embeddings extracted from the gloss-to-text teacher model generate mixed-modal augmented samples ($ 2.3). Concurrently, the Cross Cross-modalityy KD utilizes di- versified spoken language texts generated by the
Data Augmentation
1 Introduction
Sign language is essential communication tool used in deaf communities. Sign language transla- tion (SLT) has made significant progress in recent years (Camgoz et al., 2018; Yin and Read, 2020; Zhou et al., 2021; Chen et al., 2022; Zhang et al., 2023a), with the goal of converting sign language videos into spoken language texts. The conven- tional approach to SLT uses a cascaded system in
Corresponding authors: Xing Wang and Hui Xiong-

---

powerful gloss-to-text teacher models soften the target labels ($ 2.4), thereby further diversifying and enhancing the augmented samples.
KD. To ensure clarity, we define end-to-end SLT and the notation used throughout this paper.
2.1 Task Definition
We evaluate the effectiveness of XmDA on two widely used sign language translation datasets: PHOENIX-2014T and CSL-Daily. Experimental results show that XmDA outperforms the com- pared methods in terms of BLEU, ROUGE, and ChrF scores ($ 3.2). Through comprehensive anal- yses, we observed two key findings. Firstly, Cross- modality Mix-up successfully bridges the modality gap between sign and spoken languages by improv- ing the alignment between sign video features and gloss embeddings ($ 4.1). Secondly, the Cross- modality KD technique enhances the generation of spoken texts by improving the prediction of low- frequency words and the generation of long sen- tences ($ 4.2).
We formally define the setting of end-to-end SLT. The existing SLT datasets typically consist of sign- gloss-text triples, which can be denoted as {S; Gú, Ti Here, S; = {3 } represents sign video comprising Z frames, G; = {ox 1 is gloss sequence consisting of V gloss annotations, == is spoken language text con-
and T; [tu] sisting of U words. In addition, it's worth noting that the gloss G is order-consistent with the sign gestures in S, while spoken language text monotonic to S. End-to-end SLT aims to directly translate a given sign video S into a spoken lan- guage text T.
non-
2.2 Sign Language Transformers
The contributions of our work are summarized as follows:
Sign Language Transformers utilize the encoder- decoder architecture of transformer networks to recognize and translate sign language S into both sign gloss G and spoken language text T in an end- to-end manner. The original model is composed of four main components: sign embedding, a trans- lation encoder, a translation decoder, and a CTC classifier. In our work, we extend this model by introducing a fifth component, the gloss embed- ding, which provides richer representation for the translation process (e.g., gloss-to-text translation). Sign Embedding. In line with recent re- search (Camgoz al., 2020b; Zhou al., 2021), we use pre-trained visual models extract sign vidco frame representations. Specifically, we fol- low Zhang et al. (2023a) to adopt the pre-trained SMKD model (Min et al., 2021) and extract visual representations from video frames. These represen- tations are then projected into the same size the gloss embedding through linear layer. It should be noted that the parameters of the pre-trained SMKD model are frozen during the training of the Sign Language Transformers.
We propose novel Cross-modality Data Aug- mentation (XmDA) approach to address the modality gap between sign and spoken lan- guages and eases the data scarcity problem in end-to-end SLT.
Comprehensive analysis demonstrates that XmDA is effective approach to diminish the representation distance between video and text data, enhancing the performance of sign language translation, i.e., video-to-text.
We evaluate the effectiveness of the proposed XmDA on two widely used SLT datasets and demonstrate that XmDA substantially im- proves the performance of end-to-end SLT models without additional training data, pro- viding valuable alternative to data-intensive methods.
2 Methodology
We integrate the proposed Cross-modal Data Aug- mentation (XmDA) technique into the Sign Lan- guage Transformers, which is widely used in sign language translation tasks. The proposed frame- work is illustrated in Figure 1. In this section, will first revisit the Sign Language Transformers structure (Camgoz et al., 2020b). Then we pro- vide more comprehensive explanation of the pro- posed approach, including two essential compo- nents: Cross Cross-modality Mixup and Cross-modality
Translation Encoder. The translation encoder in Sign Language Transformers comprises multi- layer transformer networks. Its input is the em- bedding sequence of input tokens, such as the rep- resentation of sign video frames. Typically, the input embedding sequence is modeled using self- attention and projected into contextual representa- tions h(S). These contextual representations are fed into the decoder to generate the target spoken text translation.

---

Cross-modality KD
81
92
85
Eg
Fare
LUELE
Clese
Eyo
Pies) Pled
Piau
CTC Classifier
DistrimationsSSign
Distrihuttous
Translation Encoder
Translation Decoder
Gloss2Text T T'acceers
MLP
A
1-2
SMKD
Mix-up
Embedding
Cross-modality Mix-up
85
Figure I: The overall framework of cross-modality data augmentation methods for SLT this work. Components in gray indicate frozen parameters.
Translation Decoder. The translation decoder in Sign Language Transformers involves multi- layer transformer networks that include cross- attention and self-attention layers. These decoder networks produce spoken language sentences based on the sign video representation h(S).
frame and model the marginalizing overall possible
S to as
P(G = E P((T((()
(2)
FEI
Where G is the ground truth gloss annotations sequence and path, i.e., the predicted gloss sequence sign frames from $1 to sz and is the set of all viable paths resulting to G sequence. And the CTC loss function is defined as:
The objective function for spoken language text generation is defined tween the decoder prediction texts and reference texts.
entropy loss be-
Lcre 1 (G'
(3)
MLLE = Σ log P (tulttcu h(S))
(1)
2.3 Cross-modality Mix-up
Cross-modality Mix-up promotes the alignment of sign video features and gloss embeddings to bridge the modality gap by producing mixed-modal repre- sentations SLT input. As shown in Figure 1, given training sample ({s²)² - V {tu H=)) 2 D, the { and {g are embedded into a sequence of sign video frames features F [f1, f2, ... fz] and gloss embeddings E [c1, C, ... ev). Note that the dimensions of sign features and gloss embeddings are identical. Cross- modality Mix-up aims to obtain mixed- modell rep- resentations M by combining JF and E. CTC Sign-Gloss Aligner. To perform the Cross-modalis Mix-up, an indispensable step determine the temporal boundaries between glosses in continuous sign video. Inspired by previous work in speech recognition (Kürzinger et al., 2020),
Gloss Embedding. The gloss embedding, which is presented in Sign Language Transformers, works similarly to traditional word embedding in that it converts discrete gloss tokens into continuous vector representations by an embedding matrix. The gloss embedding is an important expansion to the original Sign Language Transformers (Camgoz et al., 2020b), which brings broader perspective to the translation process (e.g., gloss-to-text and (sign mix tlote))..
CTC Classifier. Sign Language Transformers inject gloss annotations as the intermediate super- vision to train the translation encoder. A linear projection layer followed by a softmax activation (i.c., gloss classifier) is employed to predict sign gloss distribution P (SS) of each sign video

---

we employ the CTC classifier as sign-gloss forced aligner, which returns the start position by and end position To the sign video frame flow for each corresponding gloss Jv by maximizing the marginal probability of a specific path x. More specifically, the sign-gloss forced aligner identifies the optimal path x² from II as follows:
use the data augmentation strategies (Moryossef et al., 2021: Zhang and Duh, 2021: Ye al., 2023) to train K gloss-to-text teacher models (i.e., M M22T. ... MM2TT on the given SLT dataset = [[S; Gi, T:)) Those teacher mod- els translate each G, to diversified spoken language translations (i.e., T TT2
T,K for the source sign video input S; and obtain multi-references dataset DMKD - U (X,, G. ) Here, T represents the original reference and the size of the data is K + times that of DD
x²" + argm max P (x) M(S))
Tell
V
argm max TCIl
ΣΣ
logP(ge = )7)
(4)
2.5 Overall Framework
L=0
Model Training. As described in Section 2.4, we first apply Cross-modality KD to enrich the dataset D into DMD by attaching the diversified spoken language translations T. During the training stage, apply Cross-modality Mix-up (Section 2.3) produce additional mixed-modal inputs that help bridge the modality gap between the sign video features and gloss embeddings.
Following the best sign-gloss alignment path n², the CTC classifier returns the start position and end position ry the sign video frame flow for each gloss 9e- With pre-defined mix-up ratio X, we obtain the mixed-modal representations M as:
Fll= rul, Elu),
p S X
my
(5)
p >>7 A
As depicted in Figure 1, the end-to-end SLT model is trained by minimizing the joint loss term C. defined as:
= [m1, m2, mv]
(6)
Where is sample from the uniform distribution (0, 1).
L = + Lore + LJSD
(8)
Model Inference. In the inference stage, the sys- tem utilizes the sign embedding, translation en- coder, and translation decoder to directly convert sign video input into spoken language text.
Mix-up Training. As shown in Figure to narrow the gap these two types of input representa- tions, namely, fixed-modal input representations M and unimodal input representations F. we feed them to the translation encoder and decoder, and minimize the Jensen-Shannon Divergence (JSD) between the two prediction distributions. The regu- larization loss is defined as:
Experiments
3.1 Experimental Setup
Dataset. We conduct experiments on two widely- used benchmarks for SLT, namely, PHOENIX- 2014T (Camgoz et al., 2018) and CSL-Daily (Zhou et al., 2021). The statistics of the two datasets are listed in Table 9
IT'
LSD ISD(P(tulttuu w=1
[P(tultenmMM)
2.4 Cross-modality Knowledge Distillation
Model Settings. For the baseline end-to-end SLT model, we follow Sign Language Transform- ers (Camgoz et al., 2020b) consisting of multi- ple encoder and decoder layers. Each layer com- prises eight heads and uses a hidden size 512 dimensions. For the sign embedding component described in Section 2.2, we adopt the SMKD model (Min et al., 2021) to extract visual features and apply a linear layer to map it to 512 dimen- sions.
Sequence-leve knowledge distillation (Kim and Rush, 2016) (in text mode) encourages the stu- dent model to mimic the teacher's actions at the sequence level by using generated target sequences from the teacher. Similarly, Cruss-modality KD employs the generative knowledge derived from gloss-to-text teacher model guide the generation of spoken language text in SLT.
Motivated by the multi-teacher distilla- tion (Nguyen et al., 2020; Liang et al., 2022), we
hhttps://Ett http:ttubb..rrccmin///ccc.....

---

Dev ROUGE
Test ROUGE
Method
BLEU
ChrF
BLEU
ChrF
Previous Research
Sign Language Transformers
22.38
21.32
(Camgoz et al., 2020b)
Sign Back- Tack-Translation (Zhou et al., 2021)
24.45
50.29
24.32
49.54
Contrastive Transformer (Fu et al., 2022)
21.11
47.74
21.59
47.69
Our Experiments
Sign Language Transformers
22.90
48.05
44.96
22.79
47.33
44.36
Cross-modality Mix-up
23.60
48.59
46.14
23.87
48.92
46,04
Cross-modality KD
24.82
50.04
47.98
24.77
49.53
48.04
+ XmDA
25,86
52.42
50.10
25.36
49.87
51.49
Table I: End-to-end SLT performance on PHOENIX-2014T dataset. "+ XmDA" denotes the application both Cross-modality Mix-up and Cross-modality KD.
Dev ROUGE
Test ROUGE
Method
BLEU
Chrli
BLEU
Chrli
Previous Research
Sign Language Transformers
11.88
37.06
11.79
36.74
(Zhou et al., 2021)
Sign Back-Translation (Zhou al., 2021)
20.80
49.49
21.34
49.31
Contrastive Transformer (Fu et al., 2022)
14.80
41.46
14.53
40.98
Our Experiments
Sign Language Transformers
11.55
36.32
11.14
11.61
36.43
11.15
Cross-modality Mix-up
15.67
42.36
14.57
15.47
42.60
14.68
+ Cross-modality KD
17.05
44.17
15.74
16.92
43.90
15.65
+ XmDA
21.69
49.36
19.60
21.58
49.34
19.50
Table 2: End-to-end SLT performance on CSL--Daily dataset. "+ XmDA" denotes application of both Cross-
modality Mix-up and Cross-modality KD.
In demonstrating the proposed XmDA, we uti- lize four gloss-to-text teacher models (i.c., K = 4), each trained using PGEN (Ye et al., 2023) with varying seeds and subsets of training data. The mix-up ratio for Cross-modality Mix-up is set to 0.6 (i.c., X 0.6).
Chen et al., 2022), we use standard metries com- monly used in machine translation, including tok- enized BLEU (Papineni et al., 2002) with 4-grams and Rouge-L F1 (ROUGE) (Lin. 2004) to evalu- ate the performance of SLT. Following Zhang et al. (2023a), we also report ChrF (Popović, 2015) score to evaluate the translation performance.
Furthermore, the gloss embedding matrix ini- tialized using the most effective gloss-text teacher model as per Table 15, and is fixed constant throughout the training phase. This approach helps to preserve the well-captured distributional prop- erties of gloss representations. Due to space con- straints, more details about model settings and the optimal parameters of each component are de- tailed in A.2 and A.3.
3.2 Experimental Results
We validate the proposed approaches on the Sign Language Transformers baseline on PHOENIX- 2014T and CSL-Daily. The main results are listed in Table and Table In the PHOENIX-20147) test set, Cross-modality Mix-up surpasses the baseline model by 1.08 BLEU scores, 1.59 ROUGE scores and 1.68 ChrF scores. Similarly, Croas-modality KD yields en-
Evaluation Metrics. Following previous stud- ies (Camgoz et al., 2018, 2020b; Zhou al,, 2021;

---

hancements of 1.98 BLEU scores, 2.20 ROUGE scores and 3.68 ChrF scores. The proposed XmDA method, which integrates both techniques, achieves a gain of up to 2.57 BLEU scores, 2.54 ROUGE scores and 7.13 ChrF scores. The proposed XmDA also outperforms other data augmentation tech- niques, e.g., Sign Back-Translane Translatoonll and Contrastive Transformer in Table 1, thus demonstrating the ef- fectiveness of the proposed approach.
30
20
Gloss Embedding
Sign Embedding
Mix-up Embedding
Table presents the experimental results of the CSL-Daily dataset. similar phenomenon is ob- served, indicating the universality of the proposed XmDA.
Figure Bivariate kernel density estimation visualiza- tion of sentence-leve representations: sign embeddings from baseline SLT, gloss embeddings from the gloss- to-text teacher model, and mixed-modal representations obtained by mixing sign embeddings and gloss embed- dings with À 0.6. Best viewed in color.
4 Analysis
this section, we delve into the mechanisms be- hind the improved end-to-end SLT performance achieved by the XmDA method. We investigate our approach's impact on input space and output translations, focusing on the effectiveness of the Cross-modality Mix-up and Cross-modalits KD techniques and their influence word frequency and sentence length.
picted in Figure 3, gloss embeddings in the gloss- to-text teacher model have discrete distribution (in red), indicating excellent representation for dif- ferent glosses. In contrast, the sign embedding of the baseline SLT model exhibits a smoother dis- tribution (in blue), potentially causing confusion when translating sign language videos into spo- ken texts. Remarkably, the sign embeddings ex- tracted from the "#Cross-modality Mix-up" model (in green) exhibit distribution similar to that of gloss embeddings.
4.1 Impact of Cross-modality Mix-up
In Section 2.3, we claim the XmDA approach can bridge the modality gap between video and text and promote the alignment of sign video features and gloss embeddings. To investigate this, we conduct experiments on the SLT model using XmDA in Table 1.
For quantitative analysis, employ Kernel Den- sity Estimation (KDE) to estimate the probability density functions for these three types of embed- dings (Moon and Hero, 2014). The resulting en- tropies from these KDEs are tabulated below:
We visualize multi-view representation (sign, gloss, mixed-modal) for each PHOENIX-201 test set sample. We compute sentence-level repre- sentations for each input view by averaging embed- ding sequences and apply t-SNE (Van der Maaten and Hinton, 2008) for dimensionality reduction to 2D. We then plot the bivariate kernel density esti- mation based 2D sequence-level representations. As shown in Figure 2, the 2D distributions of sign video features (blue) and gloss embeddings (red) are distinet, while mixed-modal representations (green) lie between them, indicating that mixed- modal representations serve as an effective bridge between sign and gloss representations.
Embedding Type
KDEs Entropy
Gloss Embeddings
0.19
Sign Embeddings (Baseline) Sign Embeddings (XmDA)
2.18
0.84
Table 3: Entropies resulting from KDEs different types of embeddings.
Additionally, to provide a quantitative measure of aligmment, we compared the average Euclidean distance and cosine similarity at the werd-level between sign embeddings and gloss embeddings. The results are presented in the table below:
To further investigate the benefits of mixed- modal sequence input, examine the topolog- ical structure of input embeddings for the baseline, "#Cross-modality Mix-up", and the best gloss-to- text teacher model. We employ t-SNE to visu- alize those embedding sequences in 2D. As de-
This demonstrates that incorporating the mixed- modal inputs in the training process enables better sign representation by utilizing the constraint from the teacher gloss embeddings.

---

Comparison Sign (Baseline) vs. Gloss
ED
Cos Sim
Word Frequency
Data
14.6
0.19
Low
Medium
High
Sign (XmDA) vs. Gloss
8.68
0.34
Baseline SLT
30.70
52.24
60.61
Table 4: Average Euclidean distance (ED) and cosine similarity (Cos Sim) between sign embeddings and gloss embeddings.
+ Xm Mix-up
31.79
55.20
60.09
Xm KD
33.36
55.96
61.60
XmDA
34.68
56.07
61.93
Table 5: Prediction accuracy (Fl score) of target words the test set with respect to word frequency. Xm indi- cates Cross-modality. As the higher F1 score, the better, we mark the improvement by green and degradation by red background.
40
20
@3 tamihan
0
G2 kenner
-20
thic
40
22 gewtsy
4i geens
the prediction of low-frequency words. Specifi- cally, first categorize the vocabulary into three groups based on the word frequency in the training data, including High: frequency E [2000, +oo);
Gloss Einbedding
-60
Sign Eibeedding (Baseline)
Sign Embedding (XMDA)
-60
-40
20
20
40
Medium: frequency E [100, 2000); Low: fre- quency E (0, 100]. Then, we utilize compare-mt to calculate the prediction accuracy of target words, e.g., Fl score (Snover et al., 2006), in the test set with respect to the three groups.
Figure 3: Visualization of gloss and sign representation distributions for the Baseline SLT (in blue) and "+ Cross- modality Mix-up" (in green) models by t-SNE. Best viewed in color.
Table 5 presents the results for the different ap- proaches concerning word frequency. It can be ob- served that the Cross-modality Mix-up and Cross- modality KD methods independently improve the Fl score for low-frequency and medium-frequency words compared to the Baseline SLT. When com- bined, XmDA method, which stacks both Cross- modality Mix-up and Cross-modality KD, achieves the most significant improvement, particularly in low-frequency words, demonstrating its effective- ness in predicting these words. For high-frequeccy words, both Cross-modality KD and XmDA im- prove Fl score, whereas Cross-modality Mix-up shows a slight decrease. These results highlight the potential of rroas-modality techniques in enhanc- ing the model's ability to predict low-frequency words, which contributes to more robust and di- verse translation output.
To illustrate this point, we sample a gloss an- notation sequence (e.g., r"region", "gewitter", "kommen") }) from the test set and plot the relative sign frame embeddings and gloss embeddings in Figure 3. Notes, middle frames are selected rep- resentative embeddings based sign-gloss forced aligner temporal boundaries (S 2.3). As seen, the Pross-modality Mix-up technique (green dot line) significantly improves the sign embedding repre- sentations over the baseline (blue dot linc), bring- ing sign embedding representations within visual and text modalities closer and a better aligmment.
4.2 Impact on SLT Translation Outputs
To gain deeper understanding of how the pro- posed approach enhances the end-to-end SLT, we analyze the translation outputs of PHOENIX- 2014T, as shown in Table 1, using the compare-mt toolkit?. We focus on two aspects: word frequency and sentence length.
Sentence Length:
We investigate the translation quality of examples with varied lengths, which can be biased when generating target spoken language texts. Similar to word frequency, we also use the compare- toolkit to categorize the examples of the test set into three groups based on the sen- tence length, including Long: (20, +oo) tokens; Medium: (10, 20) tokens; Short: (0, 10] tokens.
Words Frequency: Multiple reference transla- tions have been shown to improve translation qual- ity by promoting diversity and robustness in trans- lation outputs (Nguyen et al., 2020). We won- der how such diversity and robustness benefit
Table 6 presents the results concerning sentence length. It is well-known that longer sentences

---

Sentence Length
sign videos directly natural text without using gloss as an intermediary representation. Existing works attempt to formulate this task as a neural ma- chine translation (NMT) problem (Camgoz et al., 2018, 2020a; Zhou et al., 2021; ; Chen et al., 2022; Zhang et al., 2023a). However, unlike NMT, which benefits from a large-scale parallel corpus (Wang et al., 2022), end-to-end SLT greatly suffers from data scarcity.
Data
Short Medium Long
Baseline SLT
36.70
17.56
12.73
+ Xm Mix-up
38.96
18.78
13.68
Xm KD
34.32
19.67
15.23
XMDA
38.05
20.80
17.54
Table 6: Translation quality (BLEU score) of examples in the test set with respect to sentence length.
Recent studies have focused on the challenge of data scarcity, e.g., sign back-translation (Zhou al., 2021), transfer learning (Chen et al., 2022) and multi-tase learning (Zhang et al., 2023a). Along this research line, this work proposes a novel Cross- modality Data Augmentation (XmDA) framework, which is more valuable and resource-efficient alternative data augmentation method for SLT. XmDA augments the source side by mixing sign embeddings with gloss embeddings, and it boosts the target side through knowledge distillation from gloss-to-text to-text teacher models, without the need for additional data resources.
are more challenging to translate (Zheng et al., 2020). Cross- modaaity Mix-up and Cross- modaliyyy KD methods stand out, exhibiting significant im- provement in translation quality for medium and long sentences. However, the Cross-modality KD method noticeably degrades performance short sentences. In contrast, the XmDA method signifi- cantly improves translation quality for medium and long sentences without compromising the perfor- mance of short sentences. This demonstrates that the XmDA approach, which combines the strengths of both Cross-modality Mix-up and Cross-modalle KD, offers superior stability across various sen- tence lengths and effectively addresses the distri- bution of sentence length in translation tasks better than using the individual methods alone.
5.2 Mix-up Data Augmentation
Mix-up, an effective data augmentation technique, has made significant strides in the computer vision (CV) field (Zhang et al., 2017: Yoon et al., 2021; Liu et al., 2022; Hao et al., 2023; Zhang et al., 20236). Its unique approach involves creating new training samples through linear interpolating pair of randomly chosen examples and their respective labels at the surface level (Zhang et al., 2017) or feature level (Verma et al., 2019).
4.3
Case Examination of Model Outputs
To gain insights into our model's performance in specific scenarios, we present two notable obser- vations. Firstly, model exhibits strong profi- ciency in generating low-frequency words, which are often challenging for many translation systems (Table 7). Additionally, our approach showcases remarkable capability translating long sentences (Table 8), which is another common hurdle in trans- lation tasks.
Recently, the natural language processing (NLP) domain has benefited from applying the mix-up technique. It has demonstrated promising out- comes in various tasks such as machine transla- tion (Cheng al., 2021), multilingual understand- ing (Cheng et al., 2022), as well as speech recogni- tion and translation (Meng et al., 2021; Fang et al.. 2022). As far as we know, XmDA is the first study to extend mix-up data augmentation to sign lan- guage translation tasks, encompassing visual and text modalities.
5 Related Work
5.1 Sign Language Translation
Existing approaches to Sign Language Transla- tion (SLT) can be categorized into two main types: cascaded and end-to-end methods (Camgoz et al., 2020b). Cascaded methods decompose SLT into two separate tasks (Yin and Read, 2020): sign lan- guage recognition, which recognizes the gloss se- quences from continuous sign videos, and sign lan- guage gloss translation, which translates the gloss sequences into spoken language text. Conversely, End-to-end SLT methods convert
5.3 Knowledge Distillation
Knowledge Distillation (KD) (Hinton al., 2015) is a powerful technique for model compression and transfer learning, wherein a smaller student model is trained to mimic the behavior more complex teacher model or an ensemble models, allowing the student to learn a more gen-
larger,

---

System
Translation Output
Reference
ich wünsche ihnen noch einen schönen abend
Baseline
ihnen einen schönen abend und machen sie es gut
XmDA
ich wünsche ihnen einen schönen abend
Table 7: Comparison translation outputs for low-frequency words generation. The word "ich wünsch" is emphasized indicate its low-frequency nature in the dataset, i.c., frequency (0, 10).
System
Translation Output
da haben wir morgen schon die dreiBig grad morgen im süden von frankreich auch und für uns wahrscheinlich schon im südwesten die fünfundzwanzig grad dort morgen bis dreißig grad im äußersten süden und auch im südwesten da haben wir morgen auch die dreißig grad schon über frankreich und auch in süd- deutschland haben wir noch die wärme schon mal über fünfundzwanziz grad im südwesten
Reference
Baseline
XmDA
Table 8: Comparison of translation outputs for long sentence generation, i.e., length 20. The sentences illustrate the model's capability to produce coherent and contextually appropriate translations for longer input sequences.
eralized and robust representation. This results in improved performance and efficiency, particularly in resource-constrained environments.
Comprchensive analyses suggest that our approach improves the aligmment between sign video fea- tures and gloss embeddings and the prediction of low-frequency words and the generation of long sentences, thus outperforming the compared meth- ods in terms of BLEU, ROUGE and ChrF scores.
Within the NLPP domain, KD has been success- fully applied to a diverse range of tasks (Tang et al., 2019; Tan et al., 2019; Zhou et al., 2020; Jiao et al., 2020) and can be categorized into word- level (Wang et al., 2018) and sentence-level distilla- tion (Kim and Rush. 2016), with the latter gaining prominence due to its superior performance and training efficiency.
Limitations
We identify two limitations of our XmDA ap- proach:
The success of XmDA is contingent upon the quality of gloss-to-text teacher models, partic- ularly those like PGEN (Ye et al., 2023). In environments with limited resources or unique data contexts, reproducing such models might be demanding, potentially affecting XmDA's generalization.
In our work, we extend the concept of KD to cross-modality end-to-end SLT scenarios. In this setting, the end-to-end SLT student model learns to mimic the behavior of various gloss-to-text teacher models, leveraging the power of the teacher to im- prove end-to-end SLT performance.
6 Conclusion
The approach's 's efficacy is tied to the availabil- ity and integrity of gloss annotations. For sign languages or dialects lacking comprehensive gloss resources, the full potential of XmDA might not be realized, underscoring the value of rich gloss resources in sign language trans- lation.
In this paper, we propose novel Cross-modality Data Augmentation (XmDA) approach to tackle the modality gap and data scarcity challenge in end-to-end sign language translation (SLT). XmDA integrates two techniques, namely Cross-modality Mix-up and Cross-modality Knowledge Distilla- tion (KD). Cross-modality Mix-up bridges the modality gap by producing mixed-modal input rep- resentations, and Cross-modality KD augments the spoken output by transferring the generation knowl- edge from powerful gloss-to-text teacher models. With the XmDA, we achieve significant improve- ments on two widely used sign language transla- tion datasets: PHOENIX 201 4T and CSL-Daily.
8 Acknowledgement
This research was supported in part by the Na- tional Natural Science Foundation of China (Grant No. No.92370204) and the Science and Technology Planning Project of Guangdong Province (Grant No. 20234050005011)),

---

References
Wenxiang Jiao, Xing Wang, Shilin He, Irwin King. Michael Lyu, and Zhaopeng Tu. 2020. Data reju- venation: Exploiting inactive training examples for neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2255-2266.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney. and Richard Bowden. 2018. Neu- ral sign language translation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Jichao Kan, Kun Hu, Markus Hagenbuchner, Ah Chung Tsoi, Mohammed Bennamoun, and Zhiyong Wang. 2022. Sign language translation with hicrarchical spatio-temporal graph neural network. In Proceed- ings of the IEEF/CVF Winter Conference on Applica- tions of Computer Vision, pages 3367-3376
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020a. Multi-channe trans- formers for multi-articulatory sign language transla- tion. In Computer Rision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 301-319, Springer.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020b. Sign language trans- formers: Joint end-to-end sign language recognition and translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pages 10023-10033.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pages 1317-1327.
Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, and Gerhard Rigoll. 2020. Ctc- segmentation of large corpora for german end-to-end speech recognition. In International Conference on Speech and Computer, pages 267-278. Springer.
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022. A simple multi-modality transfer learning baseline for sign language transla- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5120-5130.
Xiaobo Liang, Lijun Wu, Juntao Li, Tao Qin, Min Zhang, and Tie- Yan Liu. 2022. Multi-teat Kulti-teacher dis- tillation with single model for neural machine trans- lation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:992-1002.
Yong Cheng, Ankur Bapna, Orhan Firat, Yuan Cao, Pidong Wang. and Wolfgang Macherey. 2022. Multi- lingual mix: Example interpolation improves multi- lingual neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4092 4102.
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. In Text Summarization Branches Out, pages 74-81.
Zichang Liu, Zhiqiang Tang. Xingjian Shi, Aston Zhang, Mu Li, Anshumali Shrivastava, and An- drew Gordon Wilson. 2022. Learning multimodal data augmentation in feature space. arXiv preprint ar 22X1v:221..14453.
Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. 2021. Self-supervised and supervised joint training for resounce-rich machine translation. In International Conference on Muchine Learning, PMLR.
pages
Linghui Meng, Jin Xu, Xu Tan. Jindong Wang, Tao Qin, and Bo Xu. 2021. Mixspeech: Data augmenta- tion for low- low-resource automatic speech recognition. In ICASSP 3021-2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 7008-7012, IEEE.
Qingkai Fang. Rong Ye, Lei Li, Yang Feng, and Mingxuan Wang. 2022. STEMM: Self-learning with speech-text manifold mixup for speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 7050-7062, Dublin, Ireland. Association for Computational Linguistics.
Yuccong Min, Aiming Hao, Xiujuan Chai, and Xilin Chen. 2021. Visual alignment constraint for continu- ous sign language recognition. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 11542-11551.
Biao Fu, Peigen Ye, Liang Zhang, Pei Yu, Cong Hu, Yidong Chen, and Xiaodong Shi. 2022. Conslt: token-level contrastive framework for sign language translation. arXiv preprint ar 2rXiv:2204.04916
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju. Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF Winter Conference on
Kevin R. Moon and Alfred O. Hero. 2014. Ensem- ble estimation of multivariate f-divergence. In 2014 IEEE International Symposium on Information The- ory, pages 356-360.
Applications of Computer Vision, pages 379-389.
Amit Moryossef, Kayo Yin, Graham Neubig, and Yoav Goldberg. 2021. Data augmentation for sign language gloss translation. arXiv preprint arXiv:2105.07476
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge a neural network. arXiv preprint arXiv: 503.02531.

---

Gokul NC. Manideep Ladi, Sumit Negi. Prem Selvaraj. Pratyush Kumar, and Mitesh Khapra. 2022. Address- ing resource scarcity across sign languages with mul- tilingual pretraining and vocauld-yocabulary datasets. Advances in Neural Information Processing Systems, 35:36202-36215.
Jinhui Ye. Wenxiang Jiao, Xing Wang, and Zhaopeng Tu. 2023. Scaling back translation with domain text generation for sign language gloss translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistes, pages 463-476, Dubrovnik, Croatia. Associ- ation for Computational Linguistics.
Xuan-Phi Nguyen, Shafiq Joty, Kui Wu, and Ai Ti Aw. 2020. Data diversification: A simple strategy for neural machine translation. Advances in Neural In- formation Processing Systems, 1010100110299
Kayo Yin and Jesse Read. 2020. Better sign language translation with STMCCtransformmrr In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 5975-5989, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. 2021. Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprint arXiv:2107.00233
Maja Popović. 2015. chrf: character n- n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 395-
Biao Zhang, Mathias Müller, and Rico Semnrich. 2023a. Situnet: A simple unified model for sign language translation. In International Conference on Learning Representations.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Lin- nea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annota- tion. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231.
He Zhang. Ying Sun, Weiyu Guo, Yafei Liu, Haonan Lu, Xiaodong Lin, and Hui Xiong. 2023b. Interactive interior design recommendation via coarse-to-fie to-fine multimodal reinforcement learning. arXiv preprint arXiv.2310007287.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. Mixup: Beyond empirical risk minimization. arXiv preprini 171.00710000413
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie- Yan Liu, 2019. Multilingual neural machine trans- lation with knowledge distillation. In International Conference on Learning Representations.
Xuan Zhang and Kevin Duh. 2021. Approaching sign language gloss translation as a low-resource machine translation task. In Proceedings of the 1st Interna- tional Workshop Automatic Translation for Signed and Spoken Languages (AT4SSL), pages 60-70.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task- specifie knowledge from bert into simple neural net- works. arXiv preprint 1903
Jiangbin Zheng, Zheng Zhao, Min Chen, Jing Chen, Chong Wu, Yidong Chen, Xiaodong Shi, and Yiqi Tong. 2020. An improved sign language translation model with explainable adaptations for processing long sign sentences. Computational Intelligence and Neuroxcience.
Laurens Van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne Journal of Machine Learning Research, 911).
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019, Manifold mixup: Better rep- resentations by interpolating hidden states. In In- ternational Conference on Machine Learning, pages PMLR.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non- autoregressive machine translation. In International Conference on Learning Representations.
Hao Zhou, Wengang Zhou, Weizhen Qi. Junfu Pu, and Houqiang Li. 2021. Improving sign language transla- tion with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 1316- 1325.
Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang. Shuming Shi, Zhaopeng Tu, and Michael R. Lyu. 2022. Understanding and improving sequence- to-sequence pretraining for neural machine transla- tion. In Annual Meeting of the Association for Com- putational Linguistics.
Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2018 Kdgan: Knowledge distillation with gener- ative adversarial networks. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.

---

A Appendix
A.1 Data Statisties
21.50
We evaluate the effectiveness of our proposed
21.00
22.75
XmDA two widely used sign language trans- lation datasets: PHOENIX-20147 and CSL-Daily, These datasets provide diverse characteristics and contexts, making them ideal for our evaluation. Ta- ble 9 gives an overview these datasets.
22.00
0.4 Mixsup Ratio A
Figure 4: BLEU score of Cross Cross-modality Mix-up" on PHOENIX-20144 dev set, with different mix-up ratio A. When À = 0.0, "+ Cross-modality Mix-up" degrades to baseline model.
A.2 Hlyper-paramett of Baselines
Table 10 presents the hyper-parameters of Sign Language Transformers used in this work.
choose the static strategy with X - 0.6 as the Cross- modality Mix-up setting to balance performance and computational cost.
A.3 The Optimal Components of XmDA
In this section, we present the ablation studies to explore the optimal components of our pro- posed XmDA approach. Unless otherwise stated, we primarily conduct the analyses the German PHOENIX-2014T, Note that strive to identify the opar-optimal setting for our method mainly based on our experience rather than a full-space grid search, as aggressively optimizing the system requires significant computing resources and is be- yond our means.
A.3.2 What is the Optimal Number of Teacher Models?
As discussed in Section 2.4, adopt the PGEN (Ye et al., 2023) to train multiple gloss-to- text teacher models, which will used to obtain the new version of the target spoken language texts for SLT by translating the corresponding gloss an- notations into spoken texts. We examine the opti- mal number of teacher models and the translation performance of the teacher models, sorted by their BLEU scores the dev set, which be found in Table 15.
A.3.1 What the Optimal Mix-up strategy? We investigate the optimal mix-up ratio for our Cross-modality Mix-up. Inspired by Fang et al. (2022), we consider two types of strategies: a static strategy, where the À is fixed, and dynamic strat- egy, where the À is adaptively determined based on the confidence P (7*1 h(S)) of the sign-gloss forced alignment (see Formula 4). Specifically, We constrain in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] or À Sigmoid(P (T*) h(S)) - 0.5) for experi- ments on PHOENIX-20144 When A = 1.0, all of the sign embeddings FF will be swapped by gloss embedding & (e.g., M F), while X = 0.0 means none of the sign embeddings will be swapped into gloss embeddings and *+ Cross-modality Mix-up" degrades to the baseline model.
As illustrated in Table 11, we examine the re- lationship between the number of teacher models and the performance of the sod-to-end SLT model employing Cross-modality KD. The experimental results show that increasing the number of teacher models from K=0 to K=4 leads to notable im- provement in performance, while further increas- ing the number of teachers only provides marginal gains. To balance computational cost and model performance, we select K=4 as the default setting for the Cross-modality KD.
A.4 Ablation Study of Components in XmDA In this section, we evaluate the performance of the key components used in our Cross-modality Data Augmentation (XmDA) approach. These compo- nents, including the SMKD sign features extractor, the sign-gloss forced aligner, and the gloss-to-text teacher models, play crucial roles in the success- ful implementation of XmDA. Understanding their performance and effectiveness is essential for a comprehensive understanding of our proposed ap- proach. The following subsections provide detailed
As shown in Figure 4, our observations reveal that: 1) for the static strategy (represented by the bleu line), Cross-mo Mix-up attains optimal SLT performance the development set with mix- up ratio À 0.6; and 2) Cross-modality Mix-up employing the dynamic strategy (indicated by the red dotted line) marginally outperforms the static strategy with À 0.6. Considering dynamic strat- egy requires additional computation burden, we

---

Statities
Dataset
Language
Topic
Source
#Signs
#Videos (avg.
#Signers
2200NI (Camgoz al., 2018) CSL-Daily (Zhou et al., 2021)
DGS
1,870
8,257 (9)
41/1870
Weather
TV
CSL
2,000
20,654
0/2,000
Daily
Lab
Table Statistics of the two sign language benchmark datasets used this work. DGS: German Sign Language; CSL: Chinese Sign Language; #Signs: number of unique glosses in the entire dataset; #Videos: number of sign videos in the entire dataset; avg.: average gloss in each video; #OOV: out-of-vocabulary glosses that occur in dev and test sets not the train set; #Signers: number of individuals the entire dataset.
PHOENIX-2014T
CSL-Daily
Parameter
PHOENIX-20144 CSL-Daily
Method
Dev
Test
Dev
Test
encoder-layers
SMKD
19.64
20.01
29.32
30.02
decoder-layers
attention heads
8
Table 12: Evaluation of the pre-trained SMKD sign features extractor on WER (%)(the lower the better).
etc-layers
hidden size
512
512
activation function
gelu
gelu
gloss forced aligner to identify the temporal bound- aries between glosses continuous sign video frames. Considering that the CTC classifier is up- dated during the training stage, we report the perfor- mance of the aligner when the model training has reached relatively stable state (e.g., epoch=25), thereby illustrating the effectiveness of the sign- gloss forced aligner. Specifically, we report the performance of the aligner for both the "+Cross- modality Mix-up" and the "+ XmDA" schemes Section 3. As listed Table 13, we present the WER on both the training and test sets to demon- strate the effectiveness and generalizability of the aligner.
learning rate
1 10
1 10 3
Adam
(0.9, 0.98)
(0.9, 0.98)
label-smoothing
0.1
0.1
max output length dropout
50
0.3
0.3
batch-size
128
128
Table 10: Hyperparameters of Sign Language Trans- former madels.
K
K=0
K=1
K=2
K=4
K=8
BLEU
22.90
23.54
24.33
24.82
24.93
Table 11: The impact of gloss-to-text teacher number. The BLEU S2T references the BLUE-4 score on PHOENIX-2014T dev set with "+ Cross-modality KD".
PHOENIX-2014T
CSL-Daily
Method
Train
Test
Train
Test
evaluations for each of these components.
Baseline SLT 9.48
26.34
6.03
38.48
+ Mix-up 9.93
26.01
5.86
37.60
A.4.1 Performance Evaluation of the SMKD Sign Features Extractor
+ XmDA 7.66
25.84
4.65
33.41
Table 13: Evaluation of the CTC Classifier, i.e., sign- gloss aligner, effectiveness and generalizability with WER (%) (the lower the better).
Following Zhang et (2023a), we adopt the vi- sual pre-trained model SMKD (Min et al., 2021) to extract sign video frame representations. Here, we report the Word Error Rate (WER) as the metric for sign features extractor, where measuring the simi- larity between the predicted gloss sequence and the ground truth. The results are listed in Table 12
A.4.3 Performance Evaluation of the Gloss2Text teachers model
The Gloss2Text teacher models in XmDA ap- proach crucial for Cross-modality Knowledge Distillation. Their performance directly impacts the quality of the generated spoken language texts used for softening target labels. We evaluate their translation performance in Table 15.
A.4.2 Evaluation of Sign-Gloss Forced Aligner Performance
Section 2.3, as demonstrated, we perform Cross- modality Mix-up using the CTC classifier as a sign-

---

PHOENIX-2014T
CSL-Daily
Model
Precision
Recall
F1
Precision
Recall
F1
Sign Language Transformers
0.8625
0.8884
0.8753
0.9037
0.9248
0.9142
+ Cross-modality Mix-up
0.8685
0.8929
0.8805
0,9075
0.9275
0.9174
+ Cross-modality KD
0.8691
0.8942
0.8815
0.9081
0.9272
0.9176
XmDA
0.8945
0.9157
0.9051
0.9094
0.9288
0.9190
Table 14: BERT Score evaluation results for different models PHOENIX-20L4T and CSL-Daily Test datasets.
ID
ID=0
ID=1
ID=2
ID=3
ID=4
BLEU
27.35
27.24
27.11
27.02
26.93
ID
ID=5
ID=6
ID=7
ID=8
ID=9
BLEU
26.90
26.84
26.83
26.83
26.82
Translation performance the Gexss2Text teacher models on the dev sorted by BLEU score. ID refers to the unique identifier of the Gloss-to-Text teacher model
Table 15:
A.5 BERTScore Evaluation
We have incorporated the BERTScore evaluation for comprehensive assessment of our model's per- formance. This evaluation metric provides preci- sion, recall, and F1 measures to quantify the quality translations.
The results clearly indicate that our proposed XmDA model surpasses the baseline across all met- rics, showcasing its superiority in terms of BERT- score.


Reviews:
[{"review_id": "jCybnFDFEY", "reviewer": "Reviewer_VzGU", "summary": "", "questions": "Is the cross-modality training taking place after training of the CTC classifier or is it end-to-end? It looks like there is a separate path for cross-modal mixup, and thus minimizing JSD loss between sign language distribution and multimodal distribution is not simultaneously taking place while also minimizing the MLE loss. Please explain them in section 2 clearly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper.   We summarize and address the comments below:\n\n    1. Other metrics \nFollowing your recommendation, we have incorporated the BERTScore evaluation. The results, which include precision, recall, and F1 measures, are presented below:\n\n\n| Model                           |     |     PHOENIX-2014T    |       |         |   CSL-Daily      |       |\n|-------------------------------|:-------------------:|:-----:|:-----:|:------------------:|:-----:|:-----:|\n|                                 | Precision          | Recall| F1    | Precision          | Recall| F1    |\n| Sign Language Transformers      | 0.8625             | 0.8884| 0.8753| 0.9037             | 0.9248| 0.9142|\n| + Cross-modality Mix-up         | 0.8685             | 0.8929| 0.8805| 0.9075             | 0.9275| 0.9174|\n| + Cross-modality KD             | 0.8691             | 0.8942| 0.8815| 0.9081             | 0.9272| 0.9176|\n| + XmDA                          | **0.8945**         | **0.9157**| **0.9051**| **0.9094**     | **0.9288**| **0.9190**|\n\nTable 1: BERTScore evaluation results on PHOENIX-2014T and CSL-Daily Test datasets.\n\nThe results show that the proposed XmDA outperforms the baseline in terms of BERT-score. \n\n    2. Case study and proper error analysis\n\nTo delve deeper into our model's performance, we examined the outputs and identified intriguing trends.\n\n● Low-frequency word prediction\n| System |  Output | \n| ------ | ------ | \n| Reference | **ich wünsche** ihnen noch einen schönen abend |\n| Baseline | ihnen einen schönen abend und machen sie es gut |\n| + XmDA |  **ich wünsche** ihnen einen schönen abend |\n\n● Long sentence generation\n| System |  Output | \n| ------ | ------ | \n| Reference | da haben wir morgen schon die dreißig grad morgen im süden von frankreich auch und für uns wahrscheinlich schon im südwesten die fünfundzwanzig grad |\n| Baseline | dort morgen bis dreißig grad im äußersten süden und auch im südwesten |\n| + XmDA |  da haben wir morgen auch die dreißig grad schon über frankreich und auch in süddeutschland haben wir noch die wärme schon mal über fünfundzwanzig grad im südwesten |\n\nXmDA significantly improves translations for low-frequency words and long sentences over the baseline system. The case study will be incorporated in the revised version of the paper to illustrate this improvement further.\u000b\n\n     3. About framework description (Figure 2 and Section 2)\n\nWe appreciate the feedback regarding Figure 1 and Section 2 of our paper. The proposed XmDA represents a joint training framework. In Figure 2, both dotted and solid lines operate simultaneously to train the model, with the dotted line specifically illustrating the forward path for mixed-up samples. This will be further clarified in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Well-motivated loss formulations.\n2. Cross-modal knowledge distillation and transfer seem helpful for learning sign-gloss interaction, and in that sense, the paper is novel.\nThe authors demonstrate the superiority of their proposed method on two datasets: RWTH Phoenix 2014T and CSL-Daily.\n3. Seem to work better for Long sentences.", "reasons_to_reject": "1. Authors are suggested to use other metrics to evaluate the Results (e.g. BERTScore).\n2. Often it is not sufficient to show automatic evaluation results. The author does not show any human evaluation results and does not even perform a case study and proper error analysis. This does not reflect well on the qualitative aspects of the proposed model.\n3. It is difficult to understand the methodology without Figure 1. Parts of section 2 should be written in alignment with Figure 1, and the authors are expected to follow a step-by-step description of the proposed method. (See questions to authors)", "questions_to_authors": "Is the cross-modality training taking place after training of the CTC classifier or is it end-to-end? It looks like there is a separate path for cross-modal mixup, and thus minimizing JSD loss between sign language distribution and multimodal distribution is not simultaneously taking place while also minimizing the MLE loss. Please explain them in section 2 clearly.", "paper_topic_and_main_contributions": "This paper is about end-to-end sign language translation. The authors have proposed an end-to-end system with several interesting components that obtain state of the art. Though the proposed work has some problems (see Reasons to reject), overall the ideas presented are model agnostic and thus can be applied across multimodal tasks, featuring its importance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sE5uL6Uvww", "reviewer": "Reviewer_5AGz", "summary": "", "questions": "- Are the multiple references generated in \"XmKD\" also used in \"MLE\" (the cross-entropy between \"S\" and \"T\")? If yes, it is not clearly shown in Figure 1. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper. We summarize and address the comments below:\n\n    1. About the novelty \n\nOur XmDA method comprises two essential components: Xm Mix-up and XmKD. While the core concept of Xm Mix-up bears similarity to STEMM, as outlined in Section 4.1.1, we believe that our study's primary contribution lies in addressing data scarcity in sign language translation. This point will be further clarified in the revised version.\n\n    2. Multiple references used in XmKD\n\nThank you for pointing that out. Indeed, we utilize the multiple references in the proposed XmKD. We apologize for any confusion caused by the oversight in Figure 1.  We will ensure that this is depicted in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is sound and reasonable. The experimental results, together with the analysis, support the claims.\n- The analysis on the prediction of low-frequency words and long inputs bring extra useful insights than just evaluation on metrics such as BLEU. ", "reasons_to_reject": "- The novelty of the training method could be very weak for speech-translation researchers since it is almost identical to a series of work in reducing modality gap between speech inputs and textual inputs, in particular Fang et. al 2022 (STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation). ", "questions_to_authors": "- Are the multiple references generated in \"XmKD\" also used in \"MLE\" (the cross-entropy between \"S\" and \"T\")? If yes, it is not clearly shown in Figure 1. ", "paper_topic_and_main_contributions": "This paper presents a training method that exploits the data triplet of \"video (S), gloss-labels (G) and spoken-language-translation (T)\" to reduce the modality gap in direct sign language translation (SLT). The two main components are cross-modality (Xm) 1) data augmentation (DA) and 2) knowledge distillation (KD). \n\n\"XmDA\" extends the idea of Mixup used in images or speech to create pseudo-data without using extra data, such as self-training or back-translation. The visual input is transformed to a sequence of contextual representations by a pre-trained visual model \"SMKD\" whereas the corresponding \"gloss\" labels are transformed to a sequence of textual representations by a pre-trained \"Gloss2Text\" model. By using the visual-to-gloss alignment obtained from a CTC classifier, a pseudo sequence which contains (possibly) two modalities in its temporal dimension, depending on the sampled value and the mixing ratio, is generated for training. Its loss is a sum of Jensen-Shannon Divergence (JSD) (between the translation-label distribution of XmDA and that of visual input) over the time steps of the target. \n\n\"XmKD\" uses multiple pre-trained \"Gloss2Text\" models to generate diverse targets (hard-labels), i.e, the spoken-language-translation for training. \n\nSimilar to previous work, it also uses a CTC loss to strength the alignment between \"S\" and \"G\" in the translation encoder and uses a cross-entropy loss to capture the alignment between \"S\" and \"T\" in the translation encoder-decoder. In total, there are 3 losses in training: 1) the CTC loss, 2) cross-entropy loss and the 3) JSD loss from XmDA. \n\nExperimental results on 1) PHENIX-2014T dataset and 2) CSL-Daily dataset shows the effectiveness of \"XmDA\", \"XmKD\" and their combined effect. There are also studies on the improvement on low-frequency words and long inputs. \n\n## Comment\nThe proposed method is sound and reasonable. The experimental results supports their claims. A drawback of the proposed method would be the reliance on the data triplets, e.g., the gloss-labels for modality mixing and CTC alignment. Since the data triplet is used, it would be informative to show the performance of a cascade system. Secondly, the novelty of the training method could be very weak for speech-translation researchers since it is almost identical to a series of work in reducing modality gap between speech inputs and textual inputs, in particular Fang et. al 2022 (STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation). \n\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8LdSCcbXWo", "reviewer": "Reviewer_KVjs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper. We summarize and address the comments below:\n\n    1. About the novelty \nOur XmDA method comprises two essential components: Xm Mix-up and XmKD. While the core concept of Xm Mix-up bears similarity to STEMM, as outlined in Section 4.1.1, we believe that our study's primary contribution lies in addressing data scarcity in sign language translation. This point will be further clarified in the revised version.\n\n    2. Soundness of leveraging CTC as a sign-gloss forced aligner (Section 2.3).\n\n ● 1）The CTC classifier is a more general and prevalent component in sign language translation [1, 2, 3]. Our selection of the CTC classifier enables the sign-gloss forced aligner to function effectively within the end-to-end training setting.\n\n● 2）Addressing concerns about the CTC alignment\n\na. We do not need to address <blank> and incorrect output issues because the sign-gloss forced aligner segments sign video based on the predicted gloss distribution rather than the final CTC outputs. \n\nb. As shown at Eqt. (4), sign-gloss forced aligner is to search the best borders for certain ground truth glosses by maximizing the marginal probability. It would not consider the <blank>  and other tokens due to the constraints of ground truth glosses, e.g., \"i love cat\".\n\n● 3） Regarding the performance of the CTC classifier, the cross-modal mixup is implemented during the training phase, yielding a Word Error Rate (WER) of 7.7,  as referenced in Table 9. This is on par with human performance[4]. \n\nWe appreciate your insightful comments and constructive recommendations. The suggested content will be incorporated into the revised version of the paper.\n\n    3. About the analysis.\n\n● 1） Quantitative analysis using word-level similarity or sequence-level similarity \n\na. Thank you for your suggestion. We have reproduced the visualization analysis based on the hidden representation of the final encoder layer. The outcomes align closely with our current findings. We will showcase these new results in our final version.\n\nb. In response to quantitative analysis, we utilized Kernel Density Estimation (KDE) to estimate the probability density functions on those tree types of embeddings[5,6]. The entropies resulting from these KDEs are as follows:\n\n\n| Embedding Type                  | KDEs Entropy |\n|---------------------------------|:------------:|\n| Gloss Embeddings                | 0.19         |\n| Sign Embeddings (Baseline)      | 2.18         |\n| Sign Embeddings (XmDA)          | 0.84         |\nTable 2: Entropies resulting from KDEs on different types of embeddings.\n\nc. Additionally, to provide a quantitative measure of alignment, we compared the average Euclidean distance and cosine similarity at the word level between sign embeddings and gloss embeddings. The results are detailed below:\n\n\n\n\n| Comparison                                      | Euclidean Distance | Cosine Similarity |\n|-------------------------------------------------|:------------------:|:-----------------:|\n| Sign Embedding (Baseline) vs. Gloss Embedding   | 14.6               | 0.19              |\n| Sign Embedding (XmDA) vs. Gloss Embedding       | 8.68               | 0.34              |\n\nTable 3: Average Euclidean distance and cosine similarity between sign embeddings and gloss embeddings.\n\nThe above results show that our XmDA technique significantly improves the sign embedding representations over the baseline model. \n\n● 2） Deeper analysis of the phenomenon in Table 3 and Table 4. \n\nThank you for pointing out the phenomena observed in Tables 3 and 4. In response, we examined the outputs and identified intriguing trends through two case studies:\n\n**Case A: Low-frequency word prediction**\n\n| System       | Output                                                                        |\n|--------------|-------------------------------------------------------------------------------|\n| Ref          | **ich wünsche** ihnen noch einen schönen abend                                |\n| Baseline | ihnen einen schönen abend und machen sie es gut                               |\n| + Xm KD      | jetzt **wünsche ich** ihnen noch einen schönen abend und hoffe                |\n| + XmDA       | **ich wünsche** ihnen noch einen schönen abend                                |\n\n**Case B: Long sentence generation**\n\n| System          | Output                                                                                                                              |\n|-----------------|-------------------------------------------------------------------------------------------------------------------------------------|\n| Ref             | da haben wir morgen schon die dreißig grad morgen im süden von frankreich auch und für uns wahrscheinlich schon im südwesten die fünfundzwanzig grad |\n| Baseline    | dort morgen bis dreißig grad im äußersten süden und auch im südwesten                                                               |\n| + XmDA          | da haben wir morgen auch die dreißig grad schon über frankreich und auch in süddeutschland haben wir noch die wärme schon mal über fünfundzwanzig grad im südwesten |\n\n\nXmDA significantly improves translations for low-frequency words and long sentences over the baseline system. The case study will be incorporated in the revised version of the paper to illustrate this improvement further.\u000b\n\n\n**Reference**\n\n[1]  Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation CVPR 2020\n\n[2]  A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation CVPR 2022\n\n[3]  CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment CVPR 2023\n\n[4]  Achieving human parity in conversational speech recognition  Microsoft Research Technical Report 2017\n\n[5]  Ensemble estimation of multivariate f-divergence IEEE 2014\n\n[6]  Density estimation for statistics and data analysis. CRC 1986", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper sucessfully introduces Cross-Modality Mixup and Cross-modality Knowledege Distillation, to tackle the modality gap and data scarity of end-to-end sign language translation. The experiments show that the proposed method significantly improves the SOTA (about 4BLEU on two dataset).", "reasons_to_reject": "1. The two introduced techniques are well-studied in speech translation field. Leveraging mixup and JD loss to bridge the modality gap is proposed by [1]. Leveraging gloss-to-text model to translate gloss from video-gloss data into language text is very similar to leveraging MT model to translate the source text from ASR data into target text [2].\n\n2. My main concern is the soundness of leveraging CTC as a sign-gloss forced aligner (Section 2.3). MFA (a HMM-GMM model) is widely used as speech-transcription forced aligner, there are two reasons why CTC classifier is discard.\n\n   * Firstly, the CTC classifier is more likely to produce <blank> than repeat. Suppose the gloss is \"i love cat\", the CTC classifier is more likely to produce \"_ i _ _ love _ _ cat _\" than \"i i love love love love cat cat cat\". How to deal with the blanks to segment sign video?\n   * Secondly, the CTC classifier may produce wrong output. Suppose the gloss is \"i love cat\", the CTC classifier may generate \"i do like cat\". How to deal with this session to perform force-aligned, especially the performance of CTC model is not good (according to table 8, the WER is about 20).\n\n   If this paper makes extra efforts, they should describe in detail. Otherwise, the soundness of choosing CTC classifier as forced aligner should be discussed.\n\n3. The analysis are not convincing.\n\n   * To prove the effectiveness of mixup mechanism, there are only qualitative analysis without any quantitative analysis. Word-level similarity or sequence-level similarity are more convincing. Also, visulization of the representation produced by translation encoder will be more convincing than on the word embedding level.\n   * In Table 3 and Table 4, this paper shows that the introduced two techniques work on different examples (based on word frequency and sequence length). The phenomenon is interesting, but there is no analysis.\n\n[1] Fang Q, Ye R, Li L, et al. Stemm: Self-learning with speech-text manifold mixup for speech translation. ACL 2021.\n[2] Jia Y, Johnson M, Macherey W, et al. Leveraging weakly supervised data to improve end-to-end speech-to-text translation. ICASSP 2019.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focus on end-to-end sign language translation, which aims at converting sign language videos into spoken language texts without generating itermediate gloss. To tackle the data scaricy, this paper utilizes  gloss-to-text model to perform multi-teacher sequence-level knowledge distillation . To tackle the modality gap, this paper leverages cross-modality mix-up to encourages the feature alignment. Experiments show the effectiveness of two introduced techniques and demonstrate the significant improvements over the SOTA.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes cross-modality data augmentation techniques to tackle the data scarcity issues while training end-to-end sign language translation models. In particular, the paper proposes to use cross-modal mix-up and cross-modality knowledge distillation. The experimental results demonstrate the superiority of the proposed method.

The reviewers liked the method description and analysis. They asked several qualitative and quantitative evaluation questions, but the authors clarified them during the rebuttal. Finally, reviewers pointed out that while the techniques are novel in sign language translation, they are well-studied in speech translation, limiting their excitement.
Decision: Accept-Findings