SLTUNET: A Simple Unified Model for Sign Language Translation
Authors: Biao Zhang
Mathias Müller
Rico Sennrich
Venue: ICLR 2023 poster (2023)
Keywords: Unified Modeling
Multi-task Learning
Sign Language Translation
Cross-modality Learning

Abstract: Despite recent successes with neural models for sign language translation (SLT), translation quality still lags behind spoken languages because of the data scarcity and modality gap between sign video and text. To address both problems, we investigate strategies for cross-modality representation sharing for SLT. We propose SLTUNET, a simple unified neural model designed to support multiple SLTrelated tasks jointly, such as sign-to-gloss, gloss-to-text and sign-to-text translation. Jointly modeling different tasks endows SLTUNET with the capability to explore the cross-task relatedness that could help narrow the modality gap. In addition, this allows us to leverage the knowledge from external resources, such as abundant parallel data used for spoken-language machine translation (MT). We show in experiments that SLTUNET achieves competitive and even state-of-the-art performance on PHOENIX-2014T and CSL-Daily when augmented with MT data and equipped with a set of optimization techniques. We further use the DGS Corpus for end-to-end SLT for the first time. It covers broader domains with a significantly larger vocabulary, which is more challenging and which we consider to allow for a more realistic assessment of the current state of SLT than the former two. Still, SLTUNET obtains improved results on the DGS Corpus. Code is available at https://github.com/bzhangGo/sltunet.

Published as a conference paper at ICLR 2023
SLTUNET: A SIMPLE UNIFIED MODEL FOR SIGN LANGUAGE TRANSLATION
Biao Zhang', Mathias Müller?, Rico Sennrich
1 School of Informatics, University of Edinburgh
2 Department of Computational Linguistics, University of Zurich
b. zhangeed. ac. uk, mmuel lerêcll wwwffcl........ ch, wennricheell uzh. ch
ABSTRACT
Despite recent successes with neural models for sign language translation (SLT), translation quality still lags behind spoken languages because of the data scarcity and modality gap between sign video and text. To address both problems, we investigate strategies for cross-modality representation sharing for SLT. We pro- pose SLTUNET, a simple unified neural model designed to support multiple SLT- related tasks jointly, such sign-to-gloss, gloss-to-text and sign-to-text transla- tion. Jointly modeling different tasks endows SLTUNET with the capability to explore the cross-task relatedness that could help narrow the modality gap. In addition, this allows us to leverage the knowledge from external resources, such as abundant parallel data used for spoken-language machine translation (MT). We show in experiments that SLTUNET achieves competitive and even state-of-the- art performance on PHOENIX-200 data and equipped with set of optimization techniques. We further the DGS Corpus for end-to-end SLT for the first time. It covers broader domains with a significantly larger vocabulary, which is more challenging and which consider to allow for a more realistic assessment of the current state of SLT than the for- mer two. Still, SLTUNET obtains improved results on the DGS Corpus. Code is available at https: / /gggtb...bb/ggggggggggggg..ghh com/ www.bggoaaaaaaeoeooeoo/aoooaagas /asecessstueetest ttnne..
and CSI.-Daily when augmented with MT
INTRODUCTION
The rapid development of neural networks opens the path towards the ambitious goal of universal translation that allows converting information between any languages regardless of data modalities (text, audio or video) (Zhang, 2022). While the translation for spoken languages (in text and speech) has gained wide attention (Aharoni et al., 2019; Inaguma et al., 2019; Jia et al., 2019), the study of sign language translation (SLT) task translating from sign language videos to spoken language texts still lags behind despite its significance in facilitating the communication between Deaf com- munities and spoken language communities (Camgoz et al., 2018; Yin et al., 2021). SLT represents unique challenges: it demands the capability of video understanding and sequence generation. Un- like spoken language, sign language expressed using hand gestures, body movements and facial expressions, and the visual signal varies greatly across signers, creating tough modality gap for its translation into text. The lack of supervised training data further hinders us from developing neural SLT models of high complexity due the danger model overfitting.
Addressing these challenges requires us to develop inductive biases (e.g., novel model architectures and training objectives) to enable knowledge transfer and induce universal representations for SLT. In the literature, a promising way is to design unified models that could support and be optimized via multiple tasks with data from different modalities. Such modeling could offer implicit regularization and facilitate the cross-task and cross modslityylyty transfer learning that helps narrow the modality gap and improve model's generalization, such as unified vision- Languule modeling (Jaegle et al., 2022; Bao et al., 2022; Kaiser et al., 2017), unified speech-text modeling (Zheng et al., 2021: Tang et al., 2022: Bapna et al., 2022), multilingual modeling (Devlin et al., 2019: Zhang et al., 2020: Xue et al., 2021), and general data modeling (Liang al., 2022; Baevski et al., 2022). In SLT. different annotations could be paired into different tasks, including the sign-to-gloss (Sign2Gloss), the sign- to-text (Sign2Teext), the gloss-to-text (Gloss2Text) and the text-to-gloss (Text2Gloss) task. These

---

Published as a conference paper at ICLR 2023
Add taak and position Information
Feature Extractor
Visual Encoder
CTC Loss
MLE Loss
Shared Decoder
Sign Video
Shared Encoder
(autorcgressive)
Embedding
Textual Encoder
Gloss Text
Gloss or Text
(a) SL.TUNET Illustration
Task
Task Tag
Input
Output
Training Objective
Sign2Giooss
[2gls]
sign video
gloss
aL L (gloss) aL (gloss) + LMLLE (text)
Sign2Text
[2xx]
sign video
text
Gloss2Text
[2xx]
gloss
text
M.L.E (text)
Text2Gloss
[2gls]
text gloss
MLL C (gloss)
Machine Translation
[2xx]]
source text
target text
C (Marget)
(b) Tasks Explored
Figure I: Overview the proposed SLTUNET and the tasks we explored. SLTUNET adopts separate encoders to capture mppelity-specific (visual and textual) characteristics followed by a shared encoder to induce univer- sal features. It employs an autoregressive decader shared across tasks for generation. SLTUNET optimizes the whole model via the maximum likelihood estimation (MLE) objective and optionally the connectionist tem- poral classification (CTc) objective and uses Transformer as its backbone. It supports multiple tasks, such as the sign-to-gloss (Sign2Gloss), the sign-to-text (Sign2Text), the gloss-to-text (Gloss2Text). the text-to-glos (Text2Gloss) and the machine translation task. We regard the embedding of the corresponding task tag ((2gls) or [2txt]) as the task information to guide the generation, and append it in front of the input feature sequence inspired by multilingual NMT. is hyperparameter; blocks in colour (except gray) indicate trainable param- eters; note Text2Gloss hurts SLT in our experiments and not involved in the final joint objective.
tasks are often modelled separately. Whether unified modeling for them could benefit SLT and what inductive biases are adequate for SL. are still open questions, which are the exact focus of this study.
In this paper, we propose a simple unified model for SLT, namely SLTUNET, to answer the above questions. As in Figure 1, SLTUNET follows the encoder-decoder paradigm (Bahdanau et al., 2015) with Transformer (Vaswani et al., 2017) as its backbone and supports multiple vision/language-to language generation tasks. It uses shared modules to encourage knowledge transfer and adopts separate visual/textual modules to avoid task or modality interference (Liang et al., 2022). Thanks to its unified schema, SLTUNET allows us to leverage external data resources from other related tasks, such machine translation. This partially alleviates the data scarcity issue and opens up the possibility of exploring relatively larger models for SLT. We further examine and develop set of optimization techniques ensure the trainability of SLTUNET.
We conducted extensive experiments on two popular benchmarks, PHOENIX-20141 (Camgoz et al., 2018) and CSL-Daily (Zhou al., 2021) for German and Chinese Sign Language, respectively. Following previous evaluation protocols (Camgoz et al., 2018), we test SLTUNET on several SLT- related tasks but with a single trained model. Results show that SLTUNET achieves competitive and even state-of-ther thett-rrtttt performance, surpassing strong baselines adopting pretrained language models.
We note that PHOENIX-20147 and CSL-Daily, while offering valuable testhed for SLT,, are lim- ited in various aspects. They feature a small number of signers, and are limited in linguistic variety with a small vocabulary. As a more challenging, larger-scale SLT dataset, we propose to use the Public DGS Corpus (Hanke et al., 2020a) that covers broader domains and more open vocabularies, and gives a more realistic view of the current capability of SLT. We also take care in following best practices regarding preprocessing and evaluation (Müller et al.. 2022). We find that the challeng- ing nature of the DGS Corpus results in generally low SLT performance, but we still observe some quality gains with SLTUNET. Our contributions summarized below:
2

---

Published as a conference paper at ICLR 2023
We propose simple unified model, SLTUNET, for SLT, and show that jointly modeling multiple SLT-related tasks benefits the translation.
We propose set of optimization techniques for SLTUNET aiming at an improved trade-off between model capacity and regularization, which also helps SLT models for single tasks.
We use the DGS Corpus and propose a translation protocol for end-to-end SLT, with larger scale, richer topics and more significant challenges than existing datasets.
SLTUNET performs competitively to previous methods and yields the new state-of-the-art performance on CSL-Daily.
2 RELATED WORK
Our study on SLT focuses on transforming a sign language video to a spoken language text. Previous methods can be roughly classified into two categories: cascading and end-to-end.
The cascading method relies on an intermediate output such as sign glosses (Camgoz et al., 2018) where each gloss is manual transcription for sign to reflect its meaning. Cascading systems break SLT down into two separate tasks: sign language recognition that transcribes a continuous sign video to gloss sequence (Sign2Gloss) and gloss-to-text translation that transforms the glosses spo- ken language text (Gloss2Text), Sign2Gloss requires the modeling of temporall relations of a sign video to achieve video understanding, which often demands advanced optimizations and archi- tectures, such as 2D/3DDcomvolutional or recurrent encoders (Cui et al., 2017; Koller et al., 2020), spatial-temporal multi-cue network (Zhou al., 2022), self-mutual distillation learning (Hao al., 2021), and cross- cross-modality augmentation (Pu et al., 2020), etc. By contrast, Gloss2Text resembles machine translation (MT) but suffers greatly from data scarcity (Yin & Read, 2020). Recent studies often explore techniques from MT to alleviate this problem, such as data augmentation (Moryossef al., 2021; Angelova al., 2022) and using pretrained language models (De Coster et al., 2021; Cao et al., 2022). Unfortunately, sign glosses are not equivalent to their corresponding sign video and often drop information. This imposes hard performance cap on cascading SLT.
We thus focus on the end-to-end method instead, which converts sign videos directly to natural texts (Sign2Text). Camgoz et al. (2018) pioneered this direction by framing the task as neural MT problem and showed the feasibility of the encoder-decoder paradigm (Bahdanau et al., 2015). Later studies followed this paradigm and put efforts into improving the sample efficiency and reducing the dision-language modality gap. Camgoz et al. (2020a) and Zhou et al. (2022) developed multi- channel neural models to leverage information from different visual cues (such as hand shapes and facial expressions) to enhance sign language understanding. Li et al. (2020) and Kan al. (2022) proposed hierarchical neural models to capture spatio-temporal features at multiple levels of granu- larity in sign videos. Zhou et al. (2021) explored sign back-translation to construct pseudo-parallel training data for SLT based on monolingual texts. Jin et al. (2022) investigated the use of external prior knowledge. Different from the above studies, we focus on unified modeling for SLT with the goal of transferring knowledge across different tasks and particularly improving Sign2Text.
Our study is closely related multi modality transfer learning (Chen et al., 2022), with signifi- cant differences. Chen et al. (2022) employ Sign2Gloss and Gloss2Text tasks to perform in-domain pretraining for public large-scale pretrained visual and language models, respectively, followed by specific finetuning on Sign2Text. Their method follows the pretraining-finetuiig paradigm and focuses on adapting pretrained models to SLT instead of joint unified modeling and multitask learn- ing. Note, we train SLTUNET multiple tasks without relying on pretrained language models and SLTUNET achieves state-of-the th-the- results on CSL-Daily. Although using sign glosses to regularize the neural encoder is popular in Sign2Text (Camgoz al., 2020b; Zhou et al., 2021; Chen et al., 2022), the study of jointly modeling multiple SLT-related tasks (> 2 tasks) via a single network and the exploration of MT data to improve Sign2Text have never been investigated before.
SLTUNET MODEL
We aim to design a unified model for SLT to improve the translation by utilizing diverse SLT-related tasks. To this end, we propose SLTUNET which supports general hatpa//lannuaget--lannngug gen- eration tasks. Figure la illustrates the overall architecture and Figure 1b summarizes the tasks
3

---

Published as a conference paper at ICLR 2023
explored. Note the design of SLTUNET considers the capacity trade-off practice (Zhang et al., 2021; Liang et al., 2022) with the majority of parameters shared for knowledge transfer while the rest kept separate to capture modality- features.
SLTUNET follows the encoder-decoder framework and models the conditional generation probabil- ity. In general, it takes input task tag tag informing the model which task it's handling and feature sequence X E IR [X]x and then builds neural network to predict the ground-truth reference sequence Y = {11 9/,- ...
y/rr)
(I) where a d denote the sequence length and model dimension respectively, and Y E TRl is the right-shifted input feature sequence used for autoregressive decoding. o represents the chaining of two modules. X E |x|x|xd and Y IR [g are the encoder and decoder outputs, respectively.
x 0 Encoder Encoder (X,tag) Y = Decoder (Y'.x)
We adopt Transformer (Vaswani et al., 2017) as the backbone for SLTUNET. (ecoder(-) indicates the autoregressive Transformer decoder with Ndec layers; Encoder (.) and Encoder" (-) stand for shared and modality-specific Transformer encoders with N and N enc layers, respectively. In- spired by multilingual neural MT (Johnson et al., 2017), we append the embedding of tag in front of X along the time axis and feed the concatenated sequence to the encoder.
Different tasks have different training objectives and different ways to construct X. Depending on the input modality to the encoder, SLTUNET have the following two working modes:
1) When the task has sign video inputs, Encoder (-) denotes the textual encoder in Figure la and the input feature X is obtained via a word embedding layer. We train SLTUNET via the following objective:
C(Y)X, tag) (Y|LEE(YY))))
where X denotes the input of X, (.) is the maximum likelihood estimation (MLE) objective.
2) Otherwise, Encoder (.) denotes the visual encoder in Figure la and prepare sign embed- dings X based on some pretrained visual models. In particular, we adopt the SMKD model (Hao al., 2021) and extract its visual features, i.e. the output of 1D temporal convolution, as the sign features. We further project these features to the model dimension via a linear layer to form X. Note the parameters of SMKD are frozen when training SLTUNET, The training objective is: C(Y, Z X, lag) = L (Y (Y|Y²) + aL (2
where (.) is the connectionist temporal classification (CTC) objective (Graves et al., 2006) and Z denotes the gold label sequence for CTC, which is often the gloss sequence in SLT. Differ- ent from MIF,, CTC models the probability distribution by marginalizing over all valid mappings between its input (X*) and output (Z) sequence. CTC has been widely used in SLT to regularize the sign encoder (Camgoz et al., 2018; Chen et al., 2022), and we follow this practice and use a hyperparameter to balance its effect. Note the CTC part will be dropped after training.
As shown in Figure 1b, SLTUNET offers high flexibility accommodate different SLT-related tasks. Also, it allows us to explore knowledge from other tasks by leveraging their abundant training data, such as machine translation. Formally, given a SLT training sample (sign video, gloss sequence, text translation) denoted by (V,G, T.) and MT sample (source text, target text) denoted by (S,T) T"), the final SLTUNET training objective is formulated below:
SLTUEET C(G, + ((T,GIV,22) (2x, + ((T]G,121xt)) + 1(x))) (4)
Sign2Gloss
SignZText
Gloss2Text
Machine Translation
where we adopt multi-tast learning schema and treat different tasks equally for training. Note, we exclude Text2Gloss in the final objective and only retain the CTC objective for Sign2Text based on our preliminary experiments, and we mix SLT and MT samples during training based on prede- fined ratio. At testing, examine SLTUNET under two modes the end-to-end (Sign2Text) and cascading (Sign2Gloss + Gloss2Text) mode- using a single trained model.
Optimization for SLTUNET Covering multiple tasks entails more training data and reduced risk of model overfitting. This gives us the opportunity to increase the modeling capacity for SLTUNET by adjusting the model depth and width. Meanwhile, we still need to control the degree of model regularization via e.g., dropout rates to achieve the full potential of SLTUNET. All these make the optimization of SLTUNET challenging and we will examine different methods in the experiments
4

---

Published as a conference paper at ICLR 2023
Attribute
Statistics
Dataset
Lang
Resolution
Doc. #Signers
Vocab
#OOV
#Train
#Dev
#Test
PHOENIX-200 2014T DGS 210 x 260
X
9
1,085/2,887
30/113 7,096
519
642
CSL-Daily
CSL 1920 1080
X
10
2,000/2,277
0/37
18,401
1,077
1,176
DGS3-T
DGS 640 X 360
/
330 8,580/23,363
105/647
60,306
967
1,575
Table I: Summary of different SLT datasets. Lang: language; DGS: German Sign Language; CSL: Chinese Sign Language; Doc.: whether samples are organized the form document; #Signers: number of individuals in the entire dataset; Vocab: number of glosses/spoken words in the training set (note we count characters for Chinese); #OOV: out-of-vocabulan glosses/words that occur in dev and test sets but not in the train set; #Train/DDew#Testt number of samples in the train/dev/tsst set, respectively.
4 EVALUATING END-TO-END SYSTEMS ON BARGER-SCALA DATA
Although popular benchmarks PHOENIX-20 20144T (Camgoz et al., 2018) and CSL-Daily (Zhou et al., 2021) offer a valuable testbed for SLT, we note that they suffer from limitations such as training data size, the size of their gloss and spoken language vocabulary, the number of signers, and domains and topics covered as shown in Table 1. For example, both benchmarks feature a vocabulary of <3000 spoken language words, representing just fraction of the vocabulary typical in spoken language MT systems. Thus, existing results may give too rosy an impression of the current capability of SLT models.
We therefore use the Public DGS Corpus (Hanke et al., 2020b), as a broader-domain and more realistic testbed. The DGS Corpus is a dataset featuring German Sign Language (DGS), German and English. It includes data collected from 330 signers from 12 different locations in Germany. The signers were balanced for gender, age, and region, and the data covers various linguistic domains (such story telling and conversations). Whereas previous work has focused on Gloss2Text (Müller et al., 2022: Angelova et al., 2022), our focus lies in evaluating and improving the Sign2Text task.
We create locument-level dataset split, which offers room to study contextual modeling in the future. The split contains 60,306, 967, and 1,575 samples the train, dev, and test set, respectively (see Table I and Appendix A.2 for details). We will refer to this dataset as DGS3-T for short, referring to the fact that we use release of the Public DGS Corpus and that we use for translation tasks ("T"") rather than vision tasks such sign language production (Saunders et al., 2022). Similar to previous SLT datasets, each sample in DGS3-T is a triplet consisting of a sign video, sentence- level gloss annotation and the German translation (besides other annotations). DGS3-T has large vocabulary with 8,580 glosses and 23,363 spoken language words, posing considerable practical challenges.
EXPERIMENTS
5.1
SETUP
Datasets We work on three SLT datasets: PHOENIX--0144, CSL-Daily, and DGS3-T. PHOENIX- 2014T and DGS3-T focus on German Sign Language, CSL-Daily on Chinese Sign Language. All three datasets provide triplet samples, each consisting of sign language video. sentence-level gloss annotation and their corresponding text translation. Detailed statistics are listed Table We employ MuST-C English-German (En-De, 229K samples) and English-Chinese (En-Zh, 185K samples) (Di Gangi et al., 2019) as the augmented MT data for CHOENIXX-014T//DGGSS CSL-Daily, respectively. We Iearn joint vocabulary for glosses and texts via byte pair encoding (BPE) (Sennrich et al., 2016). We employ IK BPE operations when MT data is not used, and otherwise.
and
increase it to 8K/8K/10K for
Model Settings We experiment with Transformer (Vaswani et al., 2017) and start our analysis with Baseline system optimized on Sign2Text alone with the following configurations: encoder and decoder layers of N enc = 2, 2 P enc = 0 and N& = respectively, model dimension of d 512, Reed-forward dimension of dif 2048, attention head of h 8, and CTC regularization. We
5

---

Published as a conference paper at ICLR 2023
ID System
#params B@41
1 Baseline
15.8M
22.62
1.1 sign embeddings from (Camgoz et al., 2020b)
15.8M
21.21
Explore CTC Regularization
2
CTC (a 0.3)
16.3M
24.04
2 relative positional encoding (Shaw et al., 2018) (A: = 16)
16.3M
23.92
2 a = 0.2
16.3M
23.71
2 0.4
16.3M
23.79
Explore Multi-task Learning
3 2 multi-task training (Equation 4 without MT)
16.3M
25.10
3.1 3 add Text2Gloss for training
16.3M
24.88
3.2 remove Sign2Gloss training
16.3M
23.96
3.3 3 remove Gloss2Text at training
16.3M
24.60
4 3 + add MT task (mixing ratio for MT and SLT samples 3:1, vocab size: 1K 8K)
23.6M
25.23
Explore Modality- Modeling
5
4 + add modality specific module (Nene 1, Ne 22, Ndec 3)
34.1M
26.30
5.1 5 add more modality-Sot specific parameters (N = 2, Nacc = 4)
44.6M
25.41
5 + change mix ratio from 3:1 to 5:1
34.1M
25.95
5 apply CTC regularization the output of visual encoder instead
34.1M
25.48
Explore Model Regularization
6 5 apply BPE dropout (Provilkov et al., 2020) glosses and texts of rate 0.2
34.2M
26.04
6 increase BPE dropout rate 0.3
34.2M
25.67
decrease BPE dropout rate to 0.1
34.2M
26.00
7
6 stochastic BPE dropout of stochastic rate 0.5
34.2M
26.50
8 7 apply random crop and horizontal flip (50%) to sign video frames for augmentation
34.2M
26.76
8 L2 weight regularization with coefficient of le
34.2M
26.79
9 8 change the gain hyperparameter in Xavier initialization to 0.5
34.2M
27.11
Explore Larger-Capacity Modeling
10 9 + increase model depth (N = 1, No = 5, Nacc 6)
56.2M
26.56
11 10 + reduce model dimension (d 256, h
23.1M
27.38
11 + more modality-speeiffii parameters (Neee = 2, NS = 4, Naec =
24.5M
27.13
11 + increase feed-forward layer (dys 4096)
36.8M
27.09
12 11 increase model depth ene 1, N = 7, Náec 8)
28.9M
27.39
12 + layer dropout of rate
28.9M
26.99
Explore Capacity- Regularization Capacity-Regulariztto
Balance
11 + increase stochastic rate to
23. IM
27.44
14 13 + increase feed-forward layer (dys = 4096) and its dropout rate to 0.5
36.8M
27.56
SI.TUNE
15 14 + update sign embeddings with improved SMKD model
36.8M
27.87
Table 2:
Ablation study of SLTUNET Sign2Text on the PHOENIX-201 2014T dev set. #params: number of trainable model parameters: B 4: tokenized 4-gram BLEU.
adopt the SMKD model (Hao al., 2021)' 1 to extract sign embeddings, and pretrain the model each benchmark separately on the Sign2Gloss task considering the large difference of sign videos across benchmarks. More details about datasets and model settings are given in Appendix A.I.
Evaluation We report results mainly on the SLT task. Following previous evaluation protocol (Cam- goz et al., 2018), we examine our model via the end-to-end (Sign2Text) and cascading (Sign2Gloss +Gloss2Texx) method for SLT; we measure the translation performance using tokenized BLEU with n-grams from to (B 1-B@4) (Papineni et al., 2002) and Rouge-L (ROUGE) (Lin, 2004), and we employ Word Error Rate (WER) to evaluate Sign2Gloss.?
Metrrc scripts htt.t::p/////. h http:///wcnne.ttcto////////tt//a

---

Published as a conference paper at ICLR 2023
We note that the current evaluation practices in SLT do not align with more general MT research, where B@ 1-B@3 and ROUGE are often considered inadequate to evaluate translation due to their relatively inferior correlation with human judgement. We follow the recent recommendations from MT (Koemi et al., 2021) and further report detokenized BLEU (sBLEU) and ChrF (Popović, 2015) offered by SacreBLEU (Post, 2018), while we acknowledge that how to properly evaluate translation is still an ongoing research topic. Note we always use character-level metries for Chinese translation.
5.2 RESULTS AND ANALYSIS
We perform our main analyses on PHOENIX-20144 and summarize the results in Table 2.
High-quality sign embedding and CTC regularization benefit SLT. Replacing sign embeddings offered by Camgoz et al. (2020b) (24.88 WER, on dev set) with the one from our retrained SMKD model (19.80 WER) greatly improves SLT (+1.41 BLEU, 1.1++1). Changing the visual backbone of SMKD to the 2D Resnet34 pretrained on ImageNet (He et al., 2016) delivers further quality gains (18.90 WER, +0.31 BL.EU, 14-+15). Also, adding CTC regularization helps (+1.42 BLEU, 1 1-+) resonating with previous findings (Camgoz et al., 2020b). We didn't iidn't see obvious benefit from the relative positional representation (Shaw et al., 2018).
Unified modeling via multi-task learning improves SLT and different tasks show different im- pacts. Unified modeling could facilitate knowledge transfer across tasks especially when the tasks are highly correlated. In Table 2, we observe that modeling Sign2Gloss, Sign2Text and Gloss2Text together improves SLT (+1. .06 BLEU, 2-3).. But adding Text2Gloss deteriorates the performance (- 0.22 BLEU, 3 3-1.1). This might be caused by the large gap between text translation and sign video that hinders the transfer in encoder. Sign2Gloss benefits the unified modeling more than Gloss2TText (3.2 vs. 3.3). Leveraging external resources, such as MT data, also helps SLT though the quality gain is small (+0.13 BLEU, 3-44). We still include MT in SLTUNET since it brings in rich training data that could alleviate overfitting and allow us to explore higher-capacity models.
Mixing shared parameters with adequate modality-specife parameters improves the trans- fer. Sharing parameters across modalities/tasks enables knowledge transfer but often at the cost of cross-modality/tskk interference partially due to its insufficiency in describing modality/task-specin characteristics (Wang et al., 2019; Liang et al., 2022). Previous studies also showed the trade-off between shared parameters and task-specific parameters in joint network (Zhang et al., 2021). As shown in Figure la, we incorporate modality-specific (visual and textual) encoders to mitigate the interference, which obtains significant quality boost (+1.07 BLEU, 4 4-55). Further increasing the amount of -poeaiity-specific parameters helps little, though (-0.89 BLEU,5-+5.11
Unified modeling benefits from appropriate degree of model regularization. We next exam- ine a set of regularization techniques for SLTUNET considering the low- low-resource condition of SLT. BPE dropout regularizes neural models by producing diverse subword segmentations of a word with randomness which greatly improves low-resource MT (Provilkov et al., 2020). Unfortunately, di- rectly applying
SLTUNET delivers inferior performance (-0.26 BLEU, 5-6). We then propose a simple variant, named stochastic BPE dropout, that applies BPE dropout to a random proportion of samples. We empirically set the stochastic rate to 0.5, i.e., only 50% of samples are handled by BPE dropout with the rest retained, which slightly improves SLT (+0.2 BLEU, 5-77).
In image processing, popular way of regularization is to augment the training data by applying cropping and flipping operations. We follow Hao et al. (2021) and adopt random crop and horizontal flip (50%) to sign frames, which delivers a gain of 0.26 BLEU (7+8). We find that the traditional L2 weight decay helps little, but changing the gain parameter in Xavier initialization from 1.0 to 0.5 benefits the translation (+0.35 BLEU, 8-99).
Larger-capacity modeling via tuning model depth/width with careful regularization further improves translation. Jointly modeling multiple tasks gives us the chance to explore larger-capacity modeling, but naively increasing model depth (9~>10) hurts the performance greatly (-0.55 BLEU). We then reduce the model dimension from 512 to 256 which delivers positive gains (+0.82 BLEU, 10-+11). On top of it, explore increasing modality-specinf layers or model depth, reducing
NNot we explore the near optimal setting for SLTUNET mainly based on our experience rather than a full- space grid search. Aggressively optimizing the system might offer hetter SI. performance but requires massive computing resources that we can't afford.
7

---

Published as a conference paper at ICLR 2023
Task
Sign2Gloss
Gloss2
Sign2Text
Cascading
Metric
WER.
sBLBUt
ChrPt
sBLEUt
ChrFt
sBLEUt
ChrFt
Single Task
18.36
25.42
50.52
26.56
52.03
24.48
49.49
Single Task w/ MT
19.54
26.69
51.61
27.36
52.72
24.54
49.83
Multi Task
19.24
27.09
52.14
27.87
53.01
25.36
50.75
Table 3: Ablation results of single-task and multi-task training for SLTUNET with the setup of system 15 in Table the PHOENIX 201 dev set. MT: augmenting each SLT task with MT; Cascading: cascading performance on SLT where we chain a Sign2Gloss model and Gloss2 model trained separately for single- task evaluation. Notice that we feed the reference glosses for the Gloss2Text task.
Dev
Test
Task & Systems
ROUGE
B@4
ROUGE
B@!
B@2
B@3
B @
Cascading: Sign2Gloss Gloss2Text
SL-Transf. (Camgoz al., 2020b)
22.11
48.47
35.35
27.57
22.45
(Zhou et al., 2021)
49.53
23.51
49.35
48.55
36.13
28.47
23.51
STMC-Transf. (Yin Read, 2020)
46.31
22.47
46.77
48.73
36.53
29.03
24.00
ConSLT (Fu et al., 2022)
24.31
51.29
38.62
30.79
25.48
VI.-Transfer (Chen et al., 2022)
50.23
24.63
49.59
49.94
37.28
29.67
24.60
SLTUNET (sBLEU: 26.00, ChrF: 51.96)
49.61
25.36
49.98
50.42
39.24
31.41
26.00
End-to-end: Sign2Text
SL.-Transf. (Camgoz et al., 2020b)
22.38
46.61
33.73
26.19
21.32
TN-TIN-Transf- (Zhou et al., 2021)
50.29
24.45
49.54
50.80
37.75
29.72
24.32
STMC-T (Zhou al., 2022)
48.24
24.09
46.65
50.80
37.75
29.72
24.32
PET (Jin et al., 2022)
49.97
49.54
37.19
29.30
24.02
VI.-Transfer (Chen et al., 2022)
53.10
27.61
52.65
53.97
41.75
33.84
28.39
SITUUET (sBLEU: 28.47, ChrF: 53.78)
52.23
27.87
52.11
52.92
41.76
33.99
28.47
Table 4: Results of different systems on PHOENIX-20144 B@ B@I-BG 1eB--44 tokenized BLEU with n n-grams from to 4, respectively. The numbers in bracket for SLTUNET denote sBLEEU and ChrF on the test set. Best results are highlighted in bold. SLTUNET achieves competitive and even the hest performance. Note results from previous papers might not be directly comparable they might use different tokenizers and evaluation toolkits.
model dimension, and enlarging feed-forward layers, but don't get encouraging results. We argue that adding capacity leads to higher risk of model overfitting thus demanding more regularization. Based this, we increase the stochastic BPE dropout rate to 0.6 and the feed-forwant layer to 4096. This results in an improved system (14) with BLEU gain of 0.18 (11-114..
Putting all together, SLTUNET achieves substantial improvements against Baseline. SLTUNET achieves a BLEU score of 27.87, surpassing Baseline by 5.25 BLEU, a large margin (1 151). Further ablation study in Table shows that the benefits from the unified modeling and multi-task learning are still promising under the optimized setup for SLTUNET. We summarize the final configuration (i.e. system 15 in Table 2) below: d 256, CTC regulariizztio with a = 0.3, stochastic BPE dropout with dropout rate of 0.2 and stochastic rate of 0.6, Xavier initialization with gain of 0.5, sign frame augmentation (random crop and horizontal flip), and objective Equation 4. We adopt this setup for next experiments unless otherwise specified.
4, 4,d 4096, Nº
2 enc 5, Ndec 6,
SLTUNET achieves (near) the state-of-the-art on two previous benchmarks. We compare the results of SLTUNET with previous studies on PHOENIX-20147 and CSL-Daily in Table and 5, respectively. Our model produces competitive and even state-of-the-arr results on these benchmarks regardless of using the end-to-end the cascading method. In particular, SLTUNET largely out- performs the previous best system on CSL-Daily by 1.0+ B@4 and 0.8+ ROUGE. We also show sBLEU and ChrF scores the test set facilite future research. Note VL-Transfe adopts large- scale pretrained language models and includes much more model parameters than SLTUNET (Chen al., 2022). This shows the superiority of SLTUNET sample and parameter efficiency.
8

---

Published as a conference paper at ICLR 2023
Dev
Test
Task & Systems
ROUGE
B@4
ROUGE
B@]
B@2
B@3
@@4
Cascading: Sign2Glosx + Gloss2Text
SL-Transf. (Camgoz et al., 2020h)
44.18
15.94
44.81
47.09
32.49
22.61
16.24
BN-TIN-Transf-4BT (Zhou al., 2021)
48.38
19.53
48.21
50.68
36.00
26.20
19.67
VI-Transfer (Chen al., 2022)
51.35
21.88
51.43
50.33
37.44
28.08
21.46
SLTUNET (sBLEU: 23.76, ChrF: 21.09)
52.89
22.95
53.10
54.39
40.28
30.52
23.76
End-to-end Sign2Text
SI-Transf. (Camgoz et 2020h)
37.06
11.88
36.74
37.38
24.36
16.55
11.79
BN-TIN-Transh +BT (Zhou et al., 2021)
49.49
20.80
49.31
51.42
37.26
27.76
21.34
VL- TL-Transfer (Chen et al., 2022)
53.38
24.42
53.25
53.31
40.41
30.87
23.92
SLTUNET (sBLEU: 25.01. ChrF: 21.99)
53.58
23.99
54.08
54.98
41.44
31.84
25.01
Table 5: Results of different systems on CSI.-Daily. SL.TUEET obtains the best test performance.
Dev
Test
Test
Task & Systems
ROUGE B@4
ROUGE
B@I
B@2
B@3
B@4
sBLEU
ChrF
Cascading: Sign2Gloss Gloss2Texx
SL-TTrasformer
24.38
3.00
22.13
21.30
8.69
4.13
2.21
2.21
19.33
SLTUNET
26.40
3.49
23.24
21.00
8.65
4,25
2.29
2.28
18.96
End-to-end: Sign2Text
.L-Transformer
25.37
3.13
22.50
21.53
8.32
3.85
2.00
2.00
18.55
SLTUNET
27.95
3.94
24.53
23.11
10.05
5.13
2.81
2.82
20.56
Table Results of different systems on DGS3-T. SL-Trunsfommer baseline system following SL- Transf. (Camgoz et al., 2020b) with SMKD sign embeddings. SLTUNET still delivers improved translation.
The DGS Corpus presents unique challenges and SLTUNET still obtains improved perfor- mance. For this experiment, we mix SLT and MT (MuST-C En-De) samples with a ratio of 1:1. Table 6 shows that overall neural SLT models deliver poor results on DGS3-T although SLTUNET still obtains decent quality gains. Based on manual analysis, we find that models suffer greatly from hallucinations where the generation shows limited correlation with the sign video in Table 9 in the Appendix. The challenge also reflected in the poor Sign2Gloss result, where SMKD produces WER, score of 67.00 the dev set. We argue that the large number of signers and the diverse contents present serious challenges in video understanding. and the Zipfian distribution of glosses and words, with most occurring fewer than 10 times in the training data, as shown Figure in the Appendix, further increases the learning difficulty.
6 CONCLUSION AND FUTURE WORK
In this paper, we explore unified modeling for SLT with the objective to transfer knowledge across tasks and particularly benefit SLT. We present SLTUNET, a simple encoder-decoder model that supports multiple SLT-related tasks including Sign2Gloss, Gloss2Text, Sign2Text and machine translation. SLTUNET adopts shared parameters and modality-specifee parameters to achieve its best result under set of optimization techniques. We show in experiments that SLTUNET achieves (near) state-of of-te--f-thearrr performance on traditional benchmarks.
We also emphasize that using corpus such the DGS Corpus for end-to-end SLT is more meaning- ful, as it includes more signers, more glosses, richer topics and more training data, presenting unique challenges to SLT. Our initial results show that previous progress might over-estimate the success of neural models on SLT. Further research is needed to make SLT practical on broader-domain datasets.
In the future, multilingual datasets for SLT. We are also interested in studying the feasibility of designing unified models support translation between any pair of speech, sign and text.
interested in exploring large-scal scale pretrained models and devising larger and

---

Published as a conference paper at ICLR 2023
DATA LICENSING
The license of the Public DGS Corpus does not allow any computational research except if express permission is given by the University of Hamburg.
ACKNOWLEDGMENTS
We thank the reviewers for their insightful comments. This project has received funding from the Swiss National Science Foundation (project MUTAMUR; no. 176727) and the EU Horizon 2020 project EASIER (grant agreement number 101016982).
REFERENCES
Roce Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine transla- tion. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 3874-3884, Minneapolis, Minnesota, June 2019, Association for Computational Lin- guistics. doi: 10.18653/11/119-1333. URL https::/p/// /wwww a www.ww...b............. wwwrgggnnthlooggg. N1 91188.
Galina Angelova, Eleftherios Avramidis, and Sebastian Möller. Using neural machine translation methods for sign language translation. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics: Student Research Workshop, pp. 273-283--28 284, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18665/1/22222.clssvv222 https: /tap::://aclaiiiii http://lllaaattoooo.oooooooooo/. org/ 2022 pocl-arpe 21.
URL
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language, 2022.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http:/ /ttp//aavvvvv/vvv////v//e/e/aaav oroorrgaaa abs / 1409.4444 0473.
Hangbo Bao, Wenhui Wang. Li Dong. and Furu Wei. Vl-beit: Generative vision- lision-language pretrain- ing. arXiv preprint arXiv:2206.01 2022.
Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. mslam: Massively multilingual joint pre-training for speech and text, 2022.
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Multi-channel trans- formers for multi-articulatory sign language translation. In Adrien Bartoli and Andrea Fusiello (eds.), Computer Vision - ECCV 2020 Workshops, pp. 301-319, Cham, 2020a. Springer Interna- tional Publishing. ISBN 778-3-030--682235.
Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language trans- formers: Joint end-to-end sign language recognition and translation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020b.
Yong Cao, Wei Li, Xianzhi Li, Min Chen, Guangyong Chen, Long Hu, Zhengdao Li, and Kai Hwang. Explore more guidance: task-aware instruction network for sign language translation enhanced with data augmentation. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 2679-2690, Seattle, United States, July 2022. Association for Computational Linguistics. doi: www.:22...innnnnnnaaaaaaa
URL https:/ hollontoooooyyy
hittps://ww.sth wttp::/wwwww.ignnnaaa hamburg.... de/met wemmmineddee nedgs / 1ing/1 hts//iccee cense_enn
10

---

Published as a conference paper at ICLR 2023
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 5120-5130, June 2022.
Rumpeng Cui, Hu Liu, and Changshui Zhang. Recurrent convolutional neural networks for contin- uous sign language recognition by staged optimization. In 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1610-1618, 2017. doi: 10. 109/CVPR.201 2017.15.
Mathieu De Coster, Karel D' D'Oosterlinck, Marija Pizurica, Paloma Rabaey, Severine Verlinden, Micke Van Herreweghe, and Joni Dambre. Frozen pretrained transformers for neural sign lan- Proceedings of the 1st International Workshop on Automatic Transla- fion for Signed and Spoken Languages (AT4SSL), pp. 88-97, Virtual, August 2021. Associa- tion for Machine Translation in the Americas. URL httph:/t/////. ptattps://caaatiiin http///acaaatttoyy.....o.. org/2021. mt mntsummitoon commit--tteo 4ssl.11 100.
guage translation.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1863//1/N11-142233 URL https: aoclannhollo ogy- org/N1 9- 4233.
Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. MuST- C: Multilingual Speech Translation Corpus. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 2012-2017, Minncapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18663/1/N19-12022 URL https: // //w www. acl web.. org/ antoologyyyooy//oooooooooooooooh /N1 9-022.
Biao Fu, Peigen Ye, Liang Zhang, Pei Yu, Cong Hu, Yidong Chen, and Xiaodong Shi. Conslt: token-level contrastive framework for sign language translation. arXiv preprint arXiv:2204.04916, 2022.
Alex Graves, Santiago Fernández, and Faustino Gomez. Connectionist temporal classification: La- belling unsegmented sequence data with recurrent neural networks. In In Proceedings of the International Conference on Machine Learning, ICML 2006, pp. 369-376, 2006.
Thomas Hanke. Susanne König, Reiner Konrad, Gabriele Langer, Patricia Barbeito Rey-GeiBler, Dolly Blanck, Stefan Goldschmidt, Ilona Hofmann, Sung-Eun Eun Hong, Olga Jeziorski, Thimo Kley- boldt, Lutz König. Silke Matthes, Rie Nishio, Christian Rathmann, Uta Salden, Sven Wagner, and Satu Worseck. MEINE DGS. Öffentliches Korpus der Deutschen Gebärdensprache, 3. Release, 2020a. URL hwwpt:ttt/..... / d org/ 10. 255 dgs me nedgs-3. 0.
Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Jahn. Extending the Public DGS Cor- pus in size and depth. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service the the Language Commu- nity, Technological Challenges and Application Perspectives, pp. 75-82, Marseille, France, May 2020b. European Language Resources Association (ELRA). ISBN 779-10-95546-54 URL https: aalaaccaantttt /a/aaaaannthoooooyooooooo wrt.oo/g////// 2020. si signlangglnnnn
Aiming Hao, Yuecong Min, and Xilin Chen. Self mutual distillation learning for continuous sign language recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11303-11312, October 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Hirofumi Inaguma, Kevin Duh, Tatsuya Kawahara, and Shinji Watanabe. Multilingual end-to-end speech translation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp- 570-577, 2019. doi: 10. 20110///SU444991.20100
11

---

Published as a conference paper at ICLR 2023
Andrew Jaegle, Sebastian Borgeaud, Jean -Bptisst Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general archi- tecture for structured inputs & outputs. In International Conference on Learning Representations, 2022. URL https::/p.............tt / www.revieww.... net/ fo www.rmimddf...7pp.....
Ye Jia, Ron Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct ppeech-to-Speech Translation with a Sequence- -o-equuncce Model. In Proc. Inter- speech 2019, pp. 1123-1127, 2019. doi: 10.21457/nnersspecch.200 2019 1951-
Tao Jin, Zhou Zhao, Meng Zhang, and Xingshan Zeng. Prior knowledge and memory enriched transformer for sign language translation. In Findings the Association for Compulational Lin- guistics: ACI. 2022, pp. 3766-3775, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 findings-acl..277. URL https::h///.mm http://aaaaantooooy.ooooo.ooo.aa org/ 2022.
wttp:///.......innnnn.l....o acl. 227..
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's multilingual neural machine translation system: Enabling zero-shot transla- tion. Transactions of the Association for Computational Linguistics, 5:339-351, 2017. doi: 10.1 00065. URL httphttp///ccc /wwww wwwwwb.www...c.... org/ anthology/o 001000117111221
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all, 2017.
Jichao Kan, Kun Hu, Markus Hagenbuchner, Ah Chung Tsoi, Mohammed Bennamoun, and Zhiyong Wang. Sign language translation with hierarchical spatio-temporal graph neural network. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pp. 3367- 3376, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Tom Koemi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt Hitokazu to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Transla- tion, pp. 478-494, Online, November 2021. Association for Computational Linguistics. URL https: aaaalccaatto aclanthhloggyyy.coooooooooooolol org/ 2021. wmt - 1..57.
Matsushita, and Arul Menezes. To ship
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bo- jar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical ma- chine translation. In Proceedings of the 45th Anmual Meeting of the Association for Com- putational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: / //www//.. www.ww........... orgg antholooyy PPwp-P07-200 2045.
Oscar Koller, Necati Cihan Camgoz, Hermann Ney, and Richard Bowden. Weakly supervised learn- ing with multi-stream con-Istm-hmms to discover sequential parallelism in sign language videos. IEEE Transactions on Puttern Analysis and Machine Intelligence, 42(9):2306-2300, 2020. doi: 10..100//......9.9... 29777777777.7
XongXu Li, Chenchen Xu, Xin Yu, Kaihao Zhang. Benjamin Swift, Hanna Suominen, and Hong- dong Li. Tspnet: Hierarchical feature learning via temporal semantic pyramid for sign lan- guage translation. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 12034-12045 Curran As- sociates, Inc., 2020. URL StLtttss//pocceeian www.ins....... Cc /ttpee/r//////eeee fea 8cCOdee244s www.::/888ee00000000000000000000 weeeee00000004440111111111
.ddf.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Shengtong Mo, Dani Yogatama, Louis Chiil-Ph Morency, and Ruslan Salakhutdinov. Highmmt: Towards modality and task generalization for high modality representation learning. arXiv preprint 23 2022.
12

---

Published as a conference paper at ICLR 2023
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: / www. www.wawwwwbb.... org anthology/WCc4 /W0-yy/wW441111000
Amit Moryossef, Kayo Yin, Graham Neubig. and Yoav Goldberg. Data augmentation for sign
language gloss translation. In Proceedings of the 1st International Workshop on Automatic Trans- lation for Signed and Spoken Languages (AT4SSL), pp. 1-11, Virtual, August 2021. Associa- tion for Machine Translation in the Americas. URL https: //acl antho logy. org/ 2021.
i t- 4s5-ltssssl...... 1.
Mathias Müller, Zifan Jiang, Amit Moryossef, Annette Rios, and Sarah Ebling. Considera- tions for meaningful sign language machine translation based glosses. arXiv e-prints, art. arXiv:2211, 1.5444, November 2022. doi: 10.48550/arXiv.2211 1.15464.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Associa- fion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073033.1771335. URL https: //www. acweeb org/ www.hoogyooheeeee 1040.
Maja Popović. chrF:
character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 10.18653/v1W11533000 URL https:
//aclanthology. wtt.///1//300011101111
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Pupers, pp. 186-191, Belgium, Brussels, October 2018. As- sociation for Computational Linguistics. URL httpt::///// htwp///wwwwwcc...ccccoooo.. www.aallwebb...... web. org/ aotholoooto loy/ w18-6319 6319.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPF-dropout: Simple and effective sub- word regularization. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pp. 1882-1892, Online, July 2020. Association for Computational Linguis- tics. doi: 10. 11553/11/22200.... acl-aann.lmmmi.1111. URL https: htttp::/aacaanthologg or2//020. acl aol-mane aal-main..700
Junfu Pu, Wengang Zhou, Hezhen Hu, and Houqiang Li. Boosting continuous sign language recog- nition via cross modality augmentation. In ACM International Conference on Multimedia (ACM
MM), 2020.
Ben Saunders, Necati Cihan Camgoz, and Richard Bowden. Signing at Scale: Learning to Co- Articulate Signs for Large- Scale the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Sign Language Production. In Proceedings of
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/w1/P16-1116 1162. URL https: / www. aclweb.
attaaaathholgy/aaaee 1162.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464-468, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11/111-20744 URL https:t///////.m http:::s//acaaanholooooooo
orr/N18- 2074.
Yun Tang, Hongyu Gong, Ning Dong, Changhan Wang, Wei-Ning Hsu, Jiatao Gu, Alexei Baevski, Xian Li, Abdelrahman Mohamed, Michael Auli, and Juan Pino. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), pp. 1499-1499, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2022 acl- acl-long...nn. long. 105. URL https: / hac//claanii hnt wowhoolooyyy.... oog.ooooooooo...........oooo.ooo 202/220222222222 acl aallllllne longg. 05.
13

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30. pp. 5998-6008. Curran Associates, Inc., 2017. URL http: //papers.... ni ps. cc wwwpppppeee.....ccce pappp/apr///// 81 -attentteniiii on- -enn--ssallyou-eeeeeee pdf.
Zirui Wang, Zihang Dai, Barnabás P6czos, and Jaime Carbonell. Characterizing and avoiding nega- tive transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 11 1229-113022 2019.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguiisiii doi: 10.18653//1/2211 Lnaacl-main.41 URL https: //aclanthology. org/ 2021
Kayo Yin and Jesse Read. Better sign language translation with Sran-ttrasformmr. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pp. 5975-5989, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguis- tics. doi: 10. 10.18653/w11202222 c/2lno.coling 50in000minm-m- URL https: http///aaaaaatttoaoaaaaaa org/2020. moling-main. 525.
Kayo Yin, Amit Moryossef, Julie Hochgesang, Yoav Goldberg, and Malihe Alikhani. Including signed languages in natural language processing. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 7347-7360, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021 acl-loog.500 URL https: / //aclas aeclantholl ogy- org/ 2erg/20211 2021 aclll 570.
Biao Zhang. Towards efficient universal neural machine translation. Ph.D. Thesis, 2022.
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. (628-1639, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/112002...o acc-mmnn.cmmm......mmm 148. URL https: / www.
ac web. org/ antho 1 2ogy/22200 acc maacc-mmiin n. 148.
Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. Share or not? learning to schedule language-specific capacity for multilingual translation. In International Conference on Learning Representations, 2021. URL https: : /wwp//wwweww.....////// net w/fomorrrdddgi 40Do0uyCF.
Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12736-12746. PMLR, 18-24 Jul 2021. URL pttts::/proceediee
mlr mmlr.press press vv1vv111911 / http///hennnaaaam html.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language In Proceedings of the IEEE/CVF
translation with monolingual data by sign Conference Computer Vision and Pattern Recognition (CVPR), pp. 1316-1325. June 2021.
Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions Multimedia, 24:768-779, 2022. doi: N0.1009/TMM..2211330055
A APPENDIX
A. SETUP
Datasets PHOENIX-2014 is the first publicly available SLT dataset for German Sign Language (DGS) collected from weather forecasts of the German TV station PHOENIX; CSL-Daily is a Chi- nese Sign Language (CSL) dataset recording the daily life of the deaf community, covering multiple topics such as family life, medical care, school life and
14

---

Published as a conference paper at ICLR 2023
12000
2000
100000
Count
Count
5000
1000
2000
10
200
10
20
40
Frequency
Word Frequency
100000
12500
S0000
10000
Count
Count
7500
5000
2500
Gloss per Sample
# Word per Sample
Figure Distribution gloss frequency, word text translation), the number glosses per sample and number of words per sample on the DGS3-T train set.
The MuST-C corpus is extracted from TED talks with rich contents from any discipline and culture and has nearly no overlap with the above SLT datasets. This makes an adequate candidate to study transfer learning for SLT. Note English sentences differ greatly from gloss annotations in grammar, structure and wording. To narrow the gap and facilitate the transfer, we remove punctuation from all English source sentences (Moryossef et al., 2021).
We tokenize all unprocessed texts using Moses (Koehn et al., 2007) and also exclude punctuation from MuST-C German sentences for PHOENIX-2014T.
Model Settings We tie the parameters for the input embedding in the textual encoder and the input and softmax embedding in the decoder, and the CTC layer predicts over the shared vocabulary. To avoid overfitting, we apply dropout to the residual connections and feed-forward middle layer of rate 0.4 and the attention weights of rate 0.3.
We train all SLT models using Adam (3) = 0.9, B2 = 0.998) (Kingma & Ba, 2015) with Noam learning rate schedule (Vaswani al., 2017), label smoothing of 0. and warmup step of 4K. We employ Xavier initialization to initialize model parameters with gain of 1.0. We average the best 10 checkpoints based on the dev set result on Sign2Text for the final evaluation. We use beam search for decoding for all tasks and set the beam size to 8. We tune the length penalty on the dev set.
Evaluation We adopt SacreBLEU (Post, 2018) to report detokenized BLEU (sBLEU) and ChrF. ttrefs//ss.epptooo.. Ba..ok///...///tttt. The signatures for sBLEU and ChrF are BLEU+c.mixed+ #refs. +sspueeFaalsett False+ 1.4.2, respectively. Note, on PHOENIX-200 2014T and CSL--Daily, the value of sBLEU equals to B @4. This is because texts in PHOENIX-200 20144T are well tokenized with punctuation removed while CSL-Daily uses character-level evaluation. both cases, tokenization becomes unimportant.
and
A.2 DETAILS ON THE DGS3-T TRANSLATION PROTOCOL.
We used release version of the Public DGS Corpus (Hanke et al., 2020b). We excluded videos because they have incorrect framerate of 25 instead of 50. We then randomly assign documents to either the training, development or test split. The desired number of documents in the development and test set is 10. No other preprocessing was performed create the data split.
15

---

Published as a conference paper at ICLR 2023
train # signers
validation
test
signers
unknown
signers
unknown
328
20
20
2
Table 7: Distribution of signers (individuals) in DGS3-T. T. # unknown number of signers that do not appear in the training data.
SLTUNET Loss
SignZText
Sign2Text CTC Loes
Sign2Giloss MLE Loss
10
Gloss2Test & MT MLE Loss
Loss
2
20
40
60
80
100
120
Training Steps (x400)
Figure 3: Learning curve of different losses in SLTUNET as a function of training steps on PHOENIX-2011
Identity of signers The identity of signers has great impact models that extract features di- rectly from videos. For tasks involving only glosses and text we assume that the identity of the signer less important. The overlap of individual signers between training and testing data also matters. We added additional statistics about signers in Table 7. Since most individuals appear in several recording sessions, most signers our validation and test set are "known". All validation signers also appear in the training set, 18 out of 20 test signers also appear in the training set. Generalization signers not seen the training set known to be more challenging, but the DGS Corpus already has large number of signers overall (over 300 individuals), improving generalization.
Figure shows distributions of glosses and German words in the data set.
A.3 ADDITIONAL. ANALYSIS
The convergence of different tasks in SLTUNET follows similar trend. Apart from positive knowledge transfer, sharing parameters across tasks might incur inter-task interference hurting the convergence of some tasks. Figure 3 shows the learning curve of different tasks in SLTUNET, which follows similar trend without obvious convergence disagreement across tasks. This also supports the unified modeling of different SLT-related tasks.
Aggressive mopelity-specifie modeling hurts SLT performance. Table 8 shows the results of SLTUNET with either separate encoders separate decoders. Modeling different modalities with separate modules leads to worse translation results, resonating with our findings in Table 2. Besides, sharing parameters over gloss and text on the decoder side facilitates knowledge transfer, while the transfer between sign video and gloss/text on the encoder side is harder.
A.4 CASE STUDY FOR PHOENIX-20 201 4T CSL-DAILY AND DGS3-T
16

---

Published as a conference paper at ICLR 2023
Dev
Test
Test
Task & Systems
ROUGE
B@4
ROUGE
B @@!
B@2
B @
B@4
sBLEU
ChrF
Cascading: Sign2Gloss + GelossTTxxt
SLTUNET
49.61
25.36
49.98
50.42
39.24
31.41
26.00
26.00
51.96
separate encoder
50.12
25.77
49.17
49.26
38.48
30.88
25.59
25.59
50.89
separate decoder
48.89
24.68
48.46
48.32
37.49
29.94
24.77
24.77
50.26
End-to-end: Sign2Text
SLTUNET
52.23
27.87
52.11
52.92
41.76
33.99
28.47
28.47
53.78
separate encoder
51.62
27.64
51.73
51.95
40.86
33.08
27.62
27.62
53.50
separate decoder
51.52
27.19
50.95
51.03
40.38
32.76
27.33
27.33
52.75
Table 8: Further ablation results of shared and modality-spentt speciftcc modeling for SLTUNET PHOENIX- 2014T. Experiments are based system Table 2. seporate encoder: different encoders for sign video gloss/tex;; separate decoder: different decoders for gloss and text.
Gold Gloss:
MEISTENSI TAUB-GERS GEHÖRLOSI BESUCHENI WASI WÜNSCHENI ZIEL4 WASI REEPERBAHNI TYPISCH!
Gold Text:
Die meisten Gehörlosen, die mich besuchen, wollen typischerweise auf die Reeperbahn. (Most deaf people who visit rypically want to go the Reeperbahn.)
SLTUNET:
Meistens haben wir Gehörlose besucht und uns wünschen, dass ein Ziel gibt, eine andere Familie zu bekommen. (Mostly we visited deaf people and wish that there a goal to get another family.)
Gold Gloss:
ODERI SLIST1:20f2 SALPHAI-S SALPHAI:M RUND-LANGA BEKANNTI SINDEXI Die St. Michaelis-Kinre Kirche ist auch bekannt für Hamburg. (St. Michaelis Church is also famous for Hamburg.)
Gold Text:
SLTUNET:
Zweitens gibt es den Smartturm, aber das ist nicht berühmt. (Secondly. there is the smart lower,
not famous.)
Gold Gloss:
TURMI SEHENI SPMS TURMI SEHEN-AUF3 REEPERBAHNI SEHR-GUTI BEKANNTI
Gold Text:
Der Fernsehturm und die Reeperbahn, die sind doch bekannt. (The TV tower and the Reeperbahn are well known.)
SLTUNET: Wenn dic Hörenden in Amerika mehr sind, dann muss man die Hörenden beide beide beide beiden. (If the hearing America are more, then you have have the hearing both both both both.)
Gold Gloss:
MORGEN3 FISCHI MARKT4 BEKANNTI SINDEX2
Gold Text:
Morgens geht man zum Fischmarkt, der ist bekannt. (In the morning you go to the fish market, it's well known.) Ja, das ist bekannt. (Yes, that is known.)
SLTUNET:
Table 9: Case study for SLTUNET DGS3-T. Examples from the test The model only translates
tiny part of input and suffers from hallucinations greatly. Sentences in brackets are our English translations.
17

---

Published as a conference paper at ICLR 2023
Examples from SIIOENIX-201447
Gold Gloss:
WOCHENENDE IX MEHR KALT
Gold Text:
und zum wochenende wird es dann sogar wieder ein bisschen kälter (and by the weekend it will even be bit colder again)
SLTUNET:
und am wochenende wird es dann auch wieder kälter (and on the weekend it will be colder again)
Gold Gloss:
DONNERSTAG NORDWEST REGEN REGION SONNE WOLKE WECHSELHAFT DANN FREITAG AEHNLICH WETTER am donnerstag regen in der nordhälfte in der südhälfte mal sonne mal wolken ähnliches wetter dann auch am freitag (on thursday rain in the northern half in the southern half sometimes sumny sometimes cloudy similar weather then also on friday) am donnerstag in küstennähe regen sonst mal sonne mal wolken wechsel dann am fre- itag ähnliches wetter (on thursday ruin near the coast otherwise sometimes sun sometimes clouds alternately then similar weather on friday)
Gold Text:
SLTUNET:
Gold Gloss:
SONNTAG NAECHSTE NORDWEST WOLKE SONNE WOLKE GEWITTER REGEN DABEI
Gold Text:
sonntag im nordwesten eine mischung aus sonne und wolken mit einigen teil ge- wittrigen schauern (on sunday the northwest a mixture of sun and clouds with some partly thundery showers)
SLTUNET:
am sonntag im norden und westen mal sonne mal wolken mit einzelnen gewittern (on sun- day in the north and west sometimes sunny sometimes cloudy with some thunderstorms)
Gold Gloss:
MORGEN DANN HERBST MISCHUNG HOCH NEBEL WOLKE SONNE
Gold Text:
auch morgen erwartet uns eine ruhige herbstmischung calm autumn mix of high fog clouds and sun awaits us tomerrow as well) morgen erwartet uns eine meist trübe mischung aus nebel wolken und sonne (tomorrow we can expect mostly dull mix of fog clouds and sunshine)
hochnebel wolken und sonne (a
SLTUNET:
Examples from CSL-Daaily
Gold Gloss:
#/ 11y / ft 2/ of (A] is
Gold Text:
#/ f & Bi AA 666
(When did you meet Zhang?)
SLTUNET:
# 1R 8
(When did you meet Zhang?)
Gold Gloss:
4 / to
Gold Text:
(1 want to eat noodles today.) (I want to eat noodles today.)
SLTUNET:
$ $
Gold Gloss: / / // i # $
Gold Text:
(The food very salty today, and want to drink.) (The food is very salty today and want to drink.)
SLTUNET:
Gold Gloss: // $
Gold Text:
7 i
$ #
(1 don't 't know that boy.) (That boy my son.)
SLTUNET:
Table 10: Case study for SLTUNET on CSL-Daily and PHOENIX-2014T. Examples are from the test set. Sentences in bracket are our English translations. While SLTUNET achieves better translations on these two benchmarks, it still suffers from difficulties with sign video understanding and delivers inadequate outputs.
18


Reviews:
[{"review_id": "n2OLgJ7g5c", "reviewer": "Reviewer_LzzH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on how to solve four strongly related tasks in sign language translation (SLT) with a unified framework. They propose SLTUNET, a simple unified neural model designed to support multiple SLT-related tasks jointly. By discriminating between inputs from different modalities, different tasks can be trained on the same network and thus narrowing the gap between modalities. In addition, the paper also introduces machine translation tasks on this basis to solve the problem of small scale of sign language translation datasets. The model is tested on two popular SLT datasets, and it can achieve good experimental results without using pre-trained large models, proving its effectiveness.", "summary_of_the_review": "The paper has good innovation, has done some original work, and the proposed method is concise and has good reproducibility. But the clarity of the paper is not high, especially regarding the index part. This is related to the specific performance and the effectiveness of the method and must be explained in detail. Until then, I was skeptical about the paper. If you can well answer the above questions, I am apt to accept this paper.", "strength_and_weaknesses": "Strength：\n1. The paper combines the related tasks of sign language translation in a clever way. It is only necessary to perform different processing on different modalities in the input layer, and then a unified network can be used for subsequent training. The method is simple and effective and improves performance without significantly increasing model parameters.\n2. This paper has done a lot of ablation experiments, and the experimental details shown in Table 2 can enable readers to understand the improvement brought by each method and enhance the reproducibility of the method.\n\nWeakness:\n1. There is a little description of the method in this paper. There is only an illustration chart, and no details of the model are shown. It is not stated how the different modal inputs are fused at each iteration of the model.\n2. The experimental part of the paper does not highlight its method. The main innovation of this paper is to use a unified framework for related tasks of sign language translation and expect that multi-task training can improve the performance of the model. However, there is only a small amount of content about multi-task training in the experimental part of the paper.\n3. The performance metrics in the paper are confusing. The article discusses the common performance metrics of machine translation and uses two different metrics, tokenized Bleu (B@4) and detokenized Bleu (sBLEU). However, the values of these two indicators in the experiment of the paper are exactly the same, which is Unreasonable unless a unified Tokenizer is used, namely tokenize13a. This needs to be explained in more detail in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fZMdZzUhqo", "reviewer": "Reviewer_JikT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles a challenging problem of translating sign language to natural language. The problem's challenging nature arises from the data scarcity and the modality gap between video and text. The authors approach the problem from the perspective of learning a joint latent space for video and text, guided by multiple loss functions including CTC and MLE for both gloss and text. These approaches are verified by a thorough examination of ablation studies of model capacity and regularization over challenging datasets PHOENIX-2014T and CSL-Daily. The authors also propose an end-to-end translation protocol for SLT by using an even more challenging dataset of the Public DGS corpus that covers broader domains and more open vocabularies. ", "summary_of_the_review": "The detailed study of the model design choices are appreciated and could be useful for the SLT community. Although, some empirical comparisons on the new protocol of the DGS3-T dataset are missing which are crucial for understanding the complete utility of the proposed model.\n", "strength_and_weaknesses": "Strengths\n- The proposed model is clearly defined and the ablation studies are carefully structured in Table 2. The detailed discussion of these results is also appreciated.\n\nWeaknesses\n- Design the protocol for Public DGS for an end-to-end SLT setup: It seems that the protocol involved splitting the dataset into train, test, and dev sets. Was there more to that process? Furthermore, it would also be useful to include the details of how these splits were conducted. Do speakers overlap among the splits? If yes, does the speaker overlap matter? How does the speaker's identity affect sign language translation? What is the distribution of text tokens and gloss vocabulary across the splits?\n- The back translation losses are not useful for SLT (i.e. impact of Text2Gloss in Section 5.2.2), but Zhou et. al. 2021 showed that it was quite useful. That seems contradictory and some more discussion on this would be useful for the readers.\n- The choice of visual backbone could impact the final performance. Why was the SMKD model chosen? Chen et. al. 2022 also uses a visual backbone model which is S3D (i.e. different from the one used in this paper). Chen et. al. 2022 also use CTC losses for training the model. As their model have some level of similarity to the proposed model, it would be useful to explicitly talk about the differences.\n    - Especially, as Chen et. al. 2022 is also similar to the proposed model in terms of performance, it would have been insightful to see its performance on the DGS3-T baseline in Table 6.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZhJ47FIQch", "reviewer": "Reviewer_x36u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a unified sign language translation framework that translates sign videos to natural texts. To better learn visual and text representations, the authors propose to use multi-task objectives to borrow knowledge from external datasets, like machine translation datasets. It uses a shared encoder and a shared decoder to handle gloss-to-text translation, text-to-text translation, sign-to-gloss translation, and sign-to-text translation. In addition to multi-task objectives, the authors also conduct a detailed ablation study to see how different experiment settings contribute to the final performance, like BPE dropout. Experiments show that the final architecture achieves comparable results with state-of-the-art baselines with large pre-trained networks as initialization. ", "summary_of_the_review": "The papers show promising results with only sign-to-text data and limited external data. The paper is well-written and easy to follow. The idea is simple and intuitive. In addition, the detailed ablation study shows how the specific settings contribute to the final performance. ", "strength_and_weaknesses": "Strength:\n\n1. The performance is promising. The final model does not rely on any pre-trained networks but achieves comparable performance with models with pre-trained networks as initialization. \n\n2. The multi-task objective is intuitive. Generally speaking, the size of datasets for sign language-to-text translation is limited compared with text-to-text translation datasets. Borrowing knowledge from other tasks and external task is a practical solution. \n\n3. The authors also conduct a comprehensive ablation study to see the effects of model settings to the final performance. \n\n\nWeaknesses:\n\n1. Table 2 shows that the multi-task objective improves results from 24 to 25.23, but the settings of other architecture details increase performance from  25.23 to 27.87.  Although experiment highlights are state-of-the-art performance, the contribution of the proposed multi-task objective is limited.  \n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EIIqncX6I0I", "reviewer": "Reviewer_FxDU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article proposed a unified model for sign language translation, in which multiple tasks are simulated jointly for enhancing the performance of sign language translation. ", "summary_of_the_review": "Please see review in the previous sections.", "strength_and_weaknesses": "Strengths:\nThe proposed approach allows multiple tasks (e.g., such as sign-to-gloss, gloss-to-text, and sign-to-text translation) related to sign language translation to be simulated jointly for enhancing the performance of sign language translation. \n\nWeaknesses:\n\n1. The existing experiments were insufficient to verify how multiple tasks interact to improve the performance of sign language translation.\n\n2. In End-to-End: Sign2Text in Table 4, the proposed approach just was comparable to the baseline VL-Transfer model in terms of ROUGE and B@4 on the Test set of PHOENIX-2014T.\n\n3. One of the claimed contributions is that the set of optimization technologies for SLTUNET was proposed to improve the trade-off between model capacity and regularization. It was confusing.\n\nQuestions:\n\n1. When there is an imbalance between video and gloss input, what is the result?\n\n2. Author claimed that the DGS Corpus includes larger\nscale, richer topics, and more significant challenges than existing datasets. How to evaluate them?\n\n3. When the proposed approach was applied to multiple tasks, how about the learning curving of CTC and MLE losses?\n\n4. If there replaced the shared encoder or decoder with a separate encoder or decoder, what about the performance?\n\n5. In Figure 1 (1), What do the different colors of these modules denote? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster