Scaling Sign Language Translation
Authors: Biao Zhang
Garrett Tanzer
Orhan Firat
Venue: NeurIPS 2024 poster (2024)
Keywords: Sign Language Translation
Model and Data Scaling
American Sign Language
Large-scale Pretraining
Video-to-text generation
Primary area: natural_language_processing
TL;DR: We explore data, model and language scaling for sign language translation and achieve new state-of-the-art performance on several open-domain benchmarks.

Abstract: Sign language translation (SLT) addresses the problem of translating information from a sign language in video to a spoken language in text. Existing studies, while showing progress, are often limited to narrow domains and/or few sign languages and struggle with open-domain tasks. In this paper, we push forward the frontier of SLT by scaling pretraining data, model size, and number of translation directions. We perform large-scale SLT pretraining on different data including 1) noisy multilingual Youtube SLT data,
2) parallel text corpora, and 3) SLT data augmented by translating video captions to other languages with off-the-shelf machine translation models. We unify different pretraining tasks with task-specific prompts under the encoder-decoder architecture, and initialize the SLT model with pretrained (m/By)T5 models across model sizes. SLT pretraining results on How2Sign and FLEURS-ASL\#0 (ASL to 42 spoken languages) demonstrate the significance of data/model scaling and cross-lingual cross-modal transfer, as well as the feasibility of zero-shot SLT. We finetune the pretrained SLT models on 5 downstream open-domain SLT benchmarks covering 5 sign languages. Experiments show substantial quality improvements over the vanilla baselines, surpassing the previous state-of-the-art (SOTA) by wide margins.

Scaling Sign Language Translation
Biao Zhang
Garrett Tanzer
Orhan Firat
Google DeepMind
{biaojiaxing, gtanzer, orhanf Fagoogle.com
Abstract
Sign language translation (SLT) addresses the problem of translating information from sign language in video spoken language in text. Existing studies, while showing progress, are often limited to narrow domains and/or few sign languages and struggle with open-domain tasks. In this paper, push forward the frontier of SLT by scaling pretraining data, model size, and number of translation directions. We perform lcrge-scale SLT pretraining on different data including 1) noisy multi- lingual YouTube SLT data, 2) parallel text corpora, and 3) SLT data augmented by translating video captions other languages with off- off-the-shelf machine transla- tion models. We unify different pretraining tasks with task-specific prompts under the encoder-decoder architecture, and initialize the SLT model with pretrained (m/By)T5 models across model sizes. SLT pretraining results on How2Sign and FLEURS-ASL. (ASL to spoken languages) demonstrate the significance of data/model scaling and cross-lingual cross-modal transfer, well as the feasi- bility of zero-shot SLT. We finetune the pretrained SLT models on downstream open-domain SLT benchmarks covering 5 sign languages. Experiments show sub- stantial quality improvements over the vanilla baselines, surpassing the previous state-of-the of-the-rr-ttttt (SOTA) by wide margins.
WMT23 LIS-CH
Elementary23 GSS GSS
25
5.69
5.2
161
WMT23 LSF-CH
Hoiv2Sign
18.77
2.0
7.5
18.17
WMT23 DSGS (SRF)
WMT23 DSGS (5S)
Model
Previous SOTA
Figure 1: BLEU scores different benchmarks: our model sets new SOTA results across benchmarks and sign languages. Note didn't show BLEURT because not all previous studies report BLEURT.
Introduction
Scalable neural networks trained on large amount of unlabeled and/or weakly-labeled data from multiple modalities and multiple tasks have resulted in performance significantly exceeding that of Sign language translation (SLT),
single-task models trained on particular domains 330 33 [8
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SLT Encoder
SLT Decoder
Prompt
Targel Oulput
Different
SLT
<dbb>
sign long
lang
clip fromes
clip text
Alignment
<agn>
sign lang
1gt lang
clip text
clip frames
clip text with timestamps
Machine Translation
<mb>
lang
1gt long
target
Augmented
<aug>
sign long
1g1 long
clip frames
text
(a) Encoder-decoder based SLT model and different SLT pretraining tasks. We use red, green, and blue colors to indicate the input prompt, sign frames, and target output, respectively. "sign lang": sign language name: "src lang/tgt lang": source/target spoken language name; "<u>": task-specific control tokens; "source/target text": source/target text for MT; "clip frames (clip text)": concatenation of sign frames (caption texts) corresponding to vidco clip; "translated clip text": augmented data by off-the- shelf MT models; "clip text with timestamps": concatenation of caption texts and their start and end timestamps.
a clip of N seconds
Stact time (5): so
Clip Tost: Capp, Capapares, Cap.]
Cliy Text Time Stanye S., Cup,, Suas Eist, Cup Rj, Capy]
moderaee sarthquake Mostane
(b) Clip overview. Top: a sequence of skeletons for a sign language video where the used keypoints are annotated in red; Bottom: We pretrain SLT models randomly sampled clips of N seconds from the video. Each segment in the plot represents caption, and (Cap,...., Clpp,...,,CCpp) (i.e., green segments) denotes captions fully covered by the clip. "S./ E:": the start/end time stamp for caption Cap,-
Illustration of model architecture and pretraining task for SLT. We perform large-scale pretraining and adopt multi-task learning at clip level (multiple captions) to better leverage the supervised knowledge.
Figure 2:
as a video-to-text translation task features significant cross-modalits mossattodaltty challenges in video under- standing and text generation. While extra forms of supervision such as glosses have been helpful in bridging the modality gap 4 they are nonstandarizzed/heoopeee systems available only for small datasets [12]. Researchers have instead turned to more scalable approaches such as adapting pretrained vision and text models 7 37 50) and jointly modeling with machine translation [MT, (88. Despite encouraging progress, these studies were performed at small scale with success on nar- rowed domains and on few sign languages. In open-domain SLT settings, unfortunately, they have shown limited effectiveness (28).
In this paper, we aim to improve open-domain SLT for multiple sign languages by means of large- scale SLT pretraining with more data, larger models and more languages. Inspired by the finding that jointly training SLT models with MT data enables positive knowledge transfer to SLT [58), we explore the following pretraining tasks and data: web-crawled muirilingual SLT, multilingual MT, and augmented SL.T. Although high-quality SLT training data are scarce, weakly-labeled SLT data covering diverse topics and signers are readily available from platforms like You'ubee. Prior studies have demonstrated the feasibility of collecting massive YouTube SLT data and its effectiveness on improving SLT 46 44 43), and we follow this effort in multilingual setup. Different from SLT, text-based MT datasets are massive and resource-rich across hundreds of spoken languages 3 [0). We explore subset of MADLAD-400 [ including up to 41 spoken languages for the pretraining. In addition, we construct synthetic multiway SLT data by translating video captions with an off-the- shelf MT model, which allows us strengthen direct SLT across investigate different ways of mixing these data to exploit weakly supervised SLT knowledge as well as cross-task cross- coos-mt transfer at scale.
translation directions. We
While spoken language be conveyed through either text speech, this study focuses text.

---

44 43 with As in Figure 2 we extend the unified encoder-decoder SLT framework from 46 42
extra tasks and modalities similar to [58], across different pretrained model families (T5, mT5 and ByT5) and different model sizes. We distinguish different tasks by carefully designed input prompts that contain task-specific control tokens. This affords us high flexibility in choosing what tasks and languages to incorporate into the pretraining, easing ablations and the scaling. We then finetune the pretrained SLT models on downstream SLT benchmarks to refine the learned SLT knowledge.
We evaluate the effect of scaling on 6 open-domain SLT benchmarks across 5 sign languages. FLEURS-ASL#0 42], built on FLORES-200 [TO,, gives us a testbed to analyze multiway Ameri- can Sign Language (ASL)-to-X SLT (we examine English and 41 other target languages), while the other benchmarks are for a single language pair. While pretraining results show the acquired general SLT capability, we also report finetuning results following [46]. Our main findings are below:
Adding more pretraining data, either machine translation or sign language translation data, is a promising way to improve SIT,, yielding quality gains of varying degrees.
Zero-shot ASL-to-X X translation for language pairs not seen during pretraining is achievable by jointly training on ASL-to-En SLT data and En-to-X MT data.
Augmenting SLT data by translating target captions to other languages with off-the-shelf
MT models substantially improves the translation.
Using larger models not always helpful: ByT5 Base (582M) often outperforms XL (3.74B), but model scaling does benefit SLT when modeling capacity becomes a bottle- neck (c.g., when more languages and data are used).
Learned metries (e.g., BLEURT) show higher correlation between pretrained and finetuned SLT scores than classical metrics (e.g., BLEU or ChrF).
Putting everything together, our model achieves new state-of-the-art results across the benchmarks demonstrating the significance of scaling SLT.
as shown in Figure [
2 Sign Language Translation
2.1 Modeling
43 which build upon We build on line of work using T5 model families for SLT [46 42 44
carlier SLT work [4] 61 588 using the encoder-decoder architecture [41] 47]. Figure 2a shows the overall structure. The encoder takes as input the concatenation of prompt instructing the task and a sequence of sign language video frames; the decoder predicts the text output in a target spoken language token at time. We adopt the family of pretrained (m/By)T5 models [35 53) as the backbone and adapt them to SLT via large-scale SLT pretraining followed by downstream finetuning, i.e. (m/By)T5 initialization - SLT pretraining - SLT finetuning.
We rely on web-crawled YouTube SLT data for SLT pretraining, which provide high coverage on domains and signers albeit at lower quality. Although recent debates value data quality over data quantity in pretraining 21 24 29], we argue that they were established on the availability of massive high-quality training data, which doesn't hold for SLT yet. We expect that the pretraining could capture the (weakly) supervised SLT knowledge from the crawled data as in previous studies [46).
As shown in Figure 2b we adopt the clip-level training following 42 that randomly samples a clip of N seconds from the sign video and then predicts various types of in-clip information (such as caption texts and their start and end timestamps) based on the frames of the entire clip. Detailed tasks are listed Figure 2al which are all formulated as sequence-to-squuecce tasks. They are distinguished by prompts with different control tokens and are trained with the standard maximum likelihood objective. For the baseline, we consider the following two tasks: SLT and alignment, and train it by mixing these two tasks with pre-specified mix ratio.
SLT
This is the core task that directly models the translation from clip frames to the clip text in a target language. It is indispensable for the model to acquire the translation capability.
Alignment It is auxiliary task for SLT, learning to align the input clip with its captions. We train the model to infer the start and end time stamp for each in-clip caption. Apart from regularization, this task could improve the model's understanding of sign language [4].
3

---

2.2 Scaling Model Size, Number of Languages and Pretraining Data Size
Model Scaling Scaling model size increases modeling capacity, which has been widely proven effective in improving the task performance [30, 18 19]. We study whether and how increasing model size affects the SLT performance and compare (By/m)T5 models for SLT at different scales.
Language Scaling While most SLT works focus on few sign and spoken languages, we expand our study massive languages, covering up 80 sign/spoken languages during pretraining, and sign language and 42 spoken languages at evaluation. We are interested in whether a single SLT model could support multiple sign/spoken languages with non-triveee non-trivial performance, and whether knowledge transfer could improve SLT on low-resource languages [57] 44].
Data Scaling Data scareity is the main bottleneck hindering the development of SLT. To address this issue, we investigate the following three types of data for the pretraining:
SLT We crawl multilingual YouTube SLT data following the recipe [44 except that we didn't per- form human annotation and filtering. This allows us to significantly scale up the SLT data by 3~6 times, reaching ~6,600 hours in total, albeit at much lower quality.
Machine Translation Unlike SLT, MT is a text-to-text translation task with rich parallel resources, particularly for high-resource languages (2). We explore adding multilingual MT data into the pretraining and mark this task with control token "<mt>" [58]
Augmented SLT SLT data are often one-to-one translation data, where each sign language only has translation in one spoken language. This makes the translation of a sign language other spoken languages difficult. We thus augment SLT data to one-to-many by translating the target text to other spoken languages via off the-shef MT models. As in Figure 2a we use "<aug>" to separate genuine SLT data from the augmented one [6].
3 Setup
MT Pretraining Data We use the paraliel sentence-level portion of MADLAD-400 (23 as the MT pretraining data. We extract a subset of MADLAD-400 for experiments, including 41 languages (apart from English (En)) covering diverse language families and scripts, and explore the impact of En Exx and Xx- MT data on SLT in experiments. We create two settings for the pretraining:
MT-Small: A high/mediuum-resoure subset including 11 languages es, de, fr, it, cs, pl, ru, zh, ar, ja, hi.
MT-Larre: This set includes all 41 languages. Apart from MT-Small, it has nl, pt, sv, hu, da, fi, el, sk, no, bg, 1t, lv, sl, et, ko, hr, is, sr, tr, vi, id, he, th, ms, uk, ca, ta, fil, me, cy.
Table 7 shows the statistics for each language. Unless otherwise specified, we balance the MT data distribution over languages during training by temperature sampling with rate of 5 (2).
SLT Pretraining Data We experiment with noisy captioned sign language vidcos from YouTube. This is the full set of videos pre-manual filtering in [44). Estimated statistics for each sign language are summarized in Table 7 We also have two settings for this data:
YT-ASL: 2,,000 hours of noisy captioned ASL videos; superset of YouTube-ASL 46
(modulo video churn) and the same dataset used by 43
YT-Full: 666600 hours of noisy captioned multilingual sign language videos; a superset [44]
During training, we mix the SLT data for all languages in proportion their duration. We further augment these data with other spoken languages via MADLAD-MT-3B 23 For ASL SLT data, translate the English captions 41 spoken languages listed in MTT-Large, which makes YT-ASL 43-way multilingual SLT,, namely Aug- YT-ASL; for other SLT data, we translate the target text into English, resulting in 3-way multilingual S We refer to the augmented SLT data for all sign
Note translations were performed per caption, which may lack coherence when compiled into document.

---

Task
Sign Lang
Target Lang
#Train
#DDvv
#Test
How2Sign
ASL
En
183,097
10,277
13,890
Elementary23
GSS
EI
It 35,970
512
512
LIS-CH
1,901
100
250
WMT23
LSF-CH
Fr 5,560
100
250
420
250/246
DSGS
De
310,840
(WMT22)
(SS/SRF split)
FLEURS-ASLOO
ASL
200 Flores Langs
353
Table I: Summary of downstream SLT benchmarks. "#Train/Deww@Test the number of examples in the train, dev and test split. Note the sign language video and the target these benchmarks often pre-segmented and aligned sentence level. "DGS/ASL//SS" German/ Saaaa//AmmriicannGree Sign Language; "Fn/De/Fr/itt: Enngtish/Grmman/Feench//
"LIS-CII": Italian Sign Language of Switzerland; "LSF-CH"" French Sign Language of Switzerland; "DSGS": Swiss German Sign Language.
languages Aug- YT-Full. Similar MT-Small and MT-LLarge, reorganize the augmented data to Aug- -Ta-V-------maalAuggg YarFA-l----T-l-la-aaal and Aug- Sog.VT--S--Laarg/uggg//T... Awt.://ulu----Fl--a......
SLT Pretraining Mixture We ablate across several SLT pretraining mixtures.
Baseline: Caption alignment and SLT tasks. We use the task weights from 42 including 4% for alignment.
Baseline + MT: We mix MT data into Baseline with sampling probability of part-
Baseline Augmented SLT: We replace the Baseline SLT data with the augmented SLT data and uniformly sample the target language for each example at each step.
Baseline + MT + Augmented SLT: Baseline MT but with augmented target languages, as above.
Downstream Benchmarks, Evaluation and Model Setting We thoroughly evaluate the transla- tion performance on range of open-domain SLT benchmarks, including How2Sign [44, Elemen- WMT23 [28] and FLEURS-ASLIS (signer id #0) [42]. Detailed information for each benchmark given Table [ Overall, the evaluation covers 5 source sign languages and 42 target
tary23 48
spoken languages
We report translation results for Pretraining and Finetuning. During inference, we beam search with a beam size of 5. We evaluate translation with detokenized BLEU (31] and ChrFi (32), as well neural metric, BLEURT (4). We BLEURT as the main metric (17). We initialize our SLT model with three T5 model families: T5 35 mT5 [52] and ByT5 [53]. at three different sizes:
Base, Large and XL. We optimize models with Adafactor (39), and set the maximum input,
landmark input, and text output length to 512. More setup details are given in Appendix A.
Experiments
4.1 SLT Pretraining Results
Model scaling doesn't improve SLT consistently: Base often outperforms Large/XI.. Table also shows that scaling up model size rarely results in consistent quality improvements. Different from findings on text-only tasks [35] [33, Base surpasses Large and XL in most cases, where Large often converges the slowest and performs the worst. Model scaling alone doesn't significantly reduce the video-text modality gap, although better optimization and checkpoint selection could help. XL performs relatively comparable to Base. When MT data is mixed in and modeling capacity becomes the bottleneck, the value of model scaling by XL emerges as shown in Figure and Table 5
Whiile restricted specific domains like "weather forecasts". the scope of topics in Elementary23 remains somewhat focused.
"We acknowledge that there are other SLT benchmarks available in academia. We didn't 't include them in experiments due to their licensing restrictions and/or domain limitations.
5

---

How2Sign
FLEURS-ASH (En)
SLT Data
Model
Base
Large
XL
Base
Large
XL
TS
29.54
27.95
22.96
32.8
4.18
32.41
YT-ASL
mT5
34.94
8,46
23.7
35.59
43,53
23.09
ByT5
30.36
23.51
29.2
44.84
28.47
41.65
T5
31.64
25,45
8.57
42.86
37.55
30.02
YT-Fuull
mT5
31.46
19.37
24.46
38.03
24.56
33.16
ByT5
37.13
22.61
29.59
52.48
43.01
52.71
Table 2: Pretraining performance (BLEURT 1) for different sized (By/m)TS models when pretrained on YT- ASL and YT-Full. Results are reported on the test set of How2Sign and FLEURS-ASY (-En, i.c. English as the target). Best results for each model family are highlighted in bold.
Baseinc
zero-shop Pero-shell test DLEURT on FLEURS
Baseline
MTX
25
50.0
20
67.5
17.4
on
45.0
155
42.5
1.0
40.0
text
37.5
De
C4
Ru
1I
Ang
E6
DH
C6
M
Ru
2h
A1
Avg
Added MT Languages
Added MT Languages
(a) BLEURT scores for FLEURS-ASL#O (-En). (b) Zero-shot BLEURT for FLEURS-ASL#O (-X).
Pretraining performance for Baseline + MT when varying MT languages. We show BLEURT results on FLEURS-ASLHO)
Figure 3:
set Pmnt 0.5. Note MT languages are added separately instead of jointly. Results are for ByT5 Base. "X-Enn: MT data for translation into English; "X+>En": MT data for both translation directions; "Avg": average performance over languages. MT languages are arranged in descending order from left to right based their training data quantity.
Backhone affects SLT substantially; By generally performs the best. While several previous studies selected T5
46 28 or mT5 [44 as the SLT backbone, we observe in Table 2 that By based SLT outperforms its T5 counterpart in most settings, confirming the results of 43 at scale. Given that larger models do not consistently perform better, it seems less likely that ByT5's ' superi- ority comes from its encoder-heavy parameter allocation, and more likely that is due its spelling capabilities and reduced input length gap between byte text sequences and video frame sequences. Unless otherwise stated, use ByT5 Base for the following experiments.
Scaling SLT data generally improves quality significantly. Adding more SLT data, i.c. from YT-ASL to YT-Full, largely improves the translation quality in most settings. For ByT5-based SLT particularly, the gain reaches ~7 BLEURT on How2Sign and ~11 BLEURT on FLEURS- ASL#O (En) for Base and XL, respectively. We conjecture that adding more (multilingual) SLT data helps reduce the modality gap (especially with skeletons, which lack pretrained representations) and enable tross-lingual knowledge transfer 2 57 55 44.
Mixing MT and SLT data yields positive knowledge transfer SLT. We next explore whether and how the addition of MT data benefits SLT, starting with YT-ASI. and bilingual MT data with Pmne = 0.5. Figures Ba 5 show that adding bilingual translation data improves SLT performance generally, confirming the findings of SLTUNet (58) -tat jointly training with MT enables positive knowledge transfer-at scale. The quality gains vary greatly across languages, which show little correlation with language family training data scale. For example, adding large amount of Fr-+En data (~243M sentence pairs) helps little (or even hurts) on PLEURS-ASIL (En), while adding small amount of Ja- data (~~5M sentence pairs) gives gain of at least BLEURT on How2Sign and FLEURS-ASLHO (En).
Translation direction of MT data affects transfer to SLT. There are three ways to leverage MT data for SLT: 1) X-+En, 2) En-xX and 3) both. We compare 1) and 3) in Figures Ba 55 ASL- to-En SLT. The translation direction of MT data influences SLT performance greatly and varies

---

YT-ASL
YT-FULL
YT-ASL
(D4)
achieve MT De (5y's Bisel
30.0
Ou (byf5 Bass)
teat BLEURT TLEURSS [0m2
Bexedre
MT Du- En (5y'5 Bisel
test: BLEURT FLEURS (tm)
27.5
Buseeine
25.0
22.5
20.0
17.5
(Ry'5
(By75
0.3
9.4
9.5
0.6
0.7
0.8
0.9
0.3
0.4
9.5
9.6
08
0.9
0.3
0.4
9.5
9.6
0.7
0.8
0.9
Rind
Ree
Figure 4: Pretraining performance for Baseline + MT when changing the mixing ratio of MT data pmnt on FLEURS-ASI #0 (En and De) test set. We show BLEURTY results as vary Pont from 0.3 to 0.9.
across languages. On average, X- benefits ASL-to-En SLT more than X++En: +0.08 and +0.9 BLEURT on How2Sign and FLEURS-ASLAS (EN), respectively. We speculate that including trans- lation into X uses model capacity, which, while enabling zero-shot ASL-to-X SLT as discussed be- low, results in slightly worse ASL-to-En performance. This suggests that MT data with the same target language as SLT is most effective for transfer. Table 3 shows further support where En-x surpasses Xn on multilingual SLT.
We can achieve zero-shot bilingual ASL-to-X SLT via ASL-to-En SLT En++X MT, albeit at poor quality. knowledge can be transferred from MT to SLT, one straightforward question is whether shot SLT by jointly training with MT. We do so by train- ing on ASL-to-En SLT En<+X MT data and examining zero-shot ASL-to-X SLT on FLEURS-ASIAR (X). Figure shows that this works effectively. On Pl and It, ob- serve quality gains over 12 BLEURT: on average, adding MT data improves zero SLT by ~6 BLEURT. Nev- ertheless, the overall zero-shot SLT performance is mid- dling, and the gains are unstable across languages, c.g. performance degrades for ASL-to-Hi SLT with joint MT training. Similar findings were also observed in multilin- gual MT and speech translation [57 [3]. Deeper analysis in Appendix AA. reveals that zero-shot SLT also suffers from the off-target translation problem [57]. i.e. trans- lating into a wrong target language; adding MT data can alleviate it, mostly for high-resource languages.
BLEURT
SLT Data
Dir
Small
Large
Baseline + YT-ASL
15.85
17.21
Baseline YT-Fuul
24.36
23.16
Baseline + MT-Small
En-+X 23.51
21.25
YT-ASL
X-
17.44
19.72
En++X
23.84
19.19
En-+X 27.29
23.15
YT-Full
X- 22.48
22.47
En 26.33
21.48
Baseline + MT-Large
YT-ASL YT-Full Ent+X
En++X
24.69
26.60
29.52
30.69
Table Pretraining performance for Base- line + MT with Pmt 0.9 when scal- ing up languages and data. We show averaged BLEURTT results FLEURS- ASL#0. Results are for ByT5 Base. "Dir": translation direction of data; "Small- /Large": average results over the target lan- guages included in WTT-Smal/MTT-Ln on FLEURS-ASLEN
Using higher sampling ratio for the MT data, Le. larger Pmics often improves SLT. We start with Pmnt 0.5, i.e., sampling equal amount of SLT and MT data, in the above experiments following intuition. However, the proportion of different types of data often has non- negligible influence in multilingual modeling (2) D. We next explore impact on SLT and use MT En-De for illustration. Figure 4 and 6 shows that Pmt 0.5 is sub-optimal and sampling more MT data improves SLT most settings, regardless of using 35T5 Base or XL, YT-ASL YT-Full, MT De or De and 2ow2Sign or FLEURS- ASL#O (En/De). addition, increasing the proportion of MT data also improves zero-shot ASL-to-De SLT. Note another benefit of using more MT data is to accelerate training, as loading SLT data much slower than loading text-only MT data. We use Pont = 0.9 by default in the following experiments.
Multilingual MT improves multilingual (zero-shot) SLT. The above experiments mainly analyze SLT with bilingual MT. We next investigate how multilingual MT affects multilingual (zero-shot) SLT particularly the use of MT-Small and MT-Large. We report results for ASL--t--mmal and ASL. Larg SLT on C.LEURS-ASLAS #0 where Small and Large denote the target languages covered by
7

---

MT-Small and MT-Larre, respectively. Note all SLT directions are zero-shot except the translation English.
Table summarizes the average performance. Using multilingual X MT data results in unstable ASL-tt--X SLT performance, which even hurts SLT on YT-Full. In contrast, multilingual En-*X and En++X MT data are both very helpful to SLT, where the former often outperforms the latter. By default, we still use En+xX MT data in the following experiments as to fully leverage the knowledge in MT data during pretraining.
Figures 7a and 7b further show the language breakdown results. Adding multilingual MT signifi- cantly improves ASL-to-En SLT when using YT-ASL alone, while the gain almost disappears when using larger-scale SLT data, YT-Full. Again, note that the overall zero-shot translation quality is poor - the best average BLEURT on Small and Large is 29.52 and 30.69, respectively. Achieving significant ASL-to-X SLT requires techniques beyond naive SLT and MT data mixing.
Data augmentation and large-capacity modeling are promising methods for multilingual SLT. In MT. a common solution to improve zero-shot qual- ity is to construct pseudo translation data for zero- shot directions 12. 15 56 [6]. We examine this practice for SLT. We adopt publicly pretrained MT models to generate data for more target languages for the YouTube SLT data (i.e., Augmented SLTT).
BLEURT
Setting
Small
Large
Baseline + YT-ASL
15.85
17.21
+ Buu--YT-SI--Sma
31.14
19.74
+ -MT-Small
30.51
19.70
Aug- --M------&&MM-aaggee
25.83
33.71
Results in Table 4 demonstrate the effectiveness of Augmented SLT, which significantly improves the best performance for ByT5 Base-based SLT to 36.01 and 39.85 average BLEURT on Small and Large with gain of 6,49 (29.52-+36.01) and 9.16 00.699439.855 .8), respectively. Note there are 42 languages in Large. ByT5 Base may be insuffi cient in accommodating translation for such amount of languages. Increasing the modeling capacity to XL. yields another gain of 9.11 (36.01-44.112) and 8.2 (39.8 199.55-488 -48.05 average BLEURT on Small and Large, respectively. On YT-ASL, Augmented SLT and ByT5 XL also lead to substantial qual- ity improvements by 13.84 (24.69-38.53)/155 (26.60 average BLEURT on Small/Large. The final performance even surpasses the cascading baseline, i.c. ASL-to-En SLT chained with En-to-X MT. under both YT-ASL and YT-Full. Figures Saa and 8b also show the quality improvements across languages resulted from data augmentation and 8L.
ByT5 XL
38.53
42.56
MT-3B Cascading
34.82
37.82
Baseline YT-Full
24.36
23.16
Aug- Mug-YT-Full-Saal
38.53
29.49
+ -MT-Small
36.84
25.67
Aug YYTFulllMT-Larg
36.01
39.85
ByT5 XL
45.12
48.05
MT-3B Cascading
43.54
46.32
ByT5 XI.
44.82
47.52
Table 4:
Pretraining performance (averaged BLEURT() for Baseline + Augmented SLT MT with pmn 0.9 FLEURS-ASLAW test set. MT data are multilingual in both directions. Baseline is for ByT5 Base; "MT-3B": MADLAD-MT-3B, the model used for SLT augmentation; "Cascad- ing": translating PLEURS-ASLE then performing MT other target languages.
English and
4.2 SLT Finetuning Results
We report the finetuning performance measured by BLEURT in Table We also include the BLEU and ChrF' results as well as the corresponding pretraining results in Appendix (Tables g and 9.
Finetuning on downstream benchmarks substantially improves SLT performance. Table shows that finetuning the pretrained SLT models yields substantial quality gains across benchmarks and settings. This is because the potential of pretrained models is not fully elicited by direct eval- uation due to video recording, domain and (clip-based) pretraining vs. (segment- bbseed inference mismatches, and finetuning largely mitigates these gaps. For example, pretraining with external aug- mented SLT and MT data results in even worse pretraining performance ((6)~(7)) in Table 9 After finetuning, nevertheless, model (7) significantly surpasses model (6) by 5.12 BLEURT average.
Adding multilingual SLT data (YT-Fulll into the pretraining greatly improves the performance from 14.26 (model (5)) to 32.48 BLEURT (model (8)) in Table 9 However, the quality gain after fine- tuning for YT-ASI. based models is often higher than their YT-Full counterparts, where the largest gain reaches ~28 BLEURT for model (5). We argue that pretraining on YT-ASL mainly teaches
8

---

WMT23
ID Model
H2S
E23
Avg
LIS-CH
LSF-CH
SRF
S8
0
Prevous SOTA
50.80
25.20
18.80
24.60
37,70
1
ByT5 Base
34.00
22.14
22.77
7.74
15.41
26.88
21.49
2
Baseline + YT-ASL
51.74
37.79
24.24
15.43
21.82
35,59
31.10
MT-Smmall (Pins = 0.9)
52.62
45.98
33.10
24.58
23.33
45.45
37.51
Aug- YT-ASL-Small
53.36
49.34
38.61
25.87
49.61
40,91
5 4 Aug-YT YT-ASL&MT-Large ByT5 XL
54.28
54.16
38.93
27.29
28.42
51.73
42.47
6
2 YT-Full
53.51
49.48
42.11
31,16
21.15
44.28
40.28
7 6 Aug-YT--A "TTASSLMTT-Smal
53.70
53.13
45.09
37.69
30.31
52.45
45,40
8 7 Aug- YT-ASI Lar- + ByT5 XI.
55.69
56.94
51.94
41.14
33.94
57.96
49.60
9 8 Multilingual SLT Tuning
53.47
55.57
54.54
39.26
29.33
58.08
48.38
Table 5: Finetuning performance (BLEURTT) on downstream SLT benchmarks. "H2S/E23":
How22ign/Ellmeenary23 "SRF/SS": WMT23 DSGS SRF/SS test split. "Avg": averaged performance over all benchmarks. MT data are added in both translation directions. Previous SOTA: How2Sign (43). Elemen- tary23 (48) and WMT23 SRF (28), WMT23 LIS-CH, LSF-CH. [44). All models are finetuned on each SLT benchmark separately except (9).
understanding of ASL, so pretrained performance other sign languages is poor, but finetuning can quickly adapt the learned representations to other sign languages.
Note we also finetuned the vanilla ByT5 model without SLT pretraining for reference, which achieves 7.68 and 3.10 BLEU on Elementary23 and WMT23 DSGS SS, respectively. Despite their inferiority, these results already surpass the previous SOTA, further showing the potential of ByT5,
A model's pretraining performance may be misleading when estimating its downstream fine- tuning performance, depending on the evaluation metric. Intuitively, a model with better pre- trained results should result in better finetuned results. The Spearman's correlation results in Table 6 confirm this intuition, where the correlation scores are positive across metrics. However, BLEU and ChrF have correlation score of 0.347 and 0. 186, respectively, which are very moderate. The correlation for ChrF is even not significant, which may be caused by the use of BLEU as the model selection metric. In contrast, the correlation of BLEURT reaches 0.578 and is significant at p < 0.01.
Model, data and language scaling together leads to new state-of-the-arr results. Diving deeper into Table we see clear improvements brought by scaling model size, data, and/or languages for SLT. Adding YT-ASL SLT data into the pretraining yields ~0 average BLEURT improvement (())+ Jointly training SLT with MT data produces another gain of ~~ BLEURT (())~(3)). Data augmentation adds improvement of ~33 BLEURT ((3)-(4)), which matches the quality achieved by adding large amount of extra multilingual SLT data the baseline, i.e. (4) 40.91 vs. (6) 40.28. By further increasing the amount of MT and augmented SLT data as well as the ByT5 model size, we reach an average BLEU, ChrF and BLEURT of 16.90 39.49, and 49.60, respectively (model (8)). These results also outperform previous best results, establishing the new SOTA.
BLEU ChrF
BLEURT 0.578
Spearman's 0.3471
0.186
Table 6: Spearman correlation between di- rect (i.e. pretraining) and finetuning SLT re- sults under different metries based on Tables Band 9 r/t: significant at < 0.05/0.01.
S
Multilingual finetuning improves multilingual SLT with encouraging performance, although it still underperforms bilingual finetuning on average. We next study multilingual finetuning on the direct mix of different SLT benchmarks. Table 5 ((8)~(9)) shows that multilingual SLT out- performs previous SOTA on almost all benchmarks, but underperforms its bilingual counterpart by 1.22 BLEURT on average. How balance modeling capacity among different languages in a joint model and avoid cross- linguiill///moaalii interference is well known issue in multilingual model- ing 2 55 49). and multilingual SLT also suffers (55). which we leave to future. Still, multilingual SLT facilitates transfer to LIS-CH, leading to a substantial gain of 2.6 BLEURT ((8)>(9)).
5 Related Work
The main bottleneck of SL. is data scarcity. Early studies address this issue by developing more data efficient neural architectures and/or training algorithms. Camgoz et al. 4 pioneered the study with

---

pecoder-decodee based recurrent models for SLT. which was quickly replaced by Transformer and multi-task learning with CTC regularization [5]. Zhou et al. [6]] developed spatial-temporal archi- tecture to model the collaboration of different visual cues. Another way is to transfer the knowledge from pretrained models, augmentations, and other tasks. Chen et al. 7 g proposed to leverage pretrained visual encoders and MT models to improve SLT, while Zhang et al. (58] explored trans- ferring translation knowledge from MT data directly. Zhou et al. [60] employed
back-translation to generate pscudo SLT training data. Ye et al. [54] augmented the training data by the mix-up algo- rithm. Yet another way to address data scareity is to make data less scarce. Shi et al. 40), Uthus et al. [46], and Tanzer and Zhang [44] collected large-scale SLT data from YouTube and improved data quality via manual filtering: Albanie et al. [ developed a British Sign Language translation corpus based on BBC broadcasts instead. Tanzer [g scaled up ASL data by eschewing manual filtering and tolerating misaligned or irrelvant data. We follow and scale to noisy multilingual sign language data, MT data, and augmented paralel data.
Despite the aforementioned advancements, many studies still heavily depend on sign glosses. As a bridge between sign video and target text, sign glosses ease learning, but are expensive to annotate, not always available, nonstandardized, and cannot cope with sign language grammar in general- ity 12 Recent research therefore turns to gloss-free SLT, which often underperforms gloss-based counterparts [25 59 50 and performs poorly in open-domain settings [3] [1] 28). We substantially improve gloss-frce SLT performance across benchmarks through scaling. In this regard, our work is closely related to SSVP-SLT [37] but with different focuses. SSVP-SLT improves SLT by pre- training neural sign encoder through large-scale self-supervised learning. By contrast, adopt static landmarks to represent sign frames and improve the translation by transferring knowledge from other languages and tasks. The methods used in our study are orthogonal SSVP-SLT. In addition, our work also falls into the category of improving multilingual SLT (55] 20). We didn't t evaluate our models on these multilingual benchmarks though as they are either unavailable the time of paper writing or unusable due to licensing issues.
Conclusion, Limitations, and Future Work
We presented a systematic study of data, model and language scaling for SLT via large-scale SLT pretraining. In general, scaling substantially improves SLT. We observe positive knowledge transfer from other sign language data and from machine translation data. By joint SLT and MT training, we show the feasibility of achieving zero-shot SL.. Data augmentation expanding SLT data to more spoken languages via off-the-shedt MT models significantly improves multilingual SLT. Putting ev- erything together, finetuning our pretrained SLT models leads to new state-of-the-as SLT results across 5 benchmarks covering 5 sign languages (but still far from usable quality).
Although our models have nominally been pretrained on massive number of sign languages (up to 80), we lack comprehensive and reliable multilingual benchmarks fully understand their abilities and limitations. In addition, our models are limited to encoder-decoder based (m/By)TS models, and SLT pretraining requires many computational resources, increasing the difficulty of reproduction.
In the future, we expect that continuing to scale sign language data, number of sign languages, vision pretraining/multimodaainn etc. will reap further gains. As suggested by [42] it will be important to evaluate these growing capabilities on multilingual, standardized, open-domain SLT benchmarks.
Ethics Statement
We preprocess all sign videos with simplified landmarks as form of anonymization and privacy protection. While the pretraining SLT data is larger scale than prior work, it may still suffer from demographic biases. Even if demographies were represented in proportion to the real world, and even with simplified landmarks, the resulting SLT models may not perform equally across groups and should be evaluated for fairness before real-worke deployment. Our study mainly aims to un- derstand the impact of scaling on SLT, and while we significantly improve translation quality, it is still far from usable for real-world applications. For many such applications, the other half of sign language translation-sinn language understanding in this work. Advancing both of these is critical to ensure that Heaf/Hard of Hearing signers get equal access to technology and the information that comes through it.
language generation-is also essential, whereas we focus only on sign
10

---

Acknowledgements
We thank the reviewers for their insightful comments. We thank Ankush Garg for valuable feedback on this work, Chris Dyer for constructive comments that greatly improve the quality of this paper. Sam Sepah and Google Translate team for supporting this research. We also thank the T5X team [3 for infrastructure support.
References
[1] Samuel Albanie, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, et al. Bbe-oxford british sign language dataset. arXiv preprint arXiv:2111.0363: 2021.
[2] Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Find- ings and challenges, 2019.
[3] Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, et al. Building machine translation systems for the next thousand languages. arXiv preprint arXiv:2205.0399
2022.
[4] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. Neural sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784-7793, 2018
[5] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10023-10033, 2020.
[6] Isaac Caswell, Ciprian Chelba, and David Grangier. Tagged hack-translation. In Ondíej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Ne- gri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor, editors, Proceedings of the Fourth Conference Machine Translation (Volume I: Research Papers), pages 53 63, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/11/11-5520 URL httph: httpps://aclaathologg/ wwwo/o/g//.......... /19-/1199-111 5206
[7] Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5120-5 5330, 2022.
[8] Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems, 35:17043-17056, 2022.
[9] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint 1.Xiv:111..02111 2019.
[10) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207 0.662,, 2022.
[11] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512, PMLR, 2023.

---

[12] Aashaka Desai, Maartje De Meulder, Julie A. Hochgesang, Annemarie Kocab, and Alex X. Lu. Systemic biases in sign language research: A deaf-led call reevaluate research agendas, 2024.
[13] Tu Anh Dinh. Zero-shot speech translation. arXiv preprint arXiv:2107.06010 2021.
[14]
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Flo- rian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. Beyond english- centric multilingual machine translation. Journal of Machine Learning Research, 22(107):
1-48, 2021.
[16] Markus Freitag and Orhan Firat. Complete multilingual neural machine translation. arviv preprint arXiv:2010.10239) 2020.
[17] Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André T. Martins. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In Philipp Koehn, Loie Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag. Yvette Graham, Roman Grund- kiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Mat- teo Negri, Aurélie Névéol, Mariana Neves, Martin Popel. Marco Turchi, and Marcos Zampieri, editors, Proceedings of the Seventh Conference Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL http:psss//aaaaahoooogsseo/ org. 2022. wmt 1.2
[18] Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiss Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 18800 2023.
[19] Team Gemini, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
[20] Shester Gueuwou, Sophie Siake, Colin Leong, and Mathias Müller. JWSign: highly multilingual corpus of Bible translations for more diversity in sign language processing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9907-9927, Singapore, December 2023, As- sociation for Computational Linguistics. doi: 10. 18653v/ 1/0023.212222222 hww.....ininggemml...66... URL htttps://aclanthol..yy org 2023 Wiindings-emnlln 1mmmp.6664
[21] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.1 11644, 2023.
[22] Tom Koemi, Christian Federmann, Roman Grundkiewicz, Marcin Junzzys-oommunt, Hi- tokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evalua- tion of automatic metrics for machine translation. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexan- der Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Koemi, Andre Mar- tins, Makoto Morishita, and Christof Monz, editors, Proceedings of the Sixth Conference on Machine Translation, pages 478 Online, November 2021. Association for Computational Linguistics. URL https: http:::s//acaaaaholllooo org/ 2021. wmt 1.57
12

---

[23] Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati. Romi Stella, Ankur Bapna, and Orhan Firat. MADLAD-400: A multilingual and document-level large audited dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL. htttps://opeereeiee net/ id= -d
[24] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burees and Nicholas Carlini. Deduplicating training data makes language models better. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424 8445, Dublin, Ireland, May 2022. Association for Computational Lin- URL
guistics. doi: 10. 10.11855/11/2222.acll acl-long.50 577
.rg org///22.. 2022.
[25] Kezhou Lin, Xiaohan Wang. Linchao Zhu, Ke Sun, Bang Zhang, and Yi Yang. Gloss-free end-to-end sign language translation. arXiv preprint arXiv:2305. 12876, 2023.
[26] Camillo Lugarssi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.0811 2019.
[27] Qingsong Ma. Johnny Wei, OndA Bojar, and Yvette Graham. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Florence, Italy, August 2019. Association for Computational Linguistics. URL acwweb. org/ www//oggttt/ooo wnt.tlogooooooooooooooooo 0t1///19955522
http://www.aclweb..o
[28] Mathias Müller, Malihe Alikhani, Eleftherios Avramidis, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Sarah Ebling, Cristina Españia-Bonet, Anne Göhring, Roman Grund- kiewicz, Mert Inan, Zifan Jiang, Oscar Koller, Amit Moryossef, Annette Rios, Dimitar Shteri- onov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt. Findings of the second WMT shared task on sign language translation (WMT-SLT23). In Philipp Kochn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 68-94, Singapore, December 2023, Association for Computational Linguis- ties. doi: 10. 00.11663//1/223.. wmt--022...mmmmmmmm wmt- 1.4. htt.:tttts//acaaaaaaaa hwwphhhooogy.oooog./eeeoo arg/2023. 1.4
[29] Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Qual- ity not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 35:21455-21469, 2022.
[30] Team OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[31] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318 USA, 2002. Association for Com- putational Linguistics. doi: 10.3 011111110103310735300
URL WtRLhttps:: http::::/dooooo/////// wt..///.//................... 3115/
1073083. 1073135
[32] Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondřcj Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Var- vara Logacheva, and Pavel Pecina, editors, Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 11.18653/v1/1115333 3049. URL https: httt:::s//alaanthol...o
org/W15-3049
[33] Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam ZarandZarandi, al. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305. 13:11136,...16,,, 2023.
[34] Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for mt. In Proceedings of EMNLP, 2021.
13

---

[35] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
[36] Adam Roberts, Hyung Won Chung, Gaurav Mishra, Anselm Levskaya. James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scal- ing up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):
1-8, 2023.
[37] Phillip Rust, Bowen Shi, Skyler Wang, Necati Cihan Camgöz, and Jean Maillard. Towards privacy-aware sign language translation scale. arXiv preprint arXiv:2402.09611, 2024.
[38] Marcelo Sandoval-Castaneda Yanhong Li, Bowen Shi, Diane Brentari, Karen Livescu, and Gregory Shakhnarovich. TTIC's submission to WMT-SLT 23. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 344-350, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/ 1.35..35. URL psttps:/accaanthol.oo org/ 2023. wmt 1.35
[39] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4604. PMLR, 2018.
[40] Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. Open-domain sign language translation learned from online video. Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6365-6379, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: https://aclamtthologg 0r2/2022. emnle-main m2in.4277
URL
[41] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neu- ral networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014. URL https: httpssst///prrreediiigg wwwp:iinnnnnnuur...eee.m neurips. wwtppppereielle///peeeeeeeeepeep .ddf
[42] Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multi- task evaluation, 2024. URL htttss:/arxxii org. www./oo///////e.e. abs///bb//44/ 2458.13885
[43] Garrett Tanzer. Fingerspelling within sign language translation, 2024. URL BRLLhttps:// //rxxva.vvvvvv org/ org/abs/2e 0alba//408.00000
[44] Garrett Tanzer and Biao Zhang. Youtube-sl-25: A laage-scale, open- multilingual sign language parallel corpus, 2024. URL httpttt////xxi org/abs//////o/eooooo..oooe ab///bb/////4// /407 11144
[45] Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró Nieto. Sign language translation from instructional videos, 2023.
[46] Dave Uthus, Garrett Tanzer, and Manfred Georg. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus. Advances in Neural Information Processing Systems, 36, 2024.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] Andreas Voskou, Konstantinos P Panousis, Harris Partaourides, Kyriakos Tolias, and Sotirios Chatzis. A new dataset for end-to-end sign language translation: The greek elementary school dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1966-1975, 2023.
[49] Zirui Wang, Zihang Dai, Barnabas Poczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
14

---

[50] Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2GPT: Leveraging large language models for gloss-free sign language translation. In The Twelfth International Conference on Learning Representations, 2024. URL.
http://penrevee..eetteeeeee
id=LqaEEs3UxU
[51] Baixuan Xu, Haochen Shi, Tianshi Zheng. Qing Zong, Weiqi Wang, Zhaowei Wang, and Yangqiu Song. KnowComp submission for WMT23 sign language translation task. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 351-358, Singapore, December 2023. As- sociation for Computational Linguisties. doi: 10. 10.18833/11/223. .tt2233.mmm11... 1m3-
URI. https:
//aclanthology. 2rgg/023..22 1.
[52] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre-trained text-to-text trans- former. arXiv preprint ar Xiv: 11934, 2020.
[53] Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. doi: 10. 11l66/aalaa_0006666
org/ wtt.//22.......ooooo 2022. taattaacll111 1.
[54] Jinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Hui Xiong. Cross-modality data augmentation for end-to-end sign language translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13558-13571, Singapore, December 2023. Association for Computational Lin- guistics. doi: 18.18 htt.l11.2222222... wineeiiiinngeennl..000 URL 2023 fiws.ffiningseeiiie emmlp. 904
org/
[55] Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Mislt: Towards multilingual sign language translation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5099-5109, 2022. doi: 10. 10.1109/CVPR52688.
2022.00505.
[56]
Biao Zhang and Rico Sennrich. Edinburgh's end-to-end multilingual speech translation sys- tem for IWSLT 2021. In Marcello Federico, Alex Waibel, Marta R. Costa-jussà, Jan Niehues, Sebastian Stuker, and Elizabeth Salesky, editors, Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 160-168, Bangkok, Thai- land (online), August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 1.19. URL /ttts::/aclantie
org/ 2021 iwslt-1.19
[57] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. Improving massively multi- lingual neural machine translation and zero-shot translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628-1639, Online, July 2020. Asso- ciation for Computational Linguistics. doi: 10. 10.18533v1/2020.cc maii::///.......mmm...mmmmmmmm 148. URL https: //aclanthology orgooo//// 2020. acl maci. 148
[58] Biao Zhang, Mathias Müller, and Rico Sennrich. SLTUNET: A simple unified model for sign language translation. In The Eleventh International Conference on Learning Representations, 2023. URL httt::::///ppevvvviiee net - EBS4C77p_59
[59] Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual-language pretraining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20871-20881, 2023.
[60] Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign mack-translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1316-1325, 1325, 2021.
[61] Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cus network for sign language recognition and translation. IEEE Transactions on Multimedia, 24:768-779, 2021.
15

---

A
Appendix
Table 7: Data statistics for YouTube sign language and MADLAN spoken language. We list ISO 639-3 code, language name, and the number of hours/clips/videos for sign language; for spoken language, list BCP-47 code, language name and the number of parallel examples. "K" thousand, "M": million. Note that like Tanzer and Zhang 4 pre-filtering and Tanzer [ these language labels are heuristically estimated based on public video metadata, such as caption language and text the video title, description, etc.
Sign Language (SL)
Spoken Language
ISO 639-3
Name
Hours
# Clips
Vidcos
BCP-47
Namc
# Examples
ase
American SL
2.8K
285. .2K
25.9K
cs
Spanish
292.8M
bzs
Brazilian SL
590.4
60.2K
5.9K
de
German
283.3M
pso
Polish SL
421.8
41.8K
3.2K
fr
French
243.6M
ins
Indian SI.
375.3
39.3K
5.7K
Italian
100.1M
bfi
British SI.
267.3
27.4K
2.7K
CS
Czech
53.1M M
German SL
235.2
24.2K
2.1K
pl
Polish
42.9M
fsl
French SL
193.7
19.9K
2.9K
Russian
29.0M
jsl
Japanese SL
176.5
17.9K
3.0K
Simplified Chinese
25.9M
ise
Italian SL.
161.0
16.8K
2.4K
ar Arabic
18.2M
asf
Australian SL
123.1
12.5K
1,5K
ja Japanese
M.3M
rsl
Russian SL
119.4
12.1K
1.2K
hi Hindi
1.2M
esc
Catalan SL
114.8
11.8K
1.7K
nl Dutch
93. M
csn
Colombian SL.
107.8
11.1K
478.0
Portuguese
83.7M
aed
Argentine SL
86.1
8.8K
522.0
Swedish
$1.9M
mfs
Mexican SL.
76.9
7.6K
434.0
hu
Hungarian
40.0M
kvk
Korean SL.
69.3
7.1K
612.0
da Danish
38.2M
hsh
Hungarian SL
55.9
5.8K
1.3K
fi
Finnish
34
sgg
Swiss German SL
48.7
5.1K
634.0
el
Greek
25.2M
prl
Peruvian SL
42.3
4.2K
147.0
Slovak
25.0M
fse
Finnish
4.3K
593.0
no Norwegian
19.4M
swl
Swedish SL.
38.7
4.0K
563.0
bg
Bulgarian
15.5M
asq
Austrian SL
31.8
3.2K
592.0
It Lithuanian
15.3M
tsm
Turkish SL
31.8
3.3K
446.0
lv Latvian
14.3M
dse
Dutch SL
3.2K
352.0
Slovenian
11.8M
ese
Czech SL
29.2
3.0K
336.0
el Estonian
11.0
inl
Indonesian SL
27.5
2.8K
236.0
ko Korean
5.8M
nsl
Norwegian SL
22.0
2.2K
215.0
hr
Croatian
5.3M
hks
Hong Kong SL
2.2K
248.0
is
Icelandic
M.1M
tss
Taiwan SI.
19.6
2.0K
258.0
sr Serbian Turkish
2.5M
gss
Greek SL.
18.2
1.9K
169.0
2.5M
dsl
Danish SL.
16.0
1.6K
188.0
vi Vietnamese
1.5M
csg
Chilean SL
15.2
1.5K
184.0
id Indonesian
1.4M
sib
French Belgian
14.4
1.5K
358.0
he Hebrew
1.IM
isr
Israeli SL
14.3
1.4K
289.0
th Thai ms Malay
1.IM
vietnam
Vietnamese SL
14.2
1.1K
97.0
907.5K
isg
Irish SL
13.2
1.4K
101.0
uk Ukrainian Catalan
881.0K
slovenia
Slovenian SL
12.7
1.3K
177.0
686.2K
nzs
New Zealand SL
1.3K
224.0
ta Tamil
396.9K
icl
Icelandic SL
11.1
1.2K
213.0
fil Filipino
369.8K
sls
Singapore SL
9.9
1.K
148.0
ne Nepali
277.9K
tsq
Thai SL
9.0
923.0
150.0
сy
Welsh
93.3K
pks
Pakistani SL
8.6
902.0
145.0
svk
Slovak SL
8.5
886.0
114.0
jos
Jordanian SL.
8.3
880.0
124.0
lls
Lithuanian SL.
7.6
792.0
166.0
Costa Rican SL.
7.6
789.0
45.0
psr
Portuguese
7.4
764.0
127.0
rms
Romanian SL.
7.3
747.0
60.0
xml
Malaysian SL
7.2
668.0
85.0
Ecuadorian SL
7.2
727.0
28.0
psp
Filipino SL
6.8
715.0
85.0
sfs
South African SL
5.2
543.0
41.0
ugy
Uruguay SL.
4.7
483.0
113.0
16

---

esn
Salvadoran SL
3.9
403.0
15.0
xki
Kenyan SL
3.8
384.0
23.0
serbia
Serbian
3.3
351.0
esq
Croatian SL
3.1
326.0
31.0
esl
Egyptian SL
3.0
262.0
32.0
psl
Puerto Rican SL
1.8
181.0
17.0
bengladesh
Bengali SL
1.6
165.0
15.0
gsm
Guatemalan SL.
144.0
xins
Moroccan SL.
1.2
125.0
4.0
Isp
Panamanian SL
1.2
123.0
8.0
fes
Quebec SL
0.9
91.0
27.0
eso
Estonian SL
0.9
90.0
emirati
UAE SL.
0.8
81.0
14.0
vsl
Venezuelan SL
0.7
69.0
15.0
pys
Paraguayan SL
0.7
69.0
5.0
kazakh
Kazakh SL
0.6
69.0
6.0
hds
Honduran
67.0
macau
Macau SL.
0.6
109.0
109.0
sdl
Saudi Arabian SL
0.5
49.0
7.0
dog
Dominican SL
0.5
45.0
10.0
belarus
Belarusian SL
0.4
5.0
bqn
Bulgarian SI.
0.3
31.0
9.0
sqs
Sri Lankan SL.
0.3
29.0
8.0
Isl
Latvian SL.
0.2
27.0
3.0
bvl
Bolivian SL
0.2
26.0
5.0
nsi
Nigerian SL
0.1
12.0
nsp
Nepali SL
0.1
6.0
1.0
45
Baseline
w/ MT x En
test BLEURT on How25lgn
35
31.8
14.5
15.2
33.0
31.4
33.2
31.2
27.0
Es De Fr
Ru
2h
Ar
Ja Aug
Added MT Languages
Figure 5: Pretraining performance for Baseline + MT when varying MT languages on How2Sign test set. We show BLEURT? results and set Pms = 0.5. Note only YT-ASL and bilingual MT data are used, i.c. MT languages added separately instead of jointly, Results for ByT5 Base. "X-En"" MT data for translation into English; "X+Enn: MT data both translation directions: "Avg": average performance over languages. MT languages are arranged in descending order from left to right based on the quantity of translation data available for each language.
A.1 Setup
Sign Video Preprocessing Our landmark preprocessing is identical to [46]. and we use the same random 34-second video clipping as [42]. We preprocess sign language video with its default frame rate but discard every second frame for computational efficiency. We convert each video frame to 255-dimensional normalized vector using McdiaPipe Holistic landmarks (26), which also facilitates video anonymization. The input video is eventually transformed into vector sequence and then mapped to the encoder via a linear projection layer.
Downstream Benchmarks Note the official SLT track in WMT23 for LIS-CH and LSF-CH is for sign language generation rather than translation. We reversed as SLT dataset. FLEURS-ASI.. is the subset of FLEURS-ASL
42 recorded by signer #0, i.c. 353 sentences from FLORES [10)
17

---

YT-ASL
YT-FULL
MT De En (ByT5 Base)
En (ByT5 Base)
test BLEURT How25ign
Daseline
30
28
wd - (ByTS Base)
w/ MT De (ByT5 XL)
26
Baseline
0.6
0.7
0.0
0.9
0.3
0.4
0.5
0.6
0.7
Pox
Pnx
Figure 6: Pretraining performance for Baseline MT when changing the mixing MT data pmt on How2Sign test set. We show BLEURT? results and vary Post from 0.3 to 0.9. Note only bilingual MT En-De data are explored.
translated into ASL by a Certified Deaf Interpreter. We report only signer #0 because the rest of the benchmark was complete when these experiments were run.
For these benchmarks, we only use the sign language video and target text without glosses. All SLT models in this study are gloss-free.
Model Setting For Pretraining, We pretrain models up to 1M steps using 64/64/128 TPU-v3 chips for Base/Large/XL, taking 7- 20 days. We select the best checkpoint for downstream application based on the How2Sign dev performance measured by BLEU
a batch size of 256 and constant learning rate of 0.001.
For Finetuning, we use batch size of 32 and constant learning rate of 0.0005. By default, we perform finetuning on each downstream benchmark separately. We only consider the SLT task at finetuning, and directly finetune the model on well aligned (sign video segment, target translation) pairs, which is provided in all downstream benchmarks. We tune models up to 50K steps using 16/32 TPU-v3 chips for Base/XL, taking 2~5 days. We select the best checkpoint for final evaluation based on the dev-set BLEU.
A.2 More Results and Analysis
Zero-shot SLT benefits from MT data partially because reduces off-target translation. A key factor affecting zero-shot MT is the off-target problem, where the model translates into wrong target language [57]. We examine this problem for zero-shot SLT according experiments in Figure 3b and show the results in Table 10
We noticed that zero-shot SLT also suffers from off-target translation, particularly for those lan- guages distant from English. For example, Baseline only has language accuracy of 1.7, 45.0 and 42.2 for Cs. Zh and Hi, respectively. Adding MT data generally improves the translation language accuracy, such as 1.7/78.5 to 61. 61.2/97.7 for Cs/PI. But there are also exceptions, like Zh and Hi, where the accuracy reduces from 45.0 45.0/42.2 15.0/4.8. A deeper inspection shows that jointly train- ing with MT leads to more empty outputs for these languages: the empty rate increases from 2.8/1. .44 17.3/43.3 for Zh/Hi. We argue that this may be because 1) these languages have significantly less parallel MT data, e.g. Hi only has 1.2M examples, and 2) the parallel corpus from MADLAD can also be quite noisy.
Different evaluation metrics may disagree. There is a hot debate in MT community regarding which metric we should use for translation evaluation [22]. While BLEU has been widely adopted, often shows poor correlation with human evaluation, particularly when the translation models Instead, neural metrics are recommended [17] We follow this trend and adopt
are strong 27
'We didn't adopt BLEURT for model selection because it's significantly more expensive and time- consuming than BLEU.
18

---

YT-ASL
YT-Full
Baseline
Baseline
Test BLEURT on Flores (X)
Xn
X - En
40
10
de
2h
hi
de
fr
1t
2h
hi
Languages
Languages
(a) Results for training with MT-Small.
Daseline + (YT-ASL) Baseline + (YT-Full)
50
test BLEURT on Florss (X)
-30
10
enesde 2u jaja ptsvhuda el sknobg si etioo hr hathmsukkcata flll necy
Languages
(b) Results for training with MT--Large.
Figure 7: Per-language pretraining performance for Baseline MT with Pmnt 0.9 when scaling up languages and data. We show BLEURT! results FLEURS-ASE ASL#0. We add multilingual MT data into SLT pretraining compare MT-Small with MT-Large. Results are for ByT5 Base.
BLEURT as the main metric. To be compatible with past studies and also ease future comparison, we also add BLEU and ChrF. Table shows some disagreements between BLEURT and BLEU/ChrE. For example, model (4) performs better than (comparable to) model (5) on average based on Chrli (BLEU), while BLEURT scores show clear superiority of model (5) over (4). Evaluation metric selection should be more careful due to these disagreements. In this study, rely more BLEURT for the analysis as it correlates better with human evaluation [17]
Qualitative examples of translation quality. See Table for qualitative examples of finetuned ASL to English translation on How2Sign from our best model, compared to prior state the art set by Tanzer 43 Our model resolves some small issues in (2) and (3) analyzed in Appendix C of Tanzer [43]: the remaining discrepancies in these particular examples seem to be issues with the dataset quality.
19

---

YT-ASL
YT-Full
50
test BLEURT on Flares (X)
20
Buce
18/15 Buse)
Aug YL-LL Smm 1777
Full-malll ity1b
AughTTiGh (mral (DyTS
Aug-rT-rut--mma Shy1t X11
Aug Peral HTTSmal STYTS XLI
Aug-rrrul-Smnt MT--mmmi1 10yT1 X.
Languages
Languages
(a) Results for training with MT-Small and Aug- Smal
test BLEURT on Flores (K)
40
30
AupeTsAA-- diyts
Aug Larg- Margage X/
Lalgage Milaageeey's AU
sknoby Languages
hethmmssukaa necy
(b) Results for training with MT-Large and Aug YT-ASL/Ful--Laan Large.
SLT with augmented SLT data. We show BLEURTT results for Baseline Augmented SLT + MT with pms = 0.9 on FLEURS test set. MT data are multilingual in both directions. Data augmentation substantially improves SLT performance across languages.
Figure 8: Per-language pretraining performance
20

---

WMT23
ID Model
H2S
E23
Avg
LIS-CH
LSF-CH
SRF
SS
0
Prevous SOTA
18.10
5.69
5.20
7.00
0.30
7.50
7.30
1
ByT5 Base
3.71
7.68
0.40
0.79
1.01
3.10
2.78
2 + Bascline + YT-ASL
17.94
16.58
6.59
8.65
2.14
8.69
10.10
2 MT-Small (pms 0.9)
17.30
17.90
8.84
11.86
2.04
11.01
11.49
Aug- YT-ASS--Sall
18.55
22.09
12.91
14.34
13.43
5 4 Aug-Yo YT-ASL&MT-Laage ByT5 XL
19.31
24.08
9.70
11.71
2.76
13.26
13.47
YT-Fuul
19.79
21.81
15.32
2.07
13.71
14.28
6 Aug-YT-ASI YT-ASI.
18.98
23.60
13.63
15.75
2.85
14.44
14.88
8 7 Sogg-Y-ASLLMM-AAA ByT5 XL
21.06
25.65
14.93
18.77
2.80
18.17
16.90
9 8 Multilingual SLT Tuning
19.25
23.05
16.79
17.22
2.91
15.92
15.86
(a) BLEU scores.
WMT23
Model
Avg
LIS-CH LSF-CH
SRF
Prevous
ByT5 Base
19.55
25.19
14.81
15.56
11.08
21.75
17.99
Baseline YT-ASL
38.78
41.34
27.57
17.06
38.31
31.18
3 24 MTT-Small (pint 0.9)
39.01
45.40
30.56
33.67
16.13
40.38
34.19
+ Aug- Bug-YT-ASL-Sman
39.64
47.92
31.40
34.95
17.49
43.56
35.83
4 Aug- YT-ASL&MT-Larg ByT5 XL
40.15
48.70
29.44
33.20
17.96
40.21
34.94
2 YT-Full
41.13
47.71
38.23
38.13
16.78
42.33
37.38
Aug-YT--a YT---I---SAAAA &MT-Small
42.83
38.34
8 7 Mog-YT-ASL&MT-LAO ByT5 XI
41.97
50.09
38.90
40.43
19.58
45,94
39.49
9 8 Multilinuual SLT Tuning
40.39
48.49
40.28
38.63
18.72
45.22
38.62
(b) ChrFf scores.
WMT23
ID
Model
H2S
B23
Avg
LIS-CH
LSF-CH
Prevous SOTA
50.80
25.20
18.80
24.60
37.70
ByT5 Base
34.00
22.14
22.77
7.74
15.41
26.88
21.49
Haseline YT-ASI.
$1.74
37.79
24.24
15.43
3
2 + ST-lmalll (pme 0.9)
52.62
45.98
33.10
24.58
23.33
45.45
37.51
4 + Aug. YT-ASI--Smal
53.36
49.34
38.61
28.70
25.87
49.61
40.91
5 4 + BE--YYTT-&&MM-LLA.
ByT5 XL.
54.28
54.16
38.93
27.29
28.42
51.73
42.47
6
2 YT-Full
53.51
49.48
42.11
31.16
21.15
44.28
40.28
7
+ Aug- YT-ASL&MT-Small
53.70
53.13
45.09
37.69
30.31
52.45
45.40
7 Bug--T-ASLLMMT-Larr ByT5
55.69
56.94
$1.94
41.14
33.94
57.96
49.60
8 Multilingual SLT Tuning
53.47
55.57
54.54
39.26
29.33
58.08
48.38
(c) BLEURT1 scores.
Table Finetuning performance on downstream SLT benchmarks. "H2S/E23": Hoow2iigu/Elementary23. "SRE/SS"; WMT23 DSGS SRF/SS test split. "Avg": averaged performance over all benchmarks. MT data are added in both translation directions. Previous SOTA: How2Sign (43), Elementary23 (48] and WMT23
SRF 23), WMT23 LIS-CH LSF-CH. SS (44). Scaling SLT reaches new SOTA across benchmarks. All models finetuned each SLT benchmark separately except (9).
21

---

WMT23
Model
H2S
E23
Avg.
LIS-CH LSF-CH SRF
SS
ByTS Base
+ Baseline + YT-ASL
3.77
0.06
0.15
0.15
0.15
0.77
+ MT-Small (Pms 0.9)
4.75
0.02
0.07
0.88
3 Aug-YTT Mug-YT-ASL--malll
3.31
0.01
0.12
0.43
0.31
0.72
5 4 Aug-Y----- YT-ASL&MI-Lagge
XL
2.81
0.21
0.22
0.31
0.06
0.24
0.64
6
2+ YT-Full
5.78
0.33
3.43
5.69
3.37
7 6+ Bug-YT-ASL&MM--SIE
4.10
0.05
1.67
2.65
2.21
1.86
7 Aug- YT-ASL&MI-Large ByT5 XL
4.05
2.45
4.50
BLEU scores.
WMT23
ID
Model
H2S
E23
Avg
LIS-CH LSF-CH
SRF
SS
BYT5 Base
Baseline- YT-ASL
20.65
6.62
12.67
15.1
13.89
13.95
13.81
MT-Small (pmnt 0.9)
19.55
0.05
10.75
12.89
10,91
4 3 Aug. YT-ASI--Small
15.21
0.05
9.67
9.87
5.89
14.28
9.16
5
4 Aug-YT-ASL&GR YT-ASL&MT--Lunge ByT5 XL
11.40
9.25
8.75
6.88
3.35
7.55
7.86
6
2 YT-Full
23.44
14.81
25.34
26.78
16.42
28.36
22.53
7 6 Aug-YT--N Bug-YT--AS&AN YTT-ASL&MT--Smll
18.18
10.22
19.81
22.64
10.25
25.95
17.84
7 Aug-YY--SS&&MT-Lmre ByT5 XL
13.47
22.34
25.53
26.16
14.82
26.96
21.55
(b) ChrFf scores.
WMT23
ID Model
H2S
B23
Avg
LIS-CH LSF-CH
SRF
SS
ByTS Base
2
+ Baseline + YT-ASL
30.36
9.13
9.32
6.33
9.69
10.45
12.55
MT-Small (Pme 0.9)
34.24
16.38
10.44
13.14
14.81
15.01
4 3 + Aug- YT-ASI--Smaa
25.2
1.63
23.41
10.82
10.65
22.06
15.61
5
4 + Aug-YT- Migg-Y-ASL&MT-Larrr ByT5 XL.
23.87
10.35
21.58
6.75
7.04
15.96
14.26
6
2 YT-Full
37.13
15.08
28.92
18.33
17.87
29.66
24.50
7
6 Sig-YT-ASL&MT-SS.
25.25
12.68
33.54
24.48
10.66
33.91
23:42
Aug-YT-ASL&MN -EE---TSS-L--Larg ByT5
22.41
34.14
43.07
34.26
21.52
32.48
(c) BLEURTY scores.
Table Pretraining performance on downstream SLT henchmarks.
Language
es
C5
pl
ar
ja
Baseline
90.9
85,6
92.4
82.4
78.5
95.2
45.0
72.2
76.5
42.2
Xn
95.2
93.2
94.1
91.2
61.2
97.7
85.3
15.0
75.4
73.7
(a) Language Accuracy: the accuracy of translations in correct target language.
Language
it
es
pl
ar
ja
hi
Baseline
8.2 4.0
6.5
5.1
7.6
2.3
0.0
2.8
5.1
0.3
1.4
MT En
0.6
0.6
0.6
18.7
0.3
43.3
Empty Rate: the proportion outputting empty translations.
Table 10: Analysis for zero-shot SLT in Figure 3b Higher language accuracy indicates less off off-target transla- tion, thus better quality; lower empty rate is better.
22

---

Reference
And that's great vital point technique for women's self defense. It's really great point for women's 's self defense.
Tanzer [43
Ours
It's a really great point for women's 's self defense.
Reference
In this clip I'm going to show you how to tape your cables down. In this segment I'm going to show you how draw name of the cable. In this clip I'm going to show you how to tape the cable.
Tanzer 44
Ours
Reference
In this segment we're going to talk about how to load your still for distilla- tion of lavender essential oil. In this clip we're going to talk about how install still for ditching nice for a lavender oil.
(3) Tanzer 43
Ours
In this clip, we're going to talk about how to feed the still for digestion and for lavender oil.
Reference
You are dancing, and now you are going to need the veil and you are going just grab the veil far possible.
(4) Tanzer
Now we're going to have her dance, and now we're going to have her braided her hair up as far as possible.
Ours
Your dancing, now we need the fringe to come up as far as possible.
Reference
But if you have to setup new campfire, there's two ways to do it in very low impact; one with mound fire, which we should in the campfire segment earlier and the other way to setup a low impact campfire is to have a fire pan, which is just a steel pan like the top of a trash can.
Tanzer 43
But if you have to set up new campfire, there's 's two ways to do it in low impact, one is a bond fire, which we should do a campfire stack early and the other one is to set up campfire in fire pan, that's just steel pan like the top the pan. But if you have
(5)
Ours
campfire, there's two ways do it in low impact, one is bond fire, which we should do in campfire sanding early, and the other one is set up campfire in fire pan, which is just steel pan like the tops of the pans.
up
Reference Tanzer 43
So, this is very important part of the process. This is part of the process.
(6)
Ours
And that's 's okay, part of the process.
Table 11: Qualitative examples of finetuned sentence-level ASL to English translation results on
How2Sign. instances originally selected by Tarrés al. [45) We compare the reference, the prior SOTA Tanzer
43 and our best model.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We summarize our contributions in the introduction, which are supported by empirical results in the Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Section and the Ethics Statement.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificatton, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our study builds on top of empirical experiments without theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We elaborate the pretraining and finetuning algorithm in Section 2 with opti- mization and evaluation details in Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
In terms of data: MADLAD-400 is publicly available, but our noisy YouTTbbe dataset is not. However, we provide information about the crawling method, and the covered sign languages and data statistics. The results could also be reproduced directionally with YouTube-SL-225, which is a smaller cleaner dataset.
In terms code: The used (m/By)TS model checkpoints are publicly available, but the framework used to finetune them with multimodal inputs has not been open sourced, so we are unable to release our code. However, the methodology is simple and we provide the details replicate it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
http:/uuuds//oooSuui seabbbmmmssooooooccy.y for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.com httpc/ppwlc.gggde/eeeeeeocccoeee
ssionnolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification:
We provide training and test details in Section A.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to high computational cost, we run SLT pretraining and finetuning once for each setting. To make sure the evaluation reliable, we compare different methods with several metrics, including BLEU, ChrF and BLEURT.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide details for the computational resources in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our work follows the ethics guidelines of NeurIPS 2024.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
27

---

Justification:
We discussed potential societal impacts in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We didn't 't release data or models from this study.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The used data and models are properly cited in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
28

---

The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, patasets..pperrsiii wwwppppppphhaaaecmmocpp whwode com/ datasets has
curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We didn't 't release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our study doesn't 't include crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our study doesn't 't include crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "OBBOacym2e", "reviewer": "Reviewer_brKp", "summary": "The paper focuses on scaling sign language translation (SLT) by leveraging large-scale pretraining data, increasing model size, and expanding the number of translation directions. The study demonstrates the effectiveness of data/model scaling and cross-lingual cross-modal transfer in improving SLT performance across multiple benchmarks. The authors showcase substantial quality improvements in SLT through scaling, surpassing previous state-of-the-art results by wide margins.", "questions": "1 Have the authors considered the impact of different model architectures or training strategies on SLT performance, and how these factors could influence scalability and generalization?", "limitations": "This paper adequately discusses the limitations in Section 6.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1 The paper pushes the frontier of SLT through large-scale pretraining and the exploration of various data sources. This approach demonstrates a novel application of scaling techniques in the SLT domain.\n\n2 The research employs a rigorous methodology, including the use of different pretraining tasks and modalities. The thoroughness of the experimental design and validation is evident.\n\n3 The presentation is clear and well-structured, allowing for an easy understanding of the research approach and findings. The logical flow of the paper aids in comprehending complex concepts.\n\n4 The work has significant potential to advance open-domain SLT for multiple sign languages through scalable methods. The substantial quality improvements over previous state-of-the-art results highlight the impact of the research.", "weaknesses": "1 Lack of discussion on the impact of model architectures and training strategies. The study lacks an in-depth exploration of how different model architectures or training strategies could affect SLT performance, potentially limiting the understanding of scalability and generalization.\n\n2 Limited exploration of the robustness to variations in data quality and model complexity. The research does not extensively discuss the robustness of the proposed methods to changes in data quality and model complexity, which could impact the robustness of the study results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n**Re: how different model architectures or training strategies could affect SLT performance; how these factors could influence scalability and generalization?**\n\nThanks for this question! Firstly, it would be great if you could provide more context about the \"model architectures and training strategies\" you're interested in.\n\nAccording to our experiments in Table 1, Section 4.1, model architectures and training strategies have non-ignorable influence on SLT performance. We provided ablations for different T5 model families, covering T5, mT5 and ByT5, and model sizes at different scales, from Base, Large to XL. While these models follow the same encoder-decoder Transformer-based architectures, they differ significantly in the pretraining corpus (c4 vs. mc4), model parameter allocation, and the vocabularies.\n\nTwo intriguing observations from Table 1: 1) ByT5 performs generally better than T5 and mT5 across scales, even though ByT5 and mT5 used the same pretraining corpus. 2) Model scaling doesn’t necessarily lead to improved performance; in particular, we observed consistent worse performance for Large compared to Base.\n\nWe acknowledge the existence of different modeling variations, such as decoder-only Transformer (LLaMa, Gemma) and non-Transformer model (Mamba). We believe that different architectures and training strategies endow the model with different inductive biases that affect its generalization and adaptability substantially to cross-modality tasks like SLT. But exhaustively exploring how these variations affect SLT performance is beyond the scope of this study.\n\n\n**Re: the robustness of the proposed methods to changes in data quality and model complexity**\n\nThanks for this question! The results in Table 4 and the YouTube-ASL performance [1] could demonstrate the robustness of our method. Please notice that YT-ASL is a noisy superset of YouTube-ASL, which includes significantly more but lower-quality sign language data (~2800 hours vs. ~1000 hours).\n\nBased on Table 4, using YT-ASL (ID: 2) achieves a BLEURT score of 51.74 on How2Sign, substantially outperforming the YouTube-ASL result, 46.63. Adding noisy multilingual sign language data (ID: 6) further increases the performance to 53.51. \n\nWe will add a discussion on the robustness of our methods in the revised version following your suggestion!\n\n[1] Uthus et al., 2023. YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tI9DQlS3pH", "reviewer": "Reviewer_XL93", "summary": "This paper attempts to advance the development of sign language translation (SLT) technology by using large-scale pre-training data, expanding model size, and adding translation directions. Through extensive experiments, the authors have drawn many useful conclusions. Experiments show that this work can achieve the best results in multiple open-domain SLT benchmarks covering multiple sign languages, although the translation quality needs to be further improved to meet practical needs.", "questions": "see Weaknesses", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The exploration of sign language translation in large-scale open domain is very valuable. Previously, much sign language translation work was limited to some smaller fields.\n- The authors conducted a large number of experiments and obtained some useful conclusions, which can guide the future work of sign language translation.\n- Combining all the gain-enhancing methods together, the authors achieved state-of-the-art results on multiple downstream tasks of sign language translation.", "weaknesses": "- Unfortunately, the sign language data and code used in this article will not be open source. I also can't seem to find the corresponding paper about \"Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus\" for the YT-Full dataset, which makes it difficult for later generations to reproduce this work.\n  \n- As far as I know, using the original RGB sequence as input often achieves better results than simply using the pose sequence as sign language input because it contains more information. Can the author explain why the RGB sequence is not used as input?\n  \n- The author does not seem to state the specific computing resources and time spent on the experiment in section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n**Re: the sign language data and code used in this article will not be open source**\n\nPlease note that YouTube-SL-25 has been released on arXiv (https://www.arxiv.org/abs/2407.11144), including release of the clean subset of the data. We built on their work through personal correspondence with the authors before it was widely published.\n\nWe are unable to release the source code because it uses an internal framework that has not been open sourced.\n\n**Re: Can the author explain why the RGB sequence is not used as input?**\n\nThe RGB sequence was not used as input due to the different privacy considerations, as explained in Appendix A.\n\nDirectly modeling RGB sequences from video sharing sites may raise ethical issues, as highlighted in the work \"Towards Privacy-Aware Sign Language Translation at Scale\" by Rust et al., and we believe the comparison between pose and RGB sequence is out of scope for our work.\n\nIt would be an interesting question for future work to evaluate—in light of larger sign language datasets—how much of the gap between pose and RGB inputs is actually due to extra information in the RGB input vs. availability of pretrained vision encoders.\n\n\n**Re: The author does not seem to state the specific computing resources and time spent on the experiment in section 3.**\n\nAs stated in line 160, Section 3, we moved the details for computing resources and time to Appendix B.1. Specifically, we pretrained models up to 1M steps using 64/64/128 TPU-v3 chips for Base/Large/XL, which takes about 7~20 days depending on the model scale and resource condition.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aINfOgArtx", "reviewer": "Reviewer_DP5W", "summary": "This paper presents an approach to Sign Language Translation (SLT) that aims to scale the field by addressing limitations in data, model size, and the number of translation directions. The authors' key contributions are:\n\n* Data Scaling: The authors leverage diverse and large-scale datasets, including noisy multilingual YouTube SLT data and augmented SLT data generated from video captions, enabling more robust pretraining.\n* Model Scaling: They pretrain SLT models on various sizes, initially fine-tuning them with pretrained (m/By)T5 models, demonstrating the impact of larger models on performance.\n* Cross-lingual and Cross-modal Transfer: The authors show that cross-lingual and cross-modal transfer from pretraining on multilingual data improvesSLT performance, even enabling zero-shot translation, translating sign language to a spoken language without explicit training in that direction.\n* Open-domain SLT Evaluation: The models are finetuned on five downstream open-domain SLT benchmarks covering five different sign languages, leading to substantial quality improvements over existing state-of-the-art methods.", "questions": "See questions above. Especially, can you explain the missing references and talk through characterization of the FLEURS-ASL dataset? Can you also confirm that you are not fine-tuning on FLEURS-ASL#0?\n\nRegarding Table 5 and related analysis, is there a reason why BLEURT could have a much higher correlation compared to BLEU or ChrF?\n\nHow accurate are the mediapipe landmarks on messy YouTube data? Have you looked at correlations between landmark accuracy and translation quality?", "limitations": "Seems reasonable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper and approach is well motivated.\n* The experiments are generally well thought out. \n* The writing quality and analyses are generally good and informative. e.g., Sec 4.2 brought up questions that I had been wondering and did a nice job answering. \n* The results relative to SOTA are compelling.", "weaknesses": "There are multiple references that claim to be on arXiv but don't seem to exist including the FLEURS-ASL paper. I found at least three instances:\n* Garrett Tanzer. Fleurs-asl: Including american sign language in massively multilingual multi- task evaluation. arXiv, 2024.\n* Garrett Tanzer. Fingerspelling within sign language translation. arXiv, 2024.\n* Garrett Tanzer and Biao Zhang. Youtube-sl-25: A large-scale, open-domain multilingual sign language parallel corpus. arXiv, 2024.\n\nFLEURS-ASL#0 only has 353 sentences but is used for a large percentage of the experiments. I have some question about whether or not these results are likely to generalize, especially given that these sentences were generated by a single person. Given that the FLEURS-ASL paper isn't actually on arXiv (as of July 13) I don't have a way to understand what types of sentences these include. it's important to understand the biases and breadth of what is covered. \n\nLack of Comparative Analysis: While the paper surpasses the previous state-of-the-art, a more in-depth comparative analysis of other recent SLT methods, including their strengths and weaknesses, would strengthen the paper's argumentation. Yes, Table 4 does show SOTA numbers, but it would be useful to contrast the approaches. \n\nMinor: Phoenix is referred to in Table 4 and 8 descriptions but only acronyms are used in the tables themselves -- it's unclear which acronym is which.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n**Re: the missing references**\n\nThanks for pointing this out! Please note that our work is built on prior work discussed through personal correspondence with the authors before publication (hence our use of FLEURS-ASL#0 only, which was available to us before the full dataset was complete). YouTube-SL-25 has since been published to arxiv (https://www.arxiv.org/abs/2407.11144). According to our communication with the authors, other papers should be released publicly very soon, including the FLEURS-ASL dataset.\n\n**Re:  characterization of the FLEURS-ASL dataset? Can you also confirm that you are not fine-tuning on FLEURS-ASL#0?**\n\nFLEURS-ASL#0 is a translation of 353 sentences from FLORES/FLEURS devtest set (https://github.com/facebookresearch/flores/tree/main/flores200) from English text to ASL by a Certified Deaf Interpreter, with at least 5-6 hours of preparation for each 1 hour of recorded content. Note the translations are performed in groups of sentences, not in isolation, since FLORES is constructed from documents. In this way, FLEURS-ASL represents itself as the first massively multilingual SLT evaluation benchmark, including American Sign Language and 200 spoken target languages; we evaluated the translation from ASL to 42 spoken languages in this study. Human performance on a random subset of this data (measured with a native Deaf ASL signer with credentials in sign language education) is about 14.8 BLEU / 63.4 BLEURT.\n\nRegarding model finetuning, we never finetuned our models on FLEURS-ASL#0: all the reported results are for pretrained models only. \n\n**Re: a more in-depth comparative analysis of other recent SLT methods, including their strengths and weaknesses, would strengthen the paper's argumentation.**\n\nWe’d like to highlight that the main focus of this paper is to understand and explore how pretraining data, model size, and sign language scaling improves SLT, which to the best of our knowledge has rarely been studied in the literature. While there are many other intriguing SLT methods recently, most of them focus on advanced modeling and/or application of large language models, which is beyond the scope of our study. Also note we didn’t make any claims that our method is better than the others.\n\n**Re: Phoenix is referred to in Table 4 and 8 descriptions**\n\nThis is a typo and we will fix it in our next version.\n\n**Re: is there a reason why BLEURT could have a much higher correlation compared to BLEU or ChrF?**\n\nWe hypothesize that BLEURT measures semantic similarity which captures something beyond simple exact n-gram match as in BLEU/ChrF. For example, the model often starts learning SLT by outputting some key words/phrases correctly. Such weak signals can hardly be captured by BLEU/ChrF.\n\n**Re: How accurate are the mediapipe landmarks on messy YouTube data?**\n\nThis is a great question! Our simple evaluation by eye-balling some landmark examples showed acceptable results. But we believe the landmarks generated from mediapipe include noises with varying degrees for YouTube data, particularly when there are multiple signers in the video. Still, our scaling method could capture useful signals from these data. We leave the study of how landmarks accuracy affects translation quality to the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjKHQysluI", "reviewer": "Reviewer_jVKg", "summary": "This paper proposes to improve the open-domain sign language translation by scaling pretraining data, model size, and number of translation directions. The proposed approach involves pretraining with a mixture of noisy multilingual YouTube SLT data, parallel corpora, and SLT data augmented with MT models. Experiments based on (m/By)T5 models show substantial improvements over the baselines on several benchmarks, surpassing the previous state-of-the-art (SOTA) by wide margins.", "questions": "In the zero-shot ASL-to-X translation, does the model encounter similar issues seen in zero-shot text translation, such as translations in wrong languages?", "limitations": "The author does not release the code or model.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and provides many insights. The problem is that improving the sign language translation performance in open-domain settings is important. Scaling model size, number of languages, and data size lead to impressive performance based on (m/By)T5 models.", "weaknesses": "1. In Figure 7, the legend is blocked.\n2. There is no open-source code or model, and reproducing this work requires substantial resource expenditure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\n\n**Re: In Figure 7, the legend is blocked.**\n\n\nWe will fix it in the next version.\n\n\n**Re: There is no open-source code or model**\n\nWhile we can’t release the code and model subject to policy restriction, we believe our findings and scaling results could shed light on the development of SLT research and inspire more follow-up studies. \n\n**Re: does the model encounter similar issues seen in zero-shot text translation, such as translations in wrong languages?**\n\nThis is a great question! We checked the translations in zero-shot directions and have some interesting results.\n\nWe evaluated the language accuracy and empty rate for Figure 2 (b) as below. \n\n| Language | Baseline | w/ MT X<->En |\n|---|---|---|\n| es | 90.9 | 95.2 |\n| de | 85.6 | 93.2 |\n| fr | 92.4 | 94.1 |\n| it | 82.4 | 91.2 |\n| cs | 1.7 | 61.2 |\n| pl | 78.5 | 97.7 |\n| ru | 95.2 | 85.3 |\n| zh | 45.0 | 15.0 |\n| ar | 72.2 | 75.4 |\n| ja | 76.5 | 73.7 |\n| hi | 42.2 | 4.8 |\n\n**Language Accuracy:** the accuracy of translations in the correct target language. Higher indicates better. \n\n| Language | Baseline | w/ MT X<->En |\n|---|---|---|\n| es | 8.2 | 2.0 |\n| de | 4.0 | 0.0 |\n| fr | 6.5 | 0.6 |\n| it | 5.1 | 0.6 |\n| cs | 7.6 | 0.6 |\n| pl | 2.3 | 0.0 |\n| ru | 0.0 | 7.4 |\n| zh | 2.8 | 17.3 |\n| ar | 5.1 | 18.7 |\n| ja | 0.3 | 0.3 |\n| hi | 1.4 | 43.3 |\n\n**Empty Rate:** the proportion of outputting empty translations. \n\nWe noticed that zero-shot SLT translation also suffers from off-target translation, particularly for those languages distant from English: for example Baseline only has a language accuracy of 1.7, 45.0 and 42.2 for Cs, Zh and Hi, respectively.\n\nAdding parallel translation data generally improves the translation language accuracy, such as 1.7/78.5 to 61.2/97.7 for Cs/Pl. But there are also exceptions, like Zh and Hi, where the accuracy reduces from 45.0 and 42.2 to 15.0 and 4.8. A deeper inspection reveals that jointly training with translation data leads to more empty outputs for these languages: the empty rate increases from 2.8/1.4 to 17.3/43.3 for Zh/Hi. We argue that this may be because 1) these languages have significantly less parallel MT data, e.g. Hi only has 1.2M examples, and 2) the parallel corpus from MADLAD-400 can also be quite noisy.\n\nWe will add these results in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper offers a comprehensive study of how to improve sign language translation by scaling up training in a variety of ways. Interesting recipes are offered such as using data from a noisy multilingual YT dataset, parallel text copora, and data augmentation with translation models. The reviews for the paper were quite positive, noting that it pushes the frontier of SLT. There was some concern about the unavailability of the data, which the authors addressed by providing an arXiv link to the paper that introduces YouTube-SL-25 (hopefully they will remember to update the paper appropriately, if required). While the code for the paper will not be available, it was felt that the methods contributed would still lead to useful research in the community on this specific problem.
Decision: Accept (poster)