JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing
Authors: Shester Gueuwou
Sophie Siake
Colin Leong
Mathias Müller
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Translation
Sign Languages
Dataset
Multilinguality
TL;DR: A unique highly multilingual dataset for sign languages processing with baseline experiments.

Abstract: Advancements in sign language processing have been hindered by a lack of sufficient data, impeding progress in recognition, translation, and production tasks. The absence of comprehensive sign language datasets across the world's sign languages has widened the gap in this field, resulting in a few sign languages being studied more than others, making this research area extremely skewed mostly towards sign languages from high-income countries. In this work we introduce a new large and highly multilingual dataset for sign language translation: JWSign. The dataset consists of 2,530 hours of Bible translations in 98 sign languages, featuring more than 1,500 individual signers. On this dataset, we report neural machine translation experiments. Apart from bilingual baseline systems, we also train multilingual systems, including some that take into account the typological relatedness of signed or spoken languages. Our experiments highlight that multilingual systems are superior to bilingual baselines, and that in higher-resource scenarios, clustering language pairs that are related improves translation quality.

JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing
Shester Gueuwou', Sophie Siake", Colin Leong", Mathias Müller
Kwame Nkrumah University of Science and Technology, Ghana
EFFEI Paris, France
University of Dayton, USA
4 University of Zurich, Switzerland
Abstract
community to do more research on low resource spoken languages (Ranathunga et al., 2023). This issue is exacerbated by the fact that approximately 80% of people with disabling hearing loss the world reside in middle and low-income countries (World Health Organization, 2023).
Advancements in sign language processing have been hindered by a lack of sufficient data, impeding progress in recognition, translation, and production tasks. The absence of com- prehensive sign language datasets across the world's sign languages has widened the gap in this field, resulting in a few sign languages being studied more than others, making this re- search area extremely skewed mostly towards sign languages from high-incoms countries. In this work we introduce new large and highly multilingual dataset for sign language trans- lation: JWiig. The dataset consists of 2,530 hours of Bible translations in 98 sign languages, featuring more than 1,500 individual signers. On this dataset, we report neural machine trans- lation experiments, Apart from bilingual base- line systems, we also train multilingual sys- tems, including some that take into account the typological relatedness of signed or spo- ken languages. Our experiments highlight that multilingual systems superior bilingual baselines, and that in higher-resource scenar- ios, clustering language pairs that are related improves translation quality.
Our first contribution towards addressing these challenges is to present JWSign, a highly mul- tilingual corpus of Bible translations in 98 sign languages, made accessible through an automated loader. To the best of our knowledge, JWSign is one of the largest and most diverse datasets to date in sign language processing ($3).
There is precedent in natural language process- ing (NLP) for using Bible translations as starting point for many under-resourced languages that may not have any parallel resources in other domains. Bible corpora have played major role in research in speech and text areas of NLP ($2).
We complement the JWSign dataset with base- line experiments on machine translation, training a Transformer-basee system for 36 bilingual pairs of languages in the dataset. Such bilingual systems, trained individually for each language pair (one sign language and one spoken language), are the default procedure in recent literature.
1 Introduction
There are around 300 sign languages recorded up to date (United Nations, 2021). However, sign language translation research is extremely skewed towards limited number of sign languages, primar- ily those from high- high-income countries (Müller et al., 2023), while ignoring the vast majority of sign lan- guages used in low and middle-income countries (Gueuwou al., 2023). A similar phenomenon was observed in the spoken' languages machine translation community and was shown by Ögún- remí et al. (2023) to be harmful, calling on the NLP
However, sign language translation (SLT) has proven to be challenging task, due to obstacles such as very limited amounts of data, variation among individual signers and sub-optimal tokeniza- tion methods for sign language videos. potential way to improve over bilingual systems (the predom- inant kind at the time of writing) is to build multi- lingual systems. Linguistic studies have suggested good level of similarity and mutual intelligibility among some sign languages (Power et al., 2020; Reagan, 2021) even from different continents (e.g. Ghanaian Sign Language in Africa and American Sign Language in North America). In this work, we therefore explore different multilingual settings for sign language translation ($4).
'In this work, following Müller et al. (2022a). we "use the word 'spoken' to refer to any language that is not signed, no matter whether is represented as text or audio, and no matter whether the discourse is formal (c.g. writing) or informal (e.g. dialogue)".

---

2 Related Work
from the German TV station PHOENIX, covering recordings from 2009 2013 (Camgoz et al., 2018; Yin and Read, 2020; De Coster et al., 2021; Zhou et al., 2021; Voskou et al., 2021; Chen et al., 2022b). However, Müller et al. (2023) called into question the scientific value of this dataset. In recent times, TV broadcast datasets have been introduced for several sign languages, including SWISSTXT and VRT (Camgöz et al., 2021), and the BBC-Oxford British Sign Language (BOBSL) dataset (Albanic et al., 2021) for Swiss-Germar Sign Language, Flemish Sign Language and British Sign Language respectively. Other examples are the How2Sign dataset (Duarte et al., 2021), OpenASL (Shi et al., 2022a) and YouTube (Uthus et al., 2023), fea- turing American Sign Language.
The following section explores the motivation be- hind the research by examining works that have utilized the Bible in various modalities ($2.1). It delves into the limited coverage of many sign lan- guages within popular existing datasets for sign language translation ($2.2), and provides a com- prehensive overview of the state-of-the-arr meth- ods employed for automatic translation of sign lan- guage videos into text ($2.3).
2.1 Use of Bible corpora in NLP
Previous studies have acknowledged the Bible as a valuable resource for language exploration and pro- cessing (Mayer and Cysouw, 2014) with good lin- guistic breadth and depth (Resnik et al., 1999). In machine translation, Bible translations have proven be a good starting point for machine translation research of many spoken languages, even if eventu- ally one must move to other more useful domains (Liu et al., 2021). In effect, Bible translations have shown their usefulness across different modalities including text (McCarthy et al., 2020) and audio (Black, 2019; Pratap et al., 2023), and for many low resource spoken languages especially in Africa (Dossou and Emezue, 2020; Adelani et al., 2022; Meyer et al., 2022).
All datasets mentioned above are bilingual i.e. they contain one single sign language, paired to one spoken language. However, some multilingual datasets have emerged very recently as SP-10 (Yin et al., 2022) and AfriSign (Gueuwou et al., 2023). SP-10 features 10 sign languages but sentences here are extremely short in general (e.g "How are you ?"). AfriSign comprises 6 sign languages which are actually a subset of JWSign. In contrast, JWSign is a valuable resource that surpasses most other sign language translation datasets in terms of duration, signers diversity, and coverage over different sign languages as highlighted in Table 1. Thus, aim for JWSign serve a foundational resource make sign language translation research more diverse and inclusive going forward.
Mayer and Cysouw (2014) showeased a corpus of 847 Bibles and McCarthy et al. (2020) increased it significantly both in terms of the number transla- tions (4,000 unique Bible translations) and number of languages (from 1,169 languages to 1,600 lan- guages) it supported thus forming the Johns Hop- kins University Bible Corpus (JHUBC). In the au- dio domain, the CMU Wilderness Speech Dataset (Black, 2019) is a notable resource derived from the New Testaments available on the www.bible.is website. This dataset provides aligned sentence- length text and audio from around 699 different languages. In a similar effort, Meyer et al. (2022) formed TTS: speech corpus on high-quality Bible translations of 10 African languages. Pratap al. (2023) expanded both these works and formed the MMb-lab dataset containing Bible translations in 1,107 languages.
2.3 Sign Language Translation Methods
SLT is an emerging field which aims to translate sign language videos to text/speech and/or vice- versa. One of the main challenges in SLT is finding an efficient and nigh-quality representation for sign language. This has resulted in many translation architectures using tokenization methods such as human keypoint estimation (Ko et al., 2019; Kim et al., 2020), CNN feature extraction (Zhou et al., 2021; De Coster al., 2021; Voskou et al., 2021), linguistic glosses (Müller al., 2023) or phonetic systems such as SignWriting (Jiang et al., 2023). However, most of these are frame-level tokeniza- tion methods and assume implicitly that sign lan- guage utterances can be considered as sequences of lexical units, while in reality signing uses complex structures in time and 3-dimensional space.
2.2 Sign Language Translation Datasets
Previous studies on sign language translation were predominantly relying on the RWTH-Phoenix 2014T dataset (Camgoz et al., 2018), which contains 11 hours of weather broadcast footage
All things considered, 3D CNN wendow-level

---

Dataset
#SL(s)
Vocab
#Hours
Avg
#Signers Source
PHOENIX (Camgoz et al., 2018)
3K
11
4.5
9
TV
KETI (Ko et al., 2019)
419
28
6.9
14
lab
CSL-Daily (Zhou et al., 2021)
2K
23
4.0
10
lab
BOBSL (Albanie et al., 2021)
78K
1467
4.4
39
TV
How2Sign (Duarte et al., 2021)
16K
80
8.2
11
lab
OpenASL (Shi al., 2022a)
33K
280
10.5
R=220
web
YouTubeASL (Uthus et al., 2023)
60K
984
4.8
>2519
web
SP-10 (Yin et al., 2022)
10
17K
14
4.3
79
web
AfriSign (Gueuwou et al., 2023)
20K
152
18.3
160
web
JWSign
98
729K
2530
19.3
2J 15500
web
Table 1: Comparing the JWSign dataset to other common datasets in SLT research. Vocab Vocabulary of target spoken language i.c. number unique spoken words, PHOENIX RWTH Phoenix-2014T7 #SL = number of sign language pair(s), #Hours Total duration of the dataset in hours, Avg Average duration of sample in the dataset in seconds.
feature extractors have been reported to reach the best results in this task (Chen et al., 2022a; Müller et al., 2022a; Tarrés et al., 2023). The main com- ponent of this approach is an inflated 3D convo- lutional neural network (I3D) (Carreira and Zis- serman, 2017) or S3D (Wei et al., 2016). Origi- nally designed for action recognition (Kay et al., 2017), some works (Varol al., 2021; Duarte et al., 2022: Chen et al., 2022a) have adapted and fine- tuned these networks for sign language recognition datasets such as BSL-1K (Albanie al., 2020) and WLASL (Li et al., 2020).
BAACK
3 JWSign
In this section, we give an overview of JWSign ($3.1) and list key statisties, comparing JWSign to other recent datasets ($3.2). Finally, we explain our process of creating fixed data splits for (multilin- gual) machine translation experiments ($3.3).
Figure Anecdotal signer diversity JWSign.
graphics, making it unique and valuable resource for researchers and practitioners alike.
Translators are either deaf themselves or have grown up in deaf communities, and the recordings made in studio on-the-grount in each country. Translations are not only out of English, different spoken languages are used the source depending on the country. Details about the trans- lation process at JW are included Appendix A.
3.1 Overview
JWSign is made up of verse-aligned Bible trans- lations in 98 sign languages from the Jehovah's Witnesses (JW) w This wide coverage also extends to the racial identities of the signers, with representation from American Indians/Alaska Natives, Asians, Blacks/Affrican Americans, His- panics/Latinos, Native Hawaiians/Other Pacific Is- landers, and Whites (in alphabetic order) (illus- trated in in Figure 1). Therefore, we believe that JWSign captures broad range of signer demo-
material,
3.2 Statistics of JWSign
JW features 98 sign languages spread across all the continents of the world (Table 2). All languages taken together, it has a total duration 2,530 hours and contains roughly 1,500 individual
Phttps: //wp::///wwww.... jww.w//.org///.o..

---

Europe
31
Asia
21
Number of Samples
25000
Africa
19
North America
11
South America
11
15000
Oceania
4
10000
Australia
1
5000
mis
csg
g5g
fcs
nsi
rms
sfb
Isl
tsm
sqs
sgn-CM
Table 2: Number of sign languages in JWSign per con- timent.
Language Pairs
Figure 2: Number of samples per language pair. The x-axis shows language pairs referred to by the ISO code for the sign languages only. All ISO codes are listed in Appendix E. While there 98 language pairs in total, we show a representative sample of 11, ranging from high-resource to low-resource.
people, according to our automatic analysis.
Comparison to similar datasets We show comparison of JWSign other datasets in Table 1. JWSign contains more sign languages, covering more geographic regions, than any other dataset we are aware of. For instance, SP-10 (Yin et al., 2022) features 10 sign languages mostly from Europe. and AfriSign (Gueuwou et al., 2023) has 6 sign languages from Africa. JW has higher signer diversity ($3.1) than most other datasets. We also observe that samples in other datasets generally shorter than the average duration in JWSign.
in varying lengths across different sign languages. We envision that JWSign enables linguistic studies such as these across many sign languages.
Cross-lingual frequency To measure the extent of sample overlaps across different sign languages, we measure how many times each sample (Bible verse) appears across all sign languages. The dis- tribution of this analysis is illustrated in Figure 4.
the other hand, we emphasize that JWSign a corpus of Bible translations only, hence covering a limited linguistic domain. Other datasets such as BOBSL and YouTubeASL are far more broad, cov- ering many domains and genres. Similarly, when comparing the amount of data available for in- dividual language pair, JWSign does not always offer the most data. For certain high-resource lan- guage pairs, other datasets are considerably larger. For example, BOBSI. and YouTube contain 1=,500 hours and = 1,000 hours of content in En- glish and British Sign language and American Sign language respectively. Nevertheless, for many lan- guage pairs, JWSign is an unparalleled resource for training and evaluating sign language translation models.
Number of individuals To determine the number of individuals in the dataset, we adopt the signer clustering approach proposed by Pal et al. (2023). We utilize the face recognition toolbox² to obtain [28-dimensional embedding for the signer in each video sample. Then, we use the Density Based Spatial Clustering of Application with Noise (DB- SCAN) algorithm (Ester et al., 1996) with 0.2 to cluster all embeddings of each sign language. This clustering method is based on the reasonable assumption that no signer can appear in videos for two different sign languages, given that videos recorded on-the-ground in each country. This yielded grand total of I 1,460 signers
Per-language statistics JWSign contains least 2,000 samples for 47 language pairs. The distri- bution of samples per language pair indicates that some languages are represented better than others (Figure 2). A similar trend is observed with the total duration per language pair (Appendix B).
3.3 Data splits and automated loader
For each language pair JWSign we provide a fixed, reproducible split into training, development and test data, tailored towards machine translation as the main use case.
Naturally, sign languages present variation in average sample duration across different sign lan- guages, depicted Figure This observation sheds light on the linguistic "verbosity" sign lan- guages, where the same sentence may be signed
Pt pnttps/// /ttp:p/:ggii //g thub. com/ agei tgey/face_ recogni tion "We realised this clustering approach works non-optimally for Black/Africas American and Asian signers and much bet- ter on other races. So we expect the ground-truth number of signers in JWSign to be above this value.

---

Average duration samples (In secondi)
40
Cross lingua frequency
80
30
25
40
20
20
15-
0
mts
csg
gsg
fcs
risi
ms
sfo
Isl
tsm
sqs
sgn- -gn-CM
5000 10000
25000
30000
Samples
Language Pairs
Figure 4: Cross-lingual frequency of Bible verses in JWSign. The y-axis shows the number of sign languages each verse is translated to.
Figure 3: Average duration (in seconds) per language pair. It is worth noting that the two outliers sign lan- guages that exhibit significant deviations from norm were observed to be those with very small sample size (less than 10) and long sentences, and are therefore not sufficiently representative of those sign languages.
creation of this automated loader? can be found in Appendix D.
Experimental setup
Splitting procedure Our method for splitting the data into training, development and test sets is designed to eliminate multilingual "cross- contamination" (the same sentence in two different languages appearing both in the train and test set) as much as possible. Multi-way parallel corpora such as the IWSLT 2017 multilingual task data (Cettolo et al., 2017) (where cross-cos cross-contamination does exist) are known to paint an overly optimistic picture about the translation quality that can realis- tically be obtained. A second goal is to maintain reasonable test size for machine translation.
We perform preliminary machine translation exper- iments on the JWSign dataset. In this section we explain our preprocessing steps ($4.1), how differ- ent models are trained ($4.2) and our method of automatic evaluation ($4.3).
4.1 Preprocessing
Sign language (video) data All videos have a resolution of 1280 x 720 and frame rate of 29.97 fps. We first resize the videos to smaller resolu- tion of 256 256 pixels and then apply center crop to the dimensions of 224 224 pixels (the input size expected by the subsequent step). Lastly, we apply color normalization.
We select development and test data based on an analysis of cross-lingual frequency (Figure 4). We minimize the chances sample in the test set in one sign language being found in the train set in another language, which could lead to cross- contamination when training multilingual model and possibly inflate the test set evaluation scores. More details on the splitting procedure given in Appendix C.
The preprocessed videos are then fed into pre- trained I3D model for feature extraction. We use a window size of 64 and temporal stride of 8, and the particular I3D model we use was fine-tuned by Varol et al. (2021) BSL- dataset (Albanie et al., 2020), encompass- ing over 5,000 sign classes.
expanded version of the
Automated loader We do not create new videos nor upload and store them. 5 Instead, JW con- sists of links to Bible verses on the JW 5 itself and support it with automated loader integrated in the Sign Language Datasets library (Moryossef and Müller, 2021) for better accessibil- ity and reproducibilty. More information about the
Using a fine-tuned I3D model, or more generally, vision-based approach, for feature extraction is motivated by earlier findings. For example, Müller et al. (2022a) and Tarrés et al. (2023) point out that vision-basee based approaches outperform alternatives such as feature extraction with pose estimation.
What more, Shi et al. (2022b) have shown
Hossting Jehovah's 's Witnesses data publicly is not allowed: ht tps: /wwww//wwwwwoo jw. org/en//////oeeeoooooooooo terms-of-use/
proceesiin/ http://tteeeaatteeeeeeeee wtt.eeessiinnnnguaeeeeee latatageeaatttsstsss datasets/ datasets/ju_sign
"https:///ithhue
"https: /www. jw. jw..wr/......................... www./r//////ooooo/ooo htt./oonineeeeppl
jw- ww-library-sie sign- language/

---

that a pretrained 13D model fine-tuned on a sign language corpus with greater diversity of signing categories yields more substantial benefits for sign language translation compared to a corpus with fewer signs.
coverage to 0.995 and expanded the vocabulary size to 1,500 to accommodate the larger character set. Training was done for a maximum of 100,000 updates.
We extract embeddings before the final classifica- tion layer. specifically the "mixed_Se" layer. These embeddings are 1024-dimensional vectors that are 102i-dimenlional
Multilingual systems ("M" systems) We ex- plore three different multilingual settings. First, we train a single multilingual model using the 36 highest-resource language pairs (same as for B36 above) (M36). This enables us to compare the ef- feet of training various language pairs separately and jointly. To optimize the training of multilingual models, we employ larger batch size of 128 and slightly increase the initial learning rate to le-06.
stacked together, forming vector for each sample video, where w the total number of windows in sample video.
Finally, for multilingual systems only, a 1024- dimensional vector representing the target spoken language is further appended to the extracted em- bedding stack. This particular vector serves as continuous analogue of a tag to indicate the associ- ated target spoken language (Johnson et al., 2017) and it unique for every spoken language.
We then attempt a naive model trained on all lan- guage pairs in JW wign that have training samples (M91). This amounts to 91 different language pairs, excluding seven language pairs in the Zero category which do not have any training data. We use these zero-resource language pairs only for testing. Since we anticipate that the naive multilingual strategy of M91 leads to low translation quality for the low-resource language pairs, we further ex- plore fine-tuning strategy. For this system (MFT). fine-tune the M36 model jointly on all lower- resource language pairs with training data (i.e. all training data that M36 was not trained on, 55 lan- guage pairs in total). All hyperparameters are kept the same except that we reset the optimizer accu- mulator and restart from the 0-th step. This allows us to examine cross- looss-lingual transfer from higher- resource to lower-resource languages. By training diverse range of language pairs, we can sess the model's ability to generalize and adapt unseen sign languages having very little data.
Spoken language (text) data We remove special noisy characters as * and + The resulting preprocessed text is then tokenized using a Senten- cepiece model (Kudo and Richardson, 2018).
4.2 Types of models that are compared
this paper we work exclusively on signed-to- spoken translation, translating from a sign language to a spoken language in all cases. Our models are Transformers (Vaswani et al., 2017) with 6 encoder and 3 decoder layers. Our code is based Fairseq Sign-to-Text (Tarrés et al., 2023) and is publicly available All experiments were conducted on single NVIDIA-A100 GPU.
Bilingual ("B" systems) We developed 36 bilin- gual models (referred to as B36), each focusing on a specific language pair i.e. sign language to spo- ken language. These language pairs were carefully chosen based on having a substantial number of samples (greater than 1,000 samples) in their re- spective training sets. We trained the models with batch size of 32, a learning rate of le-3 and applied a dropout rate of 0.3.
Clustered families ("C" systems) Finally, as an- other attempt at improving over naive multilingual training, leverage the phylogeny of spoken lan- guages and sign languages. We cluster the lan- guage pairs based on source sign language families (Power et al., 2020; Eberhard et al., 2023) and train on each cluster separately (CSIG). Similarly, we cluster the language pairs according to the target spoken language families (Fan et al., 2021) and train on each cluster separately as well (CSPO).
We set the Sentencepiece vocabulary size to 1,000 for most language pairs, except for those with limited number of samples (less than 10,000 in total), where we use vocabulary size of 500. For languages with very wide range of characters, such Chinese and Japanese, characters are appearing only once (hapax legom- ena). To counteract this we reduced the character
A sample of each cluster can be found in Table 3 and the full list of clusters is given in Table 6 and Table 7 in the Appendix. The intuition for these clustering experiments is to invoke positive transfer effects stemming from similarities between languages.
observed many
chttps://gittee http://gtttuuucccccseeeeeeeeeeee com/ShesterG/ JWSign-Machine -nnas---achiin---raltaaiiiioo Translation

---

Group
Argentinean (aed), Austrian (asq), Bel- gian French (sfb), Dutch (dse), Flemish also provided detailed non-aggregate results in Ap- pendix G.
Spoken Languages
Germanic
Danish (da), Dutch (nl), English (en),
German (de), Norwegian (no), Swedish (sv)
Performance Variation Across Language Pairs The table categorizes the language pairs into differ- ent groups based resource availability, namely High. Medium, Low, Very Low, and Zero. By exam- ining the performance metrics (BLEU, BLEURT, chrF) within each group, we can observe trends model performance. For example, the High and Medium groups tend to have higher scores com- pared to the Low, Very Low, and Zero groups. This suggests that having larger training dataset, indicated by the resource availability, positively impacts the translation quality.
Group
Sign Languages
Old French
(vgt), French (fsl), German (gsg), Greek (gss), Irish (isg), Israeli (isr). Italian (ise), Mexican (mfs), Quebec (fes), Spanish (ssp), Swiss German (sgg). Venezuelan (ysl)
Table Examples for clustering into language families, showing sign language cluster and spoken language cluster.
Going into more individual bilingual pair results (Table 8 in Appendix G), the highest BLEU was obtained by Japanese Sign Language to Japanese text (7.08), American Sign Language to English text (4.16) and Chinese Sign Language to Chinese (3.96). This suggests some language pairs are eas- ier for a model to learn, for instance because the grammar of Japanese sign language may be more aligned with spoken Japanese, compared to other language pairs.
4.3 Evaluation
During training, evaluate models every 2 epochs and select the checkpoint with the highest BLEU score (Papineni al., 2002) computed with Sacre- BLEU (Post, 2018), aggregated across languages for multilingual models. At test time, using a beam search of size 5, we evaluate all models on the deto- kenized text using BLEU computed with Sacre- BLEU, BLEURT-20 (Pu et al., 2021), and chrF (Popović, 2015). We note that many recent neural metrics, such as COMET (Rei al., 2020), not applicable in our case because the source languages (sign languages) are not supported.
Impact of multilingual training Here we com- pare the performance of the "B36" model (bilin- gual training on 36 language pairs separately) and the "M36" model (multilingual training of one model on the same 36 language pairs together). We observe that our evaluation metrics show con- flicting trends, since multilingual training gener- ally reduces BLEU and chrF scores, but increases BLEURT scores. Based on evidence presented in Koemi et (2021) and Freitag et al. (2022), we adopt the view that the neural metric BLEURT is more trustworthy than BLEU and chrF, in the sense of having higher agreement with human judgement. With this interpretation in mind, our results suggest that training on multiple languages simultancously increases translation quality.
5 Results and Discussion
Although all language pairs in this work are consid- ered very low-resoureed when compared to spoken languages (Goyal et al., 2022), going forward use the term High refer language pairs with more than 10,000 training samples, Medium for lan- guage pairs with training samples between 1,000 and 9,999 inclusive, Low for language pairs with training samples between 500 and 999 inclusive, Very Low for language pairs with training samples less than 500 and Zero for language pairs with no training samples.
When Low, Very Low and Zero resource lan- guage pairs are added to the multilingual training (M91), there is a light drop in scores for High and Medium language pairs when compared to when they were solely trained together (M36). Thus, while this method may offer advantages for low- resource languages by leveraging knowledge from language pairs with much more data resulting positive transfer, there is a trade-off between trans- fer and interference, as increasing the number of
Our main findings are summarized in Table 4. Due to limited computational resources, we con- ducted single runs for all reported results. To give a better overview over our individual results, for some systems on 98 different test sets, we show results aggregated into different training data sizes. For a comprehensive understanding, we have
"BLEUU+c.iixcd+ 14s.cxp+tok. 1.

---

Mided
High
Medium
Low
Very Low
Zcro
Averaage
BI.FL
BLEURT
thrF
BUEU
BURURT
theF
BUEU
BLRURT
chri
BEEU
BLEURT
chri
BERU
BLEURT
chrif
BURU
BLEURT
chri
2.37
15.87
16.07
M33
1.6
26,91
14.07
1.38
26.05
13002
1,45
26.73
13.57
1.59
26,58
13.76
12:83
30.37
12.73
MIF]
0.33
16.18
13:19
0.61
22.70
13:41
1.37
22.83
16.66
1.48
21.2
15.81
1.18
22:12
14,16
1.12
22.02
14.88
CSIG
2.37
26.04
15:35
27.28
14.98
22.42
20.47
20.49
7.24
1.44
23.01
11.34
CSPO
2,01
27.13
14.69
1.38
28.13
15.32
1.18
26.7
1284
0.61
21.7
10.29
0.91
22:75
11.27
1,16
24.25
12.34
Table 4: Evaluation results. B36- = Bilingual Training on 36 language pairs separately, M36 Multilingual Training the same 36 language pairs as B36 but joindly MFT Fine-tuning of the M36 models on all the remaining 55 language pairs available with available training data, M91 = Joint multilingual training on all the 91 language pairs that have any training data, CSIG = Results the clustered multilingual models when the source sign languages are from the same group, CSPO Results of the clustered multilingual models when the target spoken languages are from the same group.
languages in the training set can lead to a decline in performance for the High and Medium resource language pairs (Arivazhagan et al., 2019).
guage processing research which has been so far dominated by few sign languages. We conducted a series of baseline experiments using JWSign to at- tempt improving the scores of automatic sign trans- lation systems in different scenarios. We demon- strate that multilingual training leads to better trans- lation quality compared to bilingual baselines. On the other hand, our experiments did not show clear benefit for fine-tuning approach in lower- resource scenarios. Similarly, found that cluster- ing data by language family, even though intuitively promising, only beneficial higher-resource set- tings.
Fine-tuning additional language pairs The "MFT" model represents the fine-tuning of the "M36" model on all the remaining 55 language pairs with training data available. Comparing its performance with "M36" and "M91", we observe a marked drop in BLEURT scores across all re- source categories. On average, "M91" leads to a BLEURT score of 27.32, "MFT" achieves 22.62 on average. This suggests that for incorporating new additional language pairs with limited resources, training these languages from scratch mixed with High and Medium language pairs is better than fine-tuning approach.
More generally, the overall translation quality is still very low. This is in line with other recent stud- ies such as Müller al. (2022a) who report BLEU scores in similar range. Regardless, we firmly believe that as we strive to improve translation sys- tems, it is crucial to ensure carly diversification of the sign languages used to train these systems. By incorporating wide range of sign languages during the training phase, we can enhance the inclu- sivity and effectiveness of the resulting translation systems.
Clustered multilingual models "CSIG" and "CSPO" models represent the results of clustered multilingual models. In the "CSIG" model, the source sign languages are from the same group, while the "CSPO" model, the target spoken lan- guages are from the same group. In higher-resource settings, clustered multilingual models perform bet- ter than non-clustered multilingual models (M36, MFT, M91). For High and Medium resource lan- guage pairs, CSPO leads to the best BLEURT scores among all model types. For lower-resource settings, the opposite is true and clustering lan- guages based on linguistic philogeny hurts transla- tion quality as measured by BLEURT.
As part of our future research, we aim to de- velop enhanced models utilizing JWSign that can effectively handle multiple sign languages. Further- more, JWSign presents distinctive opportunity to address the existing gaps in sign language process- ing, such the development of sign language identification tool. JWSign can also serve a valuable tool for linguists to explore and compare various sign languages in an attempt to gain more insights, such as further inquiries into the typologi- cal relatedness of sign languages.
6 Conclusion and Future Work
this study, we introduced JWSign as a unique resource aimed at promoting diversity in sign lan-

---

Limitations
Ethics Statement
There are several limitations to this study that need to be considered.
Licensing We do not in any way claim ownership of the JW data. We do not create new videos, nor upload or store them. The data strictly and entirely belongs to JW. Instead, we provide links to Bible verses on the JW website itself and we support this with automated loader increase accessibility. To the best of our knowledge. we believe that this usage in accordance with the JW.org terms of use 10, which explicitly allow the distribution of links, as well as downloads/usage of media for "personal and noncommercial purposes". As we neither upload nor copy the actual data, and our aim is to enable researchers to do noncommercial research, we believe these terms are satisfied. Nevertheless, we have also taken the step of questing explicit permission by contacting JW's legal branches in the USA and Switzerland (the Office of the General Counsel in New York and the Rechtsabteilung" in Thun, Switzerland). However, we have not yet received a reply at the time of writ- ing, Should this permission be refused we certainly plan to abide by their wishes.
Dataset size Although JWSign is one of the largest dataset that was designed for Sign Lan- guage Translation, is still quite low-resourced when compared to data other modalities such as text and/or speech.
Limited domain One limitation of the JWSign dataset is that it is focused on the domain of biblical texts, which may not be representative of other types of sign language communication. This could limit the applicability of the dataset to certain types of sign language translation tasks.
Translationese effects Another limitation of the dataset is the presence of translationese effects, which can occur when translated text speech sounds unnatural or stilted compared to the original (Barrault al., 2019). This can be challenge for sign language translation systems, which must accurately convey the meaning of the source sign language in this case while also producing natural and fluent spoken language.
Our automated dataset loader includes usage notice that explicitly informs users of JW's licens- ing terms.
Recording conditions On top, the videos in the JWSign dataset were recorded in studio setting, which may not fully capture the complexity and variability sign language communication in real- world settings. Factors such as lighting, camera angles, and the absence of background noise or visual distractions could affect the sign language production and recognition process in ways that differ from natural communication contexts. This could limit the generalizability of the dataset to real-world sign language translation scenarios.
Privacy and consent We did not reach out to all individuals depicted in our dataset (an estimated 1,500 people) to ask for their consent. We believe our research poses risk to their privacy because do distribute videos and (2) we only train models for signed-to-spoken translation. This means that it impossible to recover personal infor- mation such as faces from a trained model (which we do not share the first place).
Algorithmic bias On different note, even though JWSign has signers from all races, the dataset might suffer from other biases such as gen- der, age representation and handedness. Models trained here are far from usable and reliable, and thus cannot replace a human sign language inter- preter.
Reproducibility The dataset is not hosted al- though we circumvent this with an automated loader to increase accessibility. As long as the orig- inal videos and website remain online with stable links, dataset can be reproduced exactly.
Uni-directional models this work, we re- ported baseline scores only for signed-to-spoken translation. We did not experiment at all with trans- lation systems that generate sign language utter- ances, which is also an important research problem.
10https://ww. http://wworrg/ee//eee-uussss "httpss://ww jw. www.org/dde/rehhicchh/

---

Acknowledgements
Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal. Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference lation (WMT19). In Proceedings of the Fourth Con- ference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. As- sociation for Computational Linguisties.
We thank Antonis Anastasopoulos and Chris Emezue for feedback on the manuscript. Also, we would like to thank Google Cloud for providing access computational resources through free cloud credits. MM has received funding from the EU Horizon 2020 project EASIER (grant agree- ment number 101016982).
machine trans-
Alan W Black. 2019. Cmu wilderness multilingual speech dataset. In ICASSP 2019-2019 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5971-5975, IEEE
References
Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden, 2018. Neu- ral sign language translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7784-7793.
David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajud- deen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong. Michael Beuk- man, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andre Niyongabo Rubungo, Gilles Hacheme, Eric Peter Wairagala, Muhammad Umair Nasir, Ben- jamin Ajibade, Tunde Ajayi, Yvonne Gitau, Jade Abbott, Mohamed Ahmed, Millicent Ochieng, An- uoluwapo Aremu, Perez Ogayo, Jonathan Mukiibi, Fatoumata Ouoba Kabore, Godson Kalipe, Derguene Mbaye. Allahsera Auguste Tapo, Victoire Memd- jokam Koagne, Edwin Munkoh-Buaheng. Valen- cia Wagner, Idris Abdulmumin, Ayodele Awokoya, Happy Buzaaba, Blessing Sibanda. Andiswa Bukula, and Sam Manthalu. 2022. A few thousand transla- tions go long way! leveraging pre-trained mod- els for African news translation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3053-3070, Seattle, United States. Association for Computational Linguistics.
Necati Cihan Camgöz, Ben Saunders, Guillaume Ro- chette. Marco Giovanelli, Giacomo Inches, Robin Nachtrab-Riboet and Richard Bowden. 2021. Con- tent4all open research sign language translation datasets. In 2021 16th IEEE International Confer- ence on Automatic Face and Gesture Recognition (FG 2021). pages 1-5, IEEE.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multi-person 2d pose estimation us- ing part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 7291-7299,
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kineties dataset. In proceedings of the IEEF Conference Computer Vision and Pattern Recognition, pages 0299-6308.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Nichues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2-14, Tokyo, Japan. International Workshop on Spoken Language Translation.
Samuel Albanie, Gül Varol, Lilianc Momeni, Triantafyl- Ios Afouras, Joon Son Chung, Neil Fox, and Andrew Zisserman. 2020. Bsl-Ik: Scaling up co-articulated sign language recognition using mouthing cues. In Computer Asion-ECCV 2020: 16th European Con- ference, Glasgow; UK, August 23-28, 2020, Proceed- ings, Part XI 16, pages 35-53, Springer.
Samuel Albanic, Gül Varol, Liliane Momeni, Hannah Bull, Triantafyllos Afouras, Himel Chowdhury, Neil Fox, Bencie Woll, Rob Cooper, Andrew McParland, al. 2021. Bbe-oxford british sign language dataset. arXiv preprint arXiv:21
Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. 2022a. A simple multi- modaliy transfer learning baseline for sign language transla- tion. In Proceedings of the IEEE//VV Conference on Computer Vision and Pattern Recognition (CVPR), pages 5120-5130.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges.
Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shu- jie Liu, and Brian Mak. 2022b. Two-stream network for sign language recognition and translation. arXiv preprint 2..002221..01667.
Mathieu De Coster, Karel D'Oosterlinck. Marija Pizurica, Paloma Rabaey, Severine Verlinden, Mieke Van Herreweghe, and Joni Dambre. 2021. Frozen pre- trained transformers for neural sign language transla- tion. In 18th Biennial Machine Translation Summit
Loic Barrault, Ondřej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Gra- ham, Barry Haddow, Matthias Huck, Philipp Koehn,

---

(MT' Summit 2021). pages 88-97, Association for Machine Translation in the Americas.
Computational Linguistics: EACL 2023, pages 1706- 1724, Dubrovnik, Croatia. Association for Computa- tional Linguisties.
Bonaventure F. P. Dossou and Chris C. Emezue. 2020. Ffr v1.1: Fon-french neural machine translation.
Melvin Johnson, Mike Schuster, Quoe V. Lc, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: En- abling zero-shot translation. Transactions of the As- sociation for Computational Linguistics, 5:339-351.
Amanda Duarte. Samuel Albanie, Xavier Giró-i Nieto, and Güi Varol. 2022. Sign language video retrieval with free-form textual queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14094- 14104.
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. 2021. How2sign: A large-scale multimodal dataset for continuous ameri- can sign language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjaynnaraiimhan, Fabio Viola, Tim Green, Trevor Back. Paul Natsev, al. 2017. The kinetics human action video dataset. arXiv preprint 0etp:::7700009555
San Kim, Chang Jo Kim, Han-Mu Park, Yoonyoung Jeong, Jin Yea Jang, and Hyedong Jung. 2020. Ro- bust keypoint normalization method for korean sign language translation using transformer. In 2020 Inter- national Conference on Information and Communi- cation Technology Convergence (ICTC), pages 1303- 1305, IEEE
Eberhard, David M.. Gary F. Simons, and Charles D. Fennig. 2023, Ethnologue: Languages of the world.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Min- ing, KDD 96, page 226-231. AAAI Press.
Sang-Ki Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. 2019. Neural sign language trans- lation based on human keypoint estimation. Applied sciences, 9 (13):26833
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma. Ahmed EI- Ei-Kish Siddharth Goyal, Mandeep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multi- lingual machine translation. The Journal of Machine Learning Research, 22(1) 114489948888
Tom Koemi, Christian Federmann, Roman Grund- kiewicz, Marcin Sonczys-Dommurg Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478-494, Online. Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores 101 evaluation benchmark for low-resource and multilingual ma- chine tramslation. Transactions of the Association for Computational Linguistics, 10:522-538.
Dongxu Li, Cristian Rodriguez. Xin Yu, and Hongdong Li. 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pages 1459-1469.
Ling Liu, Zach Ryan, and Mans Hulden. 2021. The use- fulness of bibles in low-resource machine translation. In Proceedings of the Workshop on Computational Methods for Endangered Languages. volume 1. pages 50.
Shester Gueuwou, Kate Takyi Mathias Müller, Marco Stanley Nyarko, Richard Adade, and Rose- Mary Owusuaa Mensah Gyening. 2023. Afrisign: Machine translation for african sign languages. In 4th Workshop on African Natural Language Process- ing.
Camillo Lugaresi, Jiuqiang Tang. Hadon Nash Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, et al. 2019. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv:1906.0811
Zifan Jiang, Amit Moryossef, Mathias Müller, and Sarah Ebling. 2023. Machine translation between spoken languages and signed languages represented in SignWriting. In Findings of the Association for

---

Thomas Mayer and Michael Cysouw. 2014, Creat- ing a massively parallel bible corpus. Oceania, 135(273):40.
Maja Popović. 2015. chrF: character n n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.
Arya D. McCarthy, Rachel Wicks, Dylan Lewis, Aaron Mueller, Winston Wu, Oliver Adams, Garrett Nicolai, Matt Post, and David Yarowsky. 2020. The Johns Hopkins University Bible corpus: 1600+ tongues for typological exploration. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 2884-2892, Marseille, France. European Language Resources Association.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguisties.
Justin M Power, Guido W Grimm, and Johann-Matti List. 2020. Evolutionary dynamics the disper- sal of sign languages. Royal Society Open Science, 7(1):191100.
Josh Meyer, David Iffoluwa Adelani, Edresson Casanova, Alp Öktem, Daniel Whitenack Julian We- ber, Salomon Kabongo, Elizabeth Salesky, Iroro Orife. Colin Leong. Perez Ogayo, Chris Emezue, Jonathan Mukiibi, Salomey Osei, Apelete Agbolo, Victor Akinode, Bernard Opoku, Samuel Olanre- waju, Jesujoha Alabi, and Shamsuddeen Muhammad. 2022. Bibletts: a large, high-fidelity, multilingual, and uniquely african speech corpus.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoory Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang. Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. Scal- ing speech technology to 1,000+ languages.
Amit Moryossef and Mathias Müllrr 2021. Sign lan- guage datasets.
Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metries for mt. In Proceedings of EMNLP.
Mathias Müller, Zifan Jiang, Amit Moryossef, Annette Rios, and Sarah Ebling. 2023, Considerations for meaningful sign language machine translation based on glosses. In Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 682-69 693, Toronto, Canada, Association for Computational Linguistics.
Surangika Ranathunga, En En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey, ACM Computing Sarveys, 55(1 1):1-37.
Timothy Reagan. 2021. Historical linguistics and the case for sign language families. Sign Language Stud- ies, 21(4):427-455
Mathias Müller, Sarah Ebling, Eleftherios Avramidis, Alessia Battisti, Michèle Berger, Richard Bowden, Annelies Braffort, Necati Cihan Camgöz, Cristina España-Bonet, Roman Grundkiewicz, Zifan Jiang, Oscar Koller, Amit Moryossef, Regula Perrollaz. Sabine Reinhard, Annette Rios, Dimitar Shteri- onov, Sandra Sidler-Miserez, Katja Tissi, and Davy Van Landuyt. 2022a. Findings of the first wmt shared task on sign language translation (wmt-slt22). In Proceedings of the Seventh Conference Machine Translation, pages 744-772, Abu Dhabi. Association for Computational I Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Process- ing (EMNLP), pages 2685-2705 Online. Association for Computational Linguistics.
Philip Resnik, Mari Broman Olsen, and Mona Diab. 1999. The bible as a parallel corpus: Annotating the "book of 2000 tongues'. Computers and the Humani- ries, 33:1 13:122-153.
Tolúlope Ögúnremí, Wilhelmina Onyothi Nekoto, and Saron Samuel. 2023. Decolonizing nip for "low- resource languages": Applying abebe birhane's rela- tional ethics. GRACE: Global Review of Al Commu- nity Fthics, 1(1).
Bowen Shi, Diane Brentari, Greg Shakhnarovich, and Karen Livescu. 2022a. Open-domain sign language translation learned from online video. arXiv preprint MiXi:2205.12877
Abhilash Pal. Stephan Huber, Cyrine Chaabani. Alessan- dro Manzotti, and Oscar Koller. 2023. On the impor- tance of signer overlap for sign language detection.
Bowen Shi. Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. 2022b. TTIC's WMT- SLT 22 sign language translation system. In Proceedings of the Seventh Conference Machine Translation (WMT), pages 989-993 Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguisties.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Laia Tarrés, Gerard I. Gállego, Amanda Duarte, Jordi Torres, and Xavier Giró-i Nieto. 2023. Sign language translation from instructional videos. In Proceedings

---

of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 5624-5634.
United Nations. 2021. International day of sign lan- guages 23 september.
David Uthus, Garrett Tanzer, and Manfred Georg. 2023. Youtube-asl: A large-scale, open-domain american sign language-english parallel corpus.
Gul Varol, Liliane Momeni, Samuel Albanie, Triantafyl- los Afouras, and Andrew Zisserman. 2021. Read and attend: Temporal localisation in sign language videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16857-16866.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Andreas Voskou, Konstantinos P Panousis, Dimitrios Kosmopoulos, Dimitris Metaxas, and Sotirios Chatzis. 2021. Stochastic transformer networks with linear competing units: Application end-to-end translation. In Proceedings of the IEEF/CVF In- ternational Conference on Computer Vision, pages 1946-11955
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional pose machines. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4724-4732.
World Health Organization. 2023. Deafness and hearing loss.
Aoxiong Yin, Zhou Zhao, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. 2022. Mislt: To- wards multilingual sign language translation. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 5099-51009.
Kayo Yin and Jesse Read. 2020. Better sign language translation with "TMC-transformmer. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 5975-5989, Barcelona. Spain (Online). International Committee on Compu- tational Linguistics.
Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. 2021. Improving sign language transla- tion with monolingual data by sign tack-translation. In Proceedings of the JEFFSCV Conference on Com- puter Vision and Pattenn Recognition, pages 1316 1325.

---

Total duration (in hours)
200
150
100
50
0
mfs
csg
gsg
fcs
nsi
ms
sfb
1sl
tsm
Sqs
sgn-CM
Language Pairs
Figure 5: Total duration (in hours) per language pair.
A Translation process at Jehovah's Witnesses
The Witnesses' approach to sign language translation is thorough and collaborative Newly recruited translators receive extensive training in translation principles and work in teams, where each member performs specific role such as translating, checking, or proofreading the material. To ensure the highest quality of translation, panel of deaf individuals from diverse backgrounds and locations review the translation and provide valuable feedback to refine the signs and expressions used in the final video. This step guarantees that the message is conveyed accurately and naturally.
In addition to their translation work, the sign-language translators participate in congregation meetings and hold Bible studies with non-Jehovah's Witnesses members of the deaf community, enabling them to stay abreast of language developments and improve their skills. This diligent approach to sign-language translation ensures translators stay up-to-date with the language.
The vidcos are recorded in studio with proper lighting and the translation is done from the region's ss official spoken language to the country's 's sign language verse by verse. This work is incremental and still ongoing- as of January 2023, the complete Bible is only available in three sign languages (American Sign Language, Brazilian Sign Language and Mexican Sign Language).
B Additional Statisties
The distribution the total number of hours in each language pair can be found at Figure 5.
Details data splitting procedure
First, we sort the samples by cross-lingual frequency in descending order, based on the number of sign languages in which they appear. This ensures that samples with the most overlap across sign languages will be found at the top of the list, while samples with the least overlap will be found the bottom (Figure 4). We proceed by partitioning the samples into three distinct and non-overlapping buckets. The test bucket consists of the 1,500 most frequently occurring samples, followed by the dev bucket containing the next 1,500 samples, and finally the train bucket comprising the remaining samples. For each language, the test, dev, and train sets are formed by intersecting the language-specific samples with their corresponding buckets. This method helps to eliminate the possibility of a sample in the test set in one sign language being found in the train set in another language, which could lead to cross-contamin cross-contamination when training a multilingual model and possibly inflate the test set evaluation scores.
Phttps: //www//wwwww// jw. www.o//////////aa/as.eeee wi twesses wtwps//swwttteeetttttttt////aaaa
language-translationn translation/

---

D Development of automated loader
To create this loader, we followed a few key steps. First, we created an index file that contains a comprehensive list of verses and their attributes, such as the video URL on the JW website, start and end times of the verse in the video, and link to the corresponding written text on the JW website. We ensured that all selected videos had frame rate of 29.970 fps (the most common fps used in the videos on the website) and resolution of 1280x720, and we eliminated any duplicates. The index file was stored in JSON format, which has the advantage of being easily updatable when the website gets updated. Next, we developed script that utilizes the information in the index file to automatically load frames/poses and corresponding text, aligning them appropriately form dataset. With this loader, users have the option to form a dataset for a specific sign language, a set of sign languages, or all sign languages as needed. The human poses estimation can be obtained from Mediapipe Holistic (Lugaresi et al., 2019) or OpenPose (Cao et al.. 2017). Human pose estimation refers to a computer vision task that involves detecting, predicting, and monitoring the positions of various joints and body parts. Both OpenPose and Mediapipe Holistic are capable of detecting various keypoints present in videos, including those the face, hands, and body.
We believe that the automated loader for JWii integrated in the Sign Language Datasets library will streamline the process of accessing sign language data.
E Dataset Statistics
Table 5 highlights the statistics about all the 98 sign languages in JWSign.
F
Language Groupings
Table 7 and Table highlights the sign language groups and spoken language groups respectively, used during Clustering Families Training.
G Results detailed
Table 8 shows the evaluation results model B36 and model M36, while Table shows the evaluation results on model M91, CSIG, CSPO and MFT.

---

sign language
iso
spoken
samples
duration
avg
train / dev
Mexican
mis
31056
184.473
22
28057 1500/ 1499
Brazilian
bzs
pt-br
30949
211.135
25
27957 1494 1498
American
use
en
29150
134.655
17
26358 1340 1449
Russian
rsl
ru
26949
110.571
15
24109 1449 1391
Italian
ise
it
20882
114.923
20
19376 / 796 / 710
Colombian
csn
20644
127.63
23
18506 / 1151 .87
Spanish
ssp
19394
85,207
16
16416 / 1483 / 1495
Korean
kvk
ko
18287
93.322
19
16030 / 1104 / 1153
Argentinean
aed
17818
106.856
22
14946 / 1435 / 1437
Chilean
15357
100.15
24
12845 1282 / 1230
Ecuadorian
ccs
13331
68,873
19
10577 1368 / 1386
Polish
pso
12994
61.294
17
10085 1435 1474
Peruvian
12843
81.079
23
9890 / 1456 1497
British
bfi
12538
54,925
16
9557 / 1485 1496
Japanese
jsl
11832
67.154
21
8929 / 1409 1494
Indian
l 31384
53.298
17
8609 1358 1417
Venezuelan
vsl
10634
$1.821
18
7720/ / 1420 1494
South African
sfs
9837
47.159
18
7046
1352 1439
Zimbabwe
zib
9463
47,889
19
6787 1313 1363
German
ggg
9335
45.164
18
6412 1432 1491
Malawi
sgn-MW
ny
8849
42.815
18
6266 / 1246 1337
French
fsl
7415
30.545
4735 1257 1423
Finnish
fi
6303
34,883
20
3432 1394 1477
Angolan
sgn-AO
pt-pt
5867
32.05
20
3490 100 1277
Australian
5597
28.75
2900 1266 1431
Cuban
5406
25.968
2868 / 1145 1393
Indonesian
5201
29.651
21
3000 1280
Filipino
psp
4406
20.638
1096 1382
Chinese
zh-CN
4280
19,278
17
2143 / 778 1359
Zambian
en
4067
24.666
22
/ 1261
Quebee
4058
19.518
18
/ 1056 1398
Bolivian
3881
27.864
26
1411 / 1117 1352
Paraguayan
pys
gn
3810
22.837
22
1506 979 1325
Kenyan
xki
3452
17.351
19
1286 / 927 1239
Czech
cse
Cs
3412
17.537
19
1498 / 593 1321
Ghanaian
gse
en
3185
16.062
19
1965 / 378 842
Hungarian
hsh hu
3125
16.011
19
851 / 837 / 1437
Taiwanese
tss
zh-TW
2754
13.707
18
799 722 1233
Swedish
swl sv
2540
13.532
20
768 / 519 1253
Portuguese
psr pt-pt
2368
12,208
19
466 / 593 1309
Nigerian
nsi en
2347
11.898
19
774 / 593 980
Slovak
svk sk
2302
13.48
22
450 / 505 1347
Honduras
hds
2290
14,964
24
594 405 1291
Costa Rican
2099
11.177
20
478 / 338 1283
Guatemalan
gsm
2081
11.763
21
517 / 309 / 1255
Panamanian
Isp
2025
12.741
23
397 / 326 / 1302
Nicaraguan
nCS
2013
13.152
24
534 / 278 / 1201
Madagascar
mzc
mg
1935
11.624
22
321 577 1037
Salvadoran
1806
10.289
21
458 / 232/ 1116
Romanian
mss
1647
9.632
22
126 339 1182

---

Mozambican
mzy pt-pt
1628
9.596
241/311 1076
Greek
gss el
1627
10.011
102/339 1186
Thai
tsq th
1456
8,049
143/201 1112
Congolese
sgn-CD fr
1334
6,914
Ivorian
sgn-CI fr
6.122
Croatian
esq hr 960
4,922
85 113/ 762
Myanmar
sgn-MM my
666
4.623
25
91 / 99 476
Tanzanian
tza
sw
651
4.007
80 / 467
Malaysian
xml
ms
643
3,936
23
57 84 502
Belgian French
sfb
fr
609
3,292
20
56 67 / 486
Vietnamese
hab
vi
606
3.18
38 / 90 478
Uruguayan
ugy es
398
1.994
19
19 34 345
New Zealand
nzs en
368
1.849
19
52 36 280
Irish
isg en
362
1,708
17
23 / 36 300
Dutch
dse nl
330
1,366
15
8/271 / 295
Serbian
ign-RS
sr
328
1.922
22
/2//6/ 220
Albanian
sqk
nsl sq
310
1,605
19
1/10/ 10/ 299
Norwegian
275
1,354
18
32 29 214
Swiss German
sge de
270
1.244
17
19 229
Latvian
Isl lv
257
1.149
17
6/22/
Austrian
de
252
1.309
19
1/15// 236 236
Danish
dsl da
243
1.301
20
48 / 32 163
Ugandan
ugn
236
1.192
19
10/ / 199
Nepali
nsp
228
1.301
21
5/12/ 1/12/21 211
Jamaican
jls
227
1.067
36 180
Flemish
vgt nl
183
0.986
20
7/ 14 162
Israeli
isr he
177
0.782
16
20/ 148
Turkish
173
0.913
9 158
Lithuanian
lls
170
0.793
5/7 158
Ethiopian
eth
162
0.84
12 18 132
Cambodian
sgn-KH km
145
0.629
1/5 139
Mongolian
0.571
120
Armenian
hy
129
0.763
5/6/ 118
Estonian
0.674
6/9/
Melanesian
sgn-PG en
9/7/
Slovenian
sgn-S1
0.596
1/2/93
Suriname
sgn-SR nl
0.487
2/2/92
Bulgarian
bgn bg
0.479
3/2/82
Rwandan
sgn-RW
TW
0.402
13/
Sri Lankan
sqs si
0.403
1/1/59
Hong Kong
hks zh-TW
60
0,356
2/1/57
Singapore
en
0,217
0/0/33
Fiji
sgn-FJ en
30
0,215
0/0/30
Lebanese
sgn-LB ar
28
0.174
23
0/0/28
Samoan
sgn-WSS sm
8
0.077
0/0/8
Burundi
sgn-BI fr Isy
3
0.017
21
0/0/3
Mauritian
en
2
0.022
41
0/0/2
Cameroon
sgn-CM
fr
2
0.014
26
0/0/2
TOTAL
98
51
472529
262
19.3
520322
Table 5: Comparing the different sign languages in JWSign. iso = ISO 639-3 sign language code, samples total
number videos, duration = total duration all videos hours), avg = average duration samples (in seconds).

---

Group
Languages
Chinese
Chinese mandarin simplified (2h-CN), Chinese mandarin traditional (zh-TW),
Japanese (ja), Korean (ko), Vietnamese (vi)
Germanic
Danish (da), Dutch (nl), English (en), German (dc), Norwegian (no), Swedish (sv)
Malayo-Pollynsian
Indonesian (id), Malagasy (mg), Malay (ms)
Niger-Conge
Amharic (am), Chichewa (ny), Kinyarwanda (rw), Swahili (sw)
Romance
French (fr), Italian (it), Portuguese-brazil (pt-br), Portuguese-portuguall (pt-pt), Ro-
manian (ro). Spanish (es)
Slavic
Bulgarian (bg). Croatian (hr), Czech (cs), Polish (pl), Russian (ru), Serbian-roman
(sr), Slovak (sk), Slovenian (sl)
Uralic
Estonian (ct). Finnish (fi), Hungarian (hu). Latvian (lv), Lithuanian (It)
Other
Albanian (sq), Arabic (ar), Armenian (hy), Cambodian (km), Greek (el), Guarani-
paraguayan (gn), Hebrew (he), Mongolian (mn), Myanmar (my), Nepali (ne), Samoan
(sm), Sinhala (si), Thai (th), Turkish (tr)
Table 6: Spoken languages Groups
Group
Languages
America
American (ase), Bolivian (bvl), Burundi (sgn-BI), Cambodian (sgn-KH), Cameroon
(sgn-CM), Colombian (csn), Congolese (sgn-CD), Costa Rican (csr), Ecuadorian
(ecs), Ethiopian (eth), Filipino (psp), Ghanaian (gse), Guatemalan (gsm), Indonesian
(inl), Ivorian (sgn-CI), Jamaican (jls), Kenyan (xki), Malawi (sgn-MW), Malaysian (xml), Myanmar (sgn-MM), Nigerian (nsi), Panamanian (Isp), Peruvian (prl), Rwan- dan (sgn-RW), Salvadoran (esn), Singapore (sls), Sri Lankan (sqs), Thai (Isq), Ugan- dan (ugn), Zambian (zsl), Zimbabwe (zib)
British
Australian (asf), British (bfi), Croatian (csq), Fiji (sgn-FJ), Indian (ins), Melanesian (sgn-PG), Nepali (nsp), New Zealand (nzs), Samoan (sgn-WS), Serbian (sgn-RS),
Slovenian (sgn-SI), South African (sfs)
Chinese
Chinese (csl). Hong Kong (hks), Japanese (jsl), Korean (kvk), Taiwanese (tss)
Old French
Argentinean (aed), Austrian (asq), Belgian French (sfb), Dutch (dse), Flemish (vgt), French (fsl), German (gsg), Greek (gss), Irish (isg), Israeli (isr), Italian (ise), Mexican (mfs), Quebec (fes), Spanish (ssp), Swiss German (sgg), Venezuelan (vsl)
Polish
Bulgarian (bqn), Czech (cse), Estonian (eso), Hungarian (hsh), Latvian (Isl), Lithua- nian (Ils), Mongolian (msr), Polish (pso), Romanian (rms), Russian (rsl), Slovak (svk)
Swedish
Brazilian (bzs), Danish (dsl), Finnish (fse), Madagascar (mzc), Norwegian (nsl),
Portuguese (psr), Swedish (swl)
Uruguay
Chilean (csg), Paraguayan (pys), Uruguayan (ugy)
Other
Angolan (sgn-AO), Honduras (hds), Nicaraguan (ncs), Suriname (sgn-SR), Turkish (tsm), Vietnamese (hab), Lebanese (sgn-LB), Albanian (sqk), Armenian (aen), Mau- ritian (lsy), Mozambican (mzy), Tanzanian (tza), Cuban (csf)
Table 7: Sign languages Groups

---

Language Pair
BLEU
BLEURT
chrF
BLEU BLEURT chrF
mfs-ves
2.33
21.18
15.33
1.62
27.75
14.74
bzs->pt-bb
2.12
18.7
14.96
1.47
21.76
12.75
ase->en
4.16
37.49
18.99
2.48
36.91
15.46
rsl-bru
3.09
26.38
17.81
1.14
24.74
12.21
isc->it
2.58
25.49
16.3
1.33
27.4
12.45
csn->es
2.04
23.73
14.07
1.32
28.03
14.48
ssp->es
2.64
20.42
17.52
1.48
27.17
14.68
kvk->ko
1.37
32.38
9.88
3.68
31.88
16.56
acd->es
21.37
15.48
1.44
26.91
14.68
csg->es
2.38
19.1
16.63
1.36
28.22
14.8
ees >es
1.76
16.94
16.38
1.52
26.36
pso->pl
1.78
17.15
17.06
0.38
15.76
11.44
pri->es
1.92
19.57
15.69
1.44
27.1
bfi->en
2.89
36.02
18.19
1.99
36.11 11
15.05
jsl->ja
7.08
23.96
14.77
5.44
25.49
14.83
ins
2.87
35.01
18.43
1.82
36.25
15.01
vsl vel->es
1.83
15.74
16.87
1.3
26.94
14.03
sfs->en
1.99
33.95
16.16
1.75
36.39
14.89
zib->en
1.47
34.43
14.92
1.55
36.69
14.39
gsg->de
1.22
19.34
16.58
0.58
19.9
12.12
sgn-MW-bny
0.93
26.81
19.06
0.31
23.95
12.17
1.36
0.48
16.28
0.47
-2.24
sgn-AO->pt
0.73
13.18
13.03
0.78
19.39
11.72
fse->i
1.2
22.33
18.45
0.35
15.41
int->id
45.08
19.16
0.23
40.53
13.02
asf->en
1.52
32.72
15.5
1.75
37.18
csf-ses
1.02
12.97
14.75
1.07
26.49
14.06
CNl->2h-CN
3.96
33.7
14.49
4,92
34.14
16.32
gse->en
1.22
31.5
16.14
1.15
35.94
14.92
psp->en
1.11
31.88
16.97
1.68
37.1
14.87
2sl->en
1.08
27.51
15.71
1.22
35.58
14.41
fos->fr
0.8
14.98
0.51
-1.76
8.32
pys->gn
0.64
17.65
15.73
0.17
13.02
12.53
ese->es
0.38
11.19
12.84
0.02
16.38
4.09
bvi-ses
0.57
9.24
14.81
1.08
27.21
14.31
xki->en 0.85
28.98
16.11
1.53
36.33
14.42
Table 8: Evaluation Results on B36 and M36.

---

M91
CSIC BLEURE
CSPO
MPT BLEUKT
Language Par
BLUU
BLEURI
chrt!
BLBU
chrfl
BLEU
ILBURT
BLEUU
chrll
brssagtohe
3,03
20.3
22.85
0.38
sse-oen
1.84
37,03
15.09
3,06
37.56
16.52
3.86
39.08
18.91
0.87
$2.06
15.35
1.21
22.55
12.63
3.12
55.39
17.8
3.75
26.88
18.63
0.04
7.6
1
ise->i
1.33
29.58
12366
2.57
28.06
14.76
1335
26.9
12,76
0.03
10.51
8.12
1.47
27.87
1342
1.16
76
14.35
1.73
26 04
14.63
0.66
15.91
16.44
14.04
2.85
14.84
30.3
9.23
10.66
27.04
26.38
og-oes
1.67
26.67
1.58
26.80
14.77
1.3
25.55
0.72
16.08
psooppl
0.36
11.46
11.12
1,00
20,40
16.3
192
21.92
16.4
0.05
12.51
8.76
pl-ses
1.74
14.19
3.00
27.24
14.6
1.51
25.98
14.19
0.6
17.93
16.78
1,62
35.87
14.55
3.13
36.26
18,99
3.04
38.01
17.98
1.05
31.82
15.54
5.06
24.97
1378
6.55
24.3
13.19
23.7
12,52
22.08
12.96
14.18
37.94
1.57
27
26.32
1,45
2.52
37.25
17.63
8.88
15.74
1188
35.76
13.84
1.63
36.69
16.71
31.73
25g-ode
0.37
19.85
1279
0,91
22.51
13.99
1.12
23.48
16.19
0.09
22.76
13,00
Myn-MW-nyy
0.21
24.5 51
13,88
0.R8
28.04
15.3
1:31
27 46
18.27
0.03
22.75
7.31
Bi-sfi
0.69
1.69
9.23
1.1
3,80
12.2
0.74
0.35
9.75
9.47
-2.69
14.1
0.86
18.1
12.54
0.88
10.7
13.32
0.39
21.31
12.47
0.68
10.42
13.98
fre onl
1,46
27577
19.57
10.83
20.04
sen
1.99
16.33
14:75
2,23
af-ccs
1:23
25.85
0.74
24-57
0.51
15.13
od sch CN
6.54
29/01
16.48
4,25
36,39
1395
332
36.21
13.2
0L18
28.8
3,82
pe-sei
1.11
35.17
14.55
1.56
35.43
15.52
1.34
36.35
16.63
0.98
$1.39
15,80
ben
1.43
26,53
142
1.11
26.05
15.13
2.09
37.35
16.41
0.93
3L88
15.71
1.06
35.00
14.03
1.44
35.07
15.07
1.68
34.54
16,16
0.78
30.55
14.97
fes-sir
0.7
1.77
9.06
1.00
4,61
11.49
0.79
9.42
0.6
1.99
13.86
ey--ggn
17.34
0.04
7.99
1.23
27.24
13385
1.53
25.93
xki sen
1:17
34.11
1383
1
35.51
15.02
1.6
35.56
16.39
1.15
31,50
15.27
lsh-oha
0.02
42,61
6.11
1:23
30.29
14.11
1:25
27.09
15,33
D.95
27.01
15.56
iss TW
2.02
19.33
26.32
0A8
17.28
6.62
0.57
18.34
5.68
4.6
21.54
21.93
usi asi-oen
1.31
36.11
14.41
LSS
36.06
15.20
2.43
36.64
16,94
1.29
52.65
16.54
oul-bwr
0.02
21.11
3,93
0.75
23.99
12.88
0.25
14.33
9:05
0.58
19.74
15.12
16.55
24.11
gen
1,35
26.15
25.21
0.26
16.36
1.1
13.56
1-1
35.04
13.05
13.60
0.63
18.26
16.15
por optipt
0.95
18.77
1300
1,16
16.82
14.36
0.55
21.1
12.28
0.69
11.12
10,20
on-oes
1.21
26.99
13.79
1.15
26.15
13.25
1.07
24.64
13.88
0.71
18.52
16.36
ovk ootk
0.04
22.33
6,29
0.38
17.47
11.05
0.26
16.92
10.55
0.92
10.33
12.96
lip-ses
1.20
26.68
14.02
1.18
25.01
13.84
1.11
24.59
14.21
0.82
18.56
17.00
2887
sgn-CD-di
14.55
myy-opipt
Dipoth
1.34
15779
15.17
13.47
7.09
38.71
27.4
me-oro
1.19
23.59
8.74
0.06
18.07
9.11
0.01
25.6
4.28
2.22
16.14
13.99
sgn Cl-sdr
0.69
0.57
9.3
0,01
4.4
0.3
0.71
1.00
9.7
0152
-2.23
14.3
pa sel
1,05
18,05
23.38
0,01
1.96
5.72
0.6
6.12
14.54
8.16
7.52
28.67
MMyony
4.5
44.03
2765
0.06
9.75
9.7
1,88
9.6
16,89
7.64
41.13
27.41
22.5
1846
14.05
15.58
56
8.1
12.56
46.99
ab-afr
0.1
.0.76
0.5
-2.12
14.13
nas sen
0.96
34,56
1291
0.91
33.97
14.81
2.04
36.66
15,00
0.96
$1.01
19.82
dsl-zeh
0,05
19.23
2.44
0.31
22.8
12.85
0.8
17.44
6.89
1.58
19.41
15.00
hab ovi
2.06
4.79
16
0,52
7,83
7.55
1.29
6.29
5.21
8.57
00.8
19.33
nd--on
0.28
19.46
3.76
0.36
1904
11.92
15.3
3.97
1.72
18.07
14.78
37.95
35.57
14.91
ugy-ses
0.81
15.11
Ja-
0.76
1.27
8.68
agn sew
0.13
6.58
0.00
13.59
4.95
9.07
11.07
0.09
10.54
ab-2am
0.37
56.43
15.7
0.01
34.34
0.87
28.35
1.37
0.3
$5.05
14.91
>de
0.81
00.96
13.17
0.59
22.58
13.3
0.68
22.86
14.88
0.15
22.9
13.4
0.28
30.23
11.51
1.08
34.37
14.36
36.95
15.63
0.48
31.41
14,1
329
36.21
Table 9: Evaluation Results on M91, CSIG, CSPO and MFT.

---

M91
CSIG BLEURT
CSPO
MFT BLBURT
Language Pair
BLEU
BLEURT
cliF
BLEU
cleF
BLEU
BLEURT
chuF
BLEU
dliF
tem-oir
9.4
dse -nl
0.05
19.32
4.77
0,00
15.37
3.81
13.01
2.83
0.1
16.67
12.31
0.02
18.54
5:21
0.00
15.41
4.6
0.01
13:42
2:75
0.14
16.00
12.05
1s1 scv
0.19
40.5
4,27
0,00
21.58
3.71
0.14
20119
5,85
0.83
22.5
10.42
0.01
24.41
0,06
19.58
0.14
16.79
8.01
0.11
24.97
asp-Sue
hyn-hng
Mks-och-TW
3.95
25.6
17.66
7.72
3.17
1721
19.37
so--RR-onl
0.03
17.68
4.66
0.01
15.76
2.05
0.1
12.3
2.68
0.12
18.70
1344
sak
1.01
34,96
15.89
0,00
21.64
1.51
0.09
18.5
3.97
0.35
26.95
8.0
<gn-Ill-bmm
2.87
49.28
32.52
22.58
D.19
20.62
0.94
2.84
47.95
78.47
17.3
sge
27.95
spa-S1-aal
sgn-lloorr
5.7
gg-CMM-orr
1.13
3,51
9.49
0.27
7.97
0.48
1.4
-3.04
10.23
0.95
-4.75
11.3
'gn -l-----omn
0.34
41.56
13:04
0.89
13.4
16.89
0.72
35.4
15:42
0.94
10.54
16.08
sgneLBBoar
0.03
57.63
1.47
0.02
16.36
1.56
0.02
32.27
1.95
1.83
24.95
11.12
Table 10: continuation: Evaluation Results on M91, CSIG, CSPO and MFT.


Reviews:
[{"review_id": "o72XlKQhaK", "reviewer": "Reviewer_x6To", "summary": "", "questions": "* Did you consider brainstorming about the limited domain language being covered in the dataset? It would be good to highlight the ethical concerns in more detail with some efforts made to study this problem in more detail. \n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q. Did you consider brainstorming about the limited domain language being covered in the dataset?It would be good to highlight the ethical concerns in more detail with some efforts made to study this problem in more detail\n\nA. Yes we did brainstorm about the limited domain language in this dataset. We dedicated Section 2.1. showcasing how data of the Bible domain has been extensively used in the advancements of NLP for languages in other modalities. A recent example is Meta’s Massively Multilingual Speech (MMS) model which was released last May. This work introduced a new dataset; MMS-lab dataset, containing Bible translations in 1,107 languages which helped achieve state of the art in multilingual speech-to-text translation. See “Our Approach” section https://about.fb.com/news/2023/05/ai-massively-multilingual-speech-technology/ ; \n\n“””\nOur Approach\nCollecting audio data for thousands of languages was our first challenge because the largest existing speech datasets cover 100 languages at most. To overcome this, we turned to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research.\nBy considering unlabeled recordings of various other Christian religious readings, we increased the number of languages available to more than 4,000. While this data is from a specific domain and is often read by male speakers, our analysis shows that our models perform equally well for male and female voices. And while the content of the audio recordings is religious, our analysis shows that this doesn’t bias the model to produce more religious language. \n“””\n\nThis suggests that carefully using data from a religious domain does not bias to produce more religious language and instead, can be a crucial catalyst to reach state of the art for multilingual machine translation. \nAlso, one other way to use this dataset will be to fine-tune smaller datasets of a more desirable domain on a pretrained model from this one. Given the extreme scarcity of sign language datasets compared to spoken languages in text or audio modalities, this dataset will be very useful to the sign language processing community.\n\nFinally, the multi-parallel nature of this dataset makes it the perfect candidate to help (sign) linguists study cross-sign linguistics patterns so as to have a better appreciation of sign languages. \n\nWe are happy to clarify any remaining questions you may have before reconsidering your scores. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper covers a wide range of sign languages and may act as a valuable resource for the sign language processing community to study the linguistic differences between various signed languages.\n\n* The paper provides a detailed set of experiments over multiple combinations for translation. Though the translation scores are not high, it might still be usefull to study these mappings from one sign language to another. \n\n* The paper clearly highlights the primary limitation of the proposed dataset of being in a limited domain, which may not be the true representative of communication in sign language. \n\n\n", "reasons_to_reject": "* Though the dataset is diverse and unique in terms of multilingualism, the number of samples still remains a significant concern for the proposed dataset. It may be out of the scope of this work to collect more samples for these languages due to resource constraints. However, the usage of this dataset in the data-driven community will be limited for developing a reliable translation system. \n\n* The primary concern with the dataset is ethical issues. As the dataset is specific to a particular domain, it is challenging to validate the impact of the dataset in a negative direction. Moreover, the authors claim that the data strictly and entirely belongs to JW. It would be good to go through an ethical review to make it available for the data-driven natural language processing community.\n\n", "questions_to_authors": "* Did you consider brainstorming about the limited domain language being covered in the dataset? It would be good to highlight the ethical concerns in more detail with some efforts made to study this problem in more detail. \n", "paper_topic_and_main_contributions": "The paper curates a new multilingual corpus for sign language processing, using Bible Translations to capture more diversity in the dataset. The proposed dataset covers 98 sign languages with a varied number of samples for each of the sign languages ranging from 2 samples to 31k samples. The paper further provides baseline results over 36 bilingual pairs. Moreover, the paper also explores the multilingual setting for sign language processing and reports the findings.\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Justification_for_Ethical_Concerns": "The primary concern with the dataset is ethical issues. As the dataset is specific to a particular domain, it is challenging to validate the impact of the dataset in a negative direction. Moreover, the authors claim that the data strictly and entirely belongs to JW. It would be good to go through an ethical review to make it available for the data-driven natural language processing community.\nValidating multicultural data for Sign languages may require more expertise."}}, {"review_id": "iQJY3MonYC", "reviewer": "Reviewer_Ty6U", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q. The results of the baselines seem quite low overall, and it could be good to have some more analysis about the reasons.\n\nA. Sign Language Translation (Sign-to-Text in this case) has proven to be a very challenging task and our baseline scores are in line with results of shared tasks as the WMT Shared Task on Sign Language Translation where BLEU scores are usually below 2 [1,2]. Additionally, when looking at the “Avg” column in Table 1, the average length of samples in our dataset is two or three times that of  other datasets. This is one of the reasons why other datasets such as PHOENIX, CSL-Daily or SP-10 usually have higher BLEU scores. Additionally, the Bible domain is a hard domain compared to other datasets with easier domain and smaller vocabulary such as weather forecast.    \n\nQ. The authors attached the detailed evaluation results in the Appendix, but it could be good to have more in-depth analysis, ex. about the variance of the results, whether there are similarities between the sign languages or if they are very different, whether there are error patterns?\n\nA. Given that we had 98 language pair directions, our first analysis tried grouping them into fewer classes i.e. “High”, “Medium”, “Low”, “Very Low” and “Zero” and having a general overview of the trend in the scores across the board. With our 1 additional page, we intend to include a more fine-grained analysis, for instance on translation of Japanese Sign Language to Japanese, having the highest scores even with a low number of training samples.  \n\nQ. Would it be possible to adjust a bit the numbers in Table 4 so that they appear larger?\n\nA. ​​Thank you, we will certainly fix this simple layout problem in the camera-ready version.\n\nQ. Table 8: a separation rule is missing for M36\n\nA. We kept this style to be consistent with Table 9 in the Appendix. We will find a way to better separate results visually in the camera-ready.\n\n[1] Findings of the First WMT Shared Task on Sign Language Translation (WMT-SLT22) (https://aclanthology.org/2022.wmt-1.71)\n\n[2] https://ocelot-wmt23.mteval.org/leaderboard/3\n\n\nWe are happy to clarify any remaining questions you may have before reconsidering your scores. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation is clear and the paper provides a nice background on the sign language datasets.\n\nThe proposed multilingual dataset has a higher diversity in terms of number of spoken languages, collected from ~1500 signers with diverse demographics, and is also significantly larger and longer compared to other datasets.", "reasons_to_reject": "The licensing terms of the data and of the work don't seem to be described in the paper, which may lead to issues using it.\n\nI have also some concerns about whether the baselines are strong enough. The results of the baselines seem quite low overall, and it could be good to have some more analysis about the reasons. The authors attached the detailed evaluation results in the Appendix, but it could be good to have more in-depth analysis, ex. about the variance of the results, whether there are similarities between the sign languages or if they are very different, whether there are error patterns?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper propose a new multilingual dataset for sign language processing, JWSign, based on Bible translations. The dataset includes 2530 hours of Bible translations in 98 sign languages.\n\nThe authors provided baseline evaluation results for bilingual and multilingual training.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Presentation: \n\n- Would it be possible to adjust a bit the numbers in Table 4 so that they appear larger?\n- Table 8: a separation rule is missing for M36", "Justification_for_Ethical_Concerns": "It is unsure whether the dataset collection and usage are compliant with the terms of use of jw.org."}, "extra_text": {}}, {"review_id": "eDtvzrUlSL", "reviewer": "Reviewer_Hyq8", "summary": "", "questions": "A. Why have you implemented a sign-to-text translation; when the dataset suggests the text-to-sign?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q. Why have you implemented a sign-to-text translation; when the dataset suggests the text-to-sign?\n\nA. We made the choice to have our first baseline on sign-to-text translation because after consultation with some members of the deaf community, we realised it is the task with the most pressing practical usage right now (compared to text-to-sign). From the perspective of breaking down language barriers, sign-to-text is more important, because in the other direction speech recognition can be used as a substitute tool / solution for deaf people to understand what hearing people are saying.\n\nQ. However, the common sign language dataset or corpus is created in reverse: original signing to textual annotations. That makes the proposed dataset incomparable to other sign language corpora, created in a reversed way.\n\nA.\n\nIt is not true that commonly, sign language datasets are created in the way you suggest  (signed language, such as ASL, first - spoken language translation, such as English, second). Almost all parallel datasets for sign languages are created by translating in the opposite direction.\n \nFor example, this is true for 6 out of the 7 datasets we compared to in Table 1:\n\nPHOENIX ; speech (from TV Broadcast) → sign \n\nKETI ; text → sign\n\nCSL-Daily ; text → sign\n\nBOBSL ; speech (from TV Broadcast) → sign \n\nHow2Sign ; text → sign\n\nOpenASL ; sign → text\n\nSP-10 ; text → sign\n\nJWSign (proposed dataset) ; text → sign \n\n\nIt is also not true that this characteristic of a dataset (which language is the original one, and which is a translation) makes it incomparable to other datasets. If you think otherwise, we kindly ask you to provide some evidence (such as a previous paper) for this.\n\nWe hope you are satisfied with our response and might reconsider your scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This greatly contributes to the sign language community; the work is relevant for cross-sign studies and translation evaluation.\n\nThe paper is clearly written, extensive statistics are given, and a lot of possible limitations and criticisms are addressed. ", "reasons_to_reject": "The proposed dataset was created with the following flow: original text to sign language translation. However, the common sign language dataset or corpus is created in reverse: original signing to textual annotations. That inconsistency has to be always stated. That makes the proposed dataset incomparable to other sign language corpora, created in a reversed way. ", "questions_to_authors": "A. Why have you implemented a sign-to-text translation; when the dataset suggests the text-to-sign?", "paper_topic_and_main_contributions": "The paper proposes a compilation of the existing Jehovah’s Witnesses Bible sign language data sets as a new multilingual (98 sign languages) data set. The paper makes the first attempts at creating sign-to-text translation systems using the proposed dataset. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel multilingual dataset for sign language processing, leveraging Bible Translations to enhance diversity. The paper also presents baseline results to bilingual pairs and explores the application of multilingual approaches in sign language processing. In general the paper seems like a good contribution, but there might be some issues involving the data licensing that needs to be addressed.
Decision: Accept-Findings