Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language
Authors: Catarina Sousa
Luisa Coheur
Mara Moita
Venue: EMNLP 2023 Findings (2023)
Keywords: Portuguese Sign Language
Machine Translation
Computational Linguistics
Natural Language Processing
Deep Learning
TL;DR: The paper presents rule-based and neural models for translating European Portuguese to Portuguese Sign Language, yielding improved results and a new corpus and a gold collection for future evaluations.

Abstract: Portuguese Sign Language (LGP) is the official language in deaf education in Portugal. Current approaches in developing a translation system between European Portuguese and LGP rely on hand-crafted rules. In this paper, we present a fully automatic corpora-driven rule-based machine translation system between European Portuguese and LGP glosses, and also two neural machine translation models. We also contribute with the LGP-5-Domain corpus, composed of five different text domains, built with the help of our rule-based system, and used to train the neural models. In addition, we provide a gold collection, annotated by LGP experts, that can be used for future evaluations. Compared with the only similar available translation system, PE2LGP, results are always improved with the new rule-based model, which competes for the highest scores with one of the neural models.

Enhancing Accessible Communication:
from European Portuguese to Portuguese Sign Language
Catarina Sousa and Luísa Coheur INESC-ID, Lisbon, Portugal Instituto Superior Técnico, University of Lisbon, Portugal
catar natasiinsous2200ttccnni... ilio.o.. pt
luisa coweur@tecnica www..ullssbb.ooo..
Mara Moita
Universidade Católica Portuguesa, Portugal maramoi ta@ucp. pt
Abstract
translation system from EP to LGP. Some of its rules are hand-crafted and some are automatically extracted from a linguistic corpus with annotations of LGP videos?, from now on COLIN. COLIN is the only existing LGP annotated corpus (to the best of our knowledge). It currently consists of 113 hours of video recordings, with 20 of them be- ing annotated at various linguistic levels (ELAN² (Sloetjes and Wittenburg, 2008) was used). The videos in the corpus were recorded between 1992 and 2019 and feature hearing-impaired signers ranging from 4 to 89 years old. PE2LGP still has some limitations, as the subset of COLIN used to develop its rules is very small (three minutes) and it is necessary to perform PE2LGP translator's grammar. In this paper, we take advantage of an extended version of the corpus used in PE2LGP and present a rulc-basse base-basddd system that is now fully automatic. We use this bale-based system to create parallel corpus between EP and LGP, from now on LGP-5-Domain (LGGP5), with text from different domains simple sentences, so- cial media, poetry, dialogue, and news. LGP5 is then used fine-tune two large multilingual neu- ral machine translation models, and evaluated on a gold collection, built by LGP experts*. With this work, we hope to contribute to benchmarking this task.
Portuguese Sign Language (LGP) is the official language in deaf education in Portugal. Current approaches in developing a translation system hetween European Portuguese and L.P rely on hand-crafted rules. In this paper, we present fully automatic corpora-driven rule-based ma- chine translation system between European Por- tuguese and LGP glosses, and also two neural machine translation models. We also contribute with the I.P-5- corpus, composed of five different text domains, built with the help of rule-based system, and used train the neural models. In addition, we provide gold collection, annotated by LGP experts, that can be used for future evaluations. Compared with the only similar available translation system, PE2LGP, results are always improved with the new rule-based model, which competes for the highest scores with one of the neural models.
manual tasks to create
1 Introduction
According the Portuguese Association of Deaf people', there are around thirty thousand deaf peo- ple that use Portuguese Sign Language (LGP) in their daily lives. However, European Portuguese (EP) and LGP are two different linguistic systems, and communication between hearing and hearing- impaired people is difficult, leading to communi- cation gap between both groups. There have been some attempts at developing a system that trans- lates EP into LGP glosses (Almeida, 2014; Gaspar, 2015; Escudeiro et al., 2015; Ferreira, 2018), but the majority only present toy examples, relying on a small set of hand-crafted rules and disregard- ing non-manual movements. PE2LGP (Gonçalves et al., 2021; Lacerda et al., 2023) is rule-based ott.ttsss.....///..
To illustrate the task at hand, Table 1 presents examples of EP sentences and their corresponding translation into LGP glosses The first row con-
Developed in the project "Corpus Lingufstico AVATAR da Língua Gestual Portuguesa (PTDC/LLT- LIN/29887/2017)"
https//anchivee.mi.llll..
*Code and datasets available at https://github. com/ 1coheur/EP2LGPE /EP.//P5..PP
'We use the same notation as in PE2LGP.

---

EP
Glosses in LGP
sign stored in database. This database contains the needed information to animate 3D avatar. Un- fortunately, no information is available regarding the creation of the translation rules and the evalua- tion of this system.
A Ana
gosta
de
(DT(A-N-A) GOSTARJ()
MASSA
massa?
(Ana likes pasta?)
(((DT(A-N-A) (q)E))(g))
PASTA
A rainha praia.
MULHER REI PRAIA IR
PE2LGP has a rule-based translation system (Gonçalves et al., 2021). avatar to perform LGP. and database with signs (Cabral et al., 2020; Lac- erda et al., 2023). Its rules were semi-automatically extracted from COLIN. In this work, we take ad- vantage of the current corpus and we also automate the whole process.
(The queen went to beach.)
(WOMAN KING BEACH GO)
Table 1: Examples of translations EP/LGP.
tains an interrogative sentence, marked by (())) facial expression denoting a question that involves raising the chin, tilting the head back, and frowning. Since "Ana" is proper name, it should be finger- spelled represented by DT(AA--A). In the declar- ative sentence of the second row "rainha" (queen) is translated to "MULHER REI" (WOMAN KING).
Another work that should be mentioned is IF2LGP (Gaspar, 2015) which consists of two mod- ules. The first is responsible for conducting syntac- tic and morphological analysis, while the second contains translation rules to convert Portuguese words into LGP glosses. However, the creation of these translation rules was based on a small dataset of ten sentences.
2 Related Work
We will only consider the translation to a sign lan- guage and not from a sign language, which is out of the scope of this paper. Early automatic trans- lation systems, such as the ones discussed in (San- Segundo al., 2008: Zhao et al.. 2000; Brour and Benabbou, 2019), adopted rule-based approach for performing the translation between the source and the target languages. An example of an (expert- defined) rule employed in the ATLASLang MTS system (Brour and Benabbou, 2019) is shown in Equation 1.
Over the past years, significant progress has been made in the field of sign language transla- tion, thanks to the advancements in statistical and neural machine translation. For instance, the ap- proach proposed by San-Segundo et al. (2008) uses a phrase-based method, trained using parallel cor- pora; also, the ATLASLang NMT system (Brour and Benabbou, 2021) employs a neural machine translation approach.
Regarding IGP, a neural approach is also ex- plored by Alves et al. (2022). It adopts hybrid structure combining rule-based and neural machine translation approaches. According to the authors. the dataset consists of 150,000 sentences. However, the grammar and the data are not available.
If Gender(w:) feminâne Then t4 w; + (female).
(1)
Equation| 1 checks if a word has a feminine gen- der and adds the term female after it. Notice that the involved types of grammar could vary. For in- stance, the work described in (Zhao et al., 2000) uses a Lexicalized Tree Adjoining Grammar to translate English words into American Sign Lan- guage glosses.
3 Towards the Portuguese Sign Language
3.1 The LGP Corpus
We were given access to COLIN, and used the forty-five minutes of the corpus that had all the needed syntactic annotations to reconstruct the ex- pert translator's grammatical rules.
Regarding LGP, there are a few attempts at creat- ing a translation system such as the ones described in (Escudeiro et al., 2015; Oliveira et al., 2019; Gaspar, 2015). VIRTUALSIGN (Escudeiro et al., 2015; Oliveira et al., 2019) is a system that per- forms bidirectional translation between Portuguese and LGP. Regarding the text-to-sign translation, the input sentence is passed through a set of grammar rules and, afterward, the system directly associates each word in the sentence with a corresponding
3.2 Improving the Rule-based Model
The corpora-driven rule-based approach used in this translation system is depicted in Figure 1. The first module automatically extracts linguistic information from the corpus, generating translation rules and a bilingual dictionary between EP and LGP. To do so, we extract from ELAN the Por- tuguese sentences, the LGP corresponding transla-

---

Information extracted from ELAN
five, will be translated into an LGP sentence with the syntactic structure Verb Nown Noun.
Construction of the translation rules
VI ADJ1 N1 ADJ2 - V1 N2 N1 (3)
Bilingual dictionary
Translation rules
This rule can be applied, for instance, to the sentence:
Automatic translation
Há grandes desenvoleimentos artísticos. (There are great artistic developments.)
Sequence of glosses
(4)
Figure 1: Pipeline of our rule- blsebased approach.
The sentence presented in Eq is translated into:
TER MUITO ARTE DESENVOLVIMENTO (HAVE- - A LOT ART DEV DEVEMMENTT
tion in glosses, and their grammatical information part-of-speech tags, subjects, and objects. Having the information regarding the LGP sentences, we analyze the EP ones to also obtain their grammat- ical information. Having information from both languages, the alignment between the Portuguese words and the LGP glosses is performed. This alignment is based on an algorithm of similarity measures, string matching, and semantic similarity. From the aligned word-sign pairs, the rules and the bilingual dictionary are created. The second mod- ule uses these translation rules and the bilingual dictionary to translate EP into LGP, where the LGP sentence is represented by sequence of glosses with markers indicating facial expressions and fin- gerspelled words. When EP sentence enters the system, the sentence is analyzed and its structure is kept. Then, the distance between the Portuguese structure and the structure of the system's 's rules is calculated. The rule with the lower distance is the most similar one to the original sentence and is applied to convert the Portuguese structure into the LGP one.
This translation occurs because there is an entry in the bilingual dictionary that is:
Haver grande TER MUITO
(There is great HAVE A LOT)
(6)
Additionally, as the rule presented in Eq. 3 sug- gests, the first adjective can now be removed as it was merged with the first verb the noun in the original sentence is moved to the last word in the translation sentence and the second adjective is con- verted into in the middle of the sentence. The alignment be- tween the rules and the source and target sentence is depicted in Fig. 2.
its lemma- and it positioned
(There (V1) great (ADJ1) artistic (ADJ2) developments, (N1)) Há (V1) grandes (ADJ1) deservolvimentos (NI) artisticos, (ADJ2)
PE2LGP has 61 general syntactic rules, some hand-crafted. Our current proposal handles 238 of such rules (218 are for declarative sentences, 7 for negative ones, and 13 for interrogatives). Eq. shows an example that states that if the Portuguese sentence has the canonical order Verbal Phrase (VP) - Noun Phrase (NP), then, the LGP sentence will have the canonical order NP- VP.
TER-MUITO (V1)
ARTE (N2)
DESENVOLVIMENTO (N1)
(HAVE-A-LOT (V1)
ART (N2) DEVELOPMENT (NI))
Figure 2: Alignment between the source sentence
(Eq, 4) and the target sentence (Eq. 5) using the rule depicted in Eq.
3.3 The Neural Models
We tuned two multilingual models: the mBART model (Liu al., 2020) and the M2M model (Fan al., 2021). Given that the LGP glosses are written in Portuguese, both the input and output languages were set to Portuguese. As a result, a Portuguese-uu Portuguese translator was created, and fine-tuning
VP NP + NP VP
PE2LGP has 90 morphosyntactic rules, and our proposal adds 228 of such rules. As an example, Eq. 3, states that, a Portuguese sentence with the syntactic structure Verb Adjective Noun- - Adjec-

---

was performed allow the models learn how translate to LGP glosses. The base models and the associated weights are from the HuggingFace's Transformers package (Wolf et al., 2020). For the mBART model, we used the mbbrt-large-50.. many-to-many-nnie checkpoint and, for the M2M model, we used the m2m/000 II2B checkpoint. The strategy implemented to fine-tune these models similar for both parameters except for the batch size and the num- ber of epochs. We used batch size of 2 for both models since it was the maximum feasible given our computational resources. We ran our model for 3 epochs since it would start to overfit if we increased the number. The employed fine-tuning data consisted of the parallel corpus created with the based approach, that is the LGP5 corpus, described next.
main only exposes the model an organized and coherent language but also broad range of topics.
4.2 The LGP5 Parallel Corpus
Our rule-based model was used to translate the gathered corpus. As a result, a new dataset compris- ing 37 7,0 EP/LGP pairs was generated (examples can be seen in Table 4, in Appendix A).
used the default hyper-
4.3
The Gold Collection
As previously said, the gold collection consists of 200 pairs of sentences EP/LGP 40 sentences from each of the five domains. These sentences were annotated by the rule-based system and given to two LGP experts. Each one validated/corrected 100 sentences.
Experiments
4
The LGPS Dataset
For the evaluation experiments, we evaluated our systems with three metrics: BLEU-4 (Papineni et al., 2002). RougeL (Lin, 2004), and TER (Snover et al., 2006).
the following, we describe the LGP5 dataset that comprises 37,500 automatically annotated sen- tences from different domains (7,500 from each domain) that were used to train the neural mod- els. Additionally, 200 sentences were manually annotated (gold collection), 40 from each domain.
In preliminary experiment, we evaluated our models test set described in (Gonçalves et al., 2021). This test set has 58 sentences EP/LGP, We also evaluated our models against PE2LGP. Table shows the results. Our rule-based approach (RB) is better than PE2LGP, and M2M is the best model overall.
4.1 Gathering Data
order to have a rich collection of simple sen- tences, we extracted Portuguese sentences from Tatoeba". With the aim to expose the models to the unique linguistic characteristics prevailing in online social interactions slang, abbreviations, and other aspects of contemporary communication commonly found social media platforms used a dataset with Portuguese tweets from Kag- gle?. Poetry texts - the complete literary work writ- ten by Fernando Pessoa, a famous Portuguese poet were also used to enable training with broader range of sentence structures. Kaggle was, again, our source of data. We also considered dialogues obtain sentences from the everyday speech of Portuguese people. For this used the dataset described in (Csaky and Recski, 2021). Finally, the last dataset was composed of news articles Training the model with sentences from this do-
PE2LGP
RB
mBART
M2M
BLEU 1 RougeL TER J
59.29
62.16
63.93
68.55
76.91
79.56
79.79
82.50
84.31
74.45
75.55
62.41
Table 2: Preliminary results.
Next, used our gold collection to evaluate the same models (Table 3).
M2M is not always the one that obtains the high- est scores, as our rule-based system the best system in the social media dataset, for all the mea- sures, and in some of the other domains, for some of the metrics. PE2LGP initially seemed to have the best TER score in the poctry domain. How- ever, closer look revealed that it failed to provide translation for four out of the forty sentences. This affected the TER calculation, making the final scores appear higher than they actually were. Con- sidering this, PE2LGP's scores were consistently
"https://tatod" http:///tatoebbbrreenn
'https://ww/ haapg//le.ommaatttttssuuuutttott por portuggeee--tweets-for Lugpees--wwer--ffr--eett-tt-----
"ht otctopss/www...aggeeee..///////s ttee--ommmeee----eeeeeewweere-e-
"dados. gov. www.pt//enndaaaaess//otciaaaaa

---

PE2LGP
RB Simple sentences
mBART
M2M
includes the words present in the reference translation but in the wrong order;
BLEU
37.93
41.37
41.53
43.01
Addition and removal of personal pro- nouns: some predicted translations wrongly add/remove personal pronouns;
RougeL. f
59.57
63.83
64.15
67.77
TER 4
114.41
97.82
101.31
96.94
Social Media
BLEU f
28.84
40.43
29.92
37.52
Proper nouns not correctly identified: proper nouns are fingerspelled in LGP, is crucial for the models to identify them, which does not always occur.
RougeL. f
57.43
62.72
54.65
60.61
TER +
206.55
62.66
216.00
174.00
Poetry
BLEU f
18.14
27.07
12.78
18.37
RougeL f
39.44
46.11
40,81
46.07
Mistakes when dealing with the female gender: some female nouns have their own sign and some are translated into MULHER (WOMAN) + MASCULINE SIGN. Some translations fail to identify the appropriate method to be followed.
TER 4
108.42
116
161.55
121.38
Dialogues
BLEU f
21.79
24.49
19.97
29.27
RougeL f
48.69
54.43
51,46
55.35
TER 4
242.14
229.08
358.46
229.08
News
In Appendix A, Table demonstrates instances where our rule-based system outperforms the fine- tuned M2M model, while Table 6 showcases the opposite. Additionally, Table in Appendix A dis-
BLEU T
10.22
16.17
8.60
15.42
RougeL f TER
43.57
50.27
40.49
50.36
77.00
72.12
120.00
74.09
TOTAL
plays sentences where both models yield identical outputs.
BLEU f
23.38
29.91
22.56
28.72
RougeL f
49.74
55.47
50.31
56.03
TER |
149.70
115.54
191.46
139.10
Conclusion and Future Work
Table 3: Results.
We contribute with fully automatic rule-based approach to translate EP to LGP and also with two neural models. We also contribute with automat- ically labeled dataset (37,500 pairs EP/LGP) with texts from 5 different domains and gold collection of 200 sentences (40 from each style). Our rule- based system and the neural M2M model share the podium in all scenarios. We benchmark, in this way, the task of EP to LGP translation.
worse than what was initially thought, and in real- ity, rule-based model had the best TER score in the poetry domain.
is not possible fully determine the best- performing model. However, our rule-based model and the fine-tuned M2M stand out. Within these systems, it clear that the performance depends on the type of sentences to be translated. Specifically. M2M achieved higher results in simpler sentences and in the domain of dialogues, whereas RB excels in the domains of social media and poetry. From able better
For future work. different methods of evaluation should be used to measure fluency and adequacy. To better validate the results, the gold collection should be extended.
Ethics statement: In order to uphold the quality of our work for the end-users, that is deaf people speaking LGP, the gold set was created by LGP experts, who received certification for their work. Furthermore, both the data and code utilized in research are accessible, in order to promote transparency and reproducibility.
this, can infer that the M2M generalize for unseen data for simpler sentence structures. the other hand, the rules extracted from the corpus proved to be more effective for contemporary and poetry sentence styles, aligning with the informal and formal discourse present in the videos within the corpus.
Analyzing the translations from both the rule- based and the M2M models and comparing them with the reference translation, we perceived that the majority of the errors are due to:
Limitations
We identify the following limitations of this work:
The gold set should be extended with more sentences for each domain;
Words misplaced: the predicted translation

---

A more detailed error analysis should be con- ducted, analyzing the idiosyncrasies of each domain:
Richard Csaky and Gábor Recski. 2021. The Gutenberg dialogue dataset. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 138-159, Online. Association for Computational Lin- guistics.
We should test the understandability of the sequence of glosses, even if they are not in the correct order; that is, we should test which errors are critical and which are not.
Paula Escudeiro, Nuno Escudeiro, Rosa Reis, Jorge Lopes, Marcelo Norberto, Ana Bela Baltasar, Ma- ciel Barbosa, and José Bidarra. 2015. Virtual sign-a a real time bidirectional translator of portuguese sign language. Procedia Computer Science, 67:252-262.
A Acknowledgements
We would like to thank the annotators Ana Sofia Fernandes and Ema Marques for generously ded- icating their valuable time and expertise to assist in the development of the gold collection used and provided in this paper.
Angela Fan, Shruti Bhosale, Holger Schwenk. Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond iglish-centrie multi- lingual machine translation. The Journal of Machine Learning Research, 22( ((1))4839-4883
This research was supported by the Portuguese Recovery and Resilience Plan through the project C645008882-0000000 (Center for Responsible AI), and through Fundação para Ciência a Tecnologia (FCT), specifically through the INESC-ID multi-annual funding with reference UIDB/50021/2020.
Rui Ferreira. 2018. PE2LGP 3.0: from European Por- tuguese to Portuguese Sign language. Master's thesis, Instituto Superior Técnico.
Luís Romão Gaspar. 2015. Ie2mmrr--teee au- tomático de fala em língua portuguesa para língua gestual portuguesa. Master's thesis, Instituto Politéc- nico de Leiria.
Matilde Gonçalves, Luisa Coheur, Hugo Nicolau, and Ana Mineiro. 2021. PE2LGP: tradutor de português europeu para língua gestual portuguesa em glosas. Linguamática, 13(1):3-21.
References
Inês Almeida. 2014. Exploring challenges in avatar- based translation from European Portuguese to Por- tuguese Sign Language. Master's thesis, Instituto Superior Técnico.
Inés Lacerda, Hugo Nicolau, and Luísa Coheur. 2023. Towards realistic sign language animations. 23rd ACM International Conference on Intelligent Virtual Agents.
Vasco Alves, Jorge Ribeiro, Pedro Faria, and Luis Romero, 2022. Neural machine translation approach automatic translations between Portuguese lan- guage and Portuguese Sign Language Glosses. In 2022 17th Tberian Conference Information Sys- tems and Technologies (CISTI), pages 1-7. IEEE.
Chin-Yes Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Mourad Brour and Abderrahim Benabbou. 2019. AT- LASLang MTS 1: Arabic text language into Arabic sign language machine translation system. Procedia computer science, 148:236-245.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Fduunv, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Mourad Brour and Abderrahim Benabbou. 2021. AT- LASLang NMT: Arabic text language into Arabic sign language neural machine translation. Journal of King Saud Universit-CCoppute and Information Sciences, 33(9):
Tiago Oliveira, Nuno Escudeiro, Paula Escudeiro, Emanuel Rocha, and Fernando Maciel Barbosa. 2019. The virtualsign channel for the communi- cation between deaf and hearing users. IEEE Re- vista Iberoamericana de Tecnologias del Aprendizaje, 1484):188-199
Pedro Cabral, Matilde Gonçalves, Hugo Nicolau, Luisa Coheur, and Ruben Santos. 2020. PE2LGP anima- tor: A tool to animate a Portuguese Sign Language avatar. In Proceedings of the LREC2020 9th Work- shop the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Chal- lenges and Application Perspectives, pages 33-38, Marseille, France. European Language Resources Association (ELRA).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

---

Rubén San San-Segndoo Barra, Córdoba, L Fernando D'Haro, F Fernández, Javier Ferreiros, Juan Manuel Lucas, Javier Macias-Guarase, Juan Manuel Montero, and José Manuel Pardo. 2008. Speech to sign lan- guage translation system for Spanish. Speech Com- munication, 50(1 1-192)1009--1022
Han Sloetjes and Peter Wittenburg. 2008. Annotation by category: ELAN and ISO DCR. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Associa- tion (ELRA).
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of trans- lation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231, Cambridge, hustachusetts, USA. Association for Machine Translation in the Americas.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the artt natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
Liwei Zhao, Karin Kipper. William Schuler, Christian Vogler, Norman Badler, and Martha Palmer. 2000. A machine translation system from English to Amer- ican Sign Language. In Proceedings of the Fourth Conference the Association for Machine Transla- Technical Pupers, pages 54-67, Cuernavaca, Mexico. Springer.
tion in the Americas:
A Appendix A

---

EP sentences
LGP sentences
Domain
Preciso ir dormir.
EU PRECISAR DORMIR
Simple Sentences
need sleep.)
NEEE SLEEP)
Simple Sentences
Queres uma xícara café?
(XÍCARA CAFÉ TU QUERER
Simple Sentences
(Do you want a cup of coffee?)
((CUP COFFEE YOU ()
Simple Sentences
Ve não é uma minoria
VC MINORIA (NÃO)
Social Media
(You are not minority)
(YOU MINORITY (NO))
Social Media
Até questionei minha sanidade
ATÉ: SANIDADE MHU EU QUESTIONAR
Social Media
even questioned my sanity)
(EVEN SANITY MY QUESTIIO)
Social Media
Cumpres& a tua vida?
(VIDA TEU TU CUMPRIR]<)
Literature
(Do you fulfill your life?)
(ILIE YOUR YOU FULFILL
Literature
E. afaga o pequeno monte
PEQUENO MONTE ELE AFAGAR
Literature
(And strokes the small mound)
SMALL MOUND HE STROKE
Literature
sim, minha senhora.
SIM MEU MULHER SENHOR
Dialogue
(Yes, my lady.)
(YES MY WOMAN SIR)
Dialogue
acho triste a rapariga.
TRISTE MULHER RAPAZ EU ACHAR
Dialogue
(I think the girl is sad.)
SAD WOMAN BOY THINK
Dialogue
Atingiu 192 partilhas
192 PARTILHA ELE ATINGIR
News
It reached 192 shares
192 SHARE REACH
News
Se haveria alternativa aos cortes salariais
SALARIAIS ALTERNATIVA CORTES HAVER
News
(If there was an alternative to wage
(WAGE ALTERNATIVE CUTS THERE IS)
News
cuts)
Table 4: Examples of parallel sentences.
RB
M2M
Reference
NÓS COMER TUDO ACREDI-
COMER TUDO ACREDITAR EU CONSEGUIR (NÀO)
NÓS COMER TUDO ISSO EU ACREDITAR CONSEGUIR (NÃO)
TAR CONSEGUIR (NÃO)
(WE EAT EVERYTHING THAT BE- LIEVE CAN (NO))
(EAT EVERYTHING THAT BE- CAVE (NO))
(WE EAT EVERYTHING THAT I BE- LIEVE CAN (NO))
TAI. DRUS MEU PERMITIR (NÃO)
TAL. DEUS MEU TU PERMITIR (NÃO)
MEU DEUS TAI. PERMITIR (NÃO)
(SUCH GOD MY ALLOW (NO))
(SUCH GOD MY YOU ALLOW (NO))
(MY GOD SUCH ALLOW (NO))
ONDE D'((T--MM) EU MORAR LEMBRAR CONSEGUIR (NÃO)
ONDE DT((T-OM)) EU LEMBRAR MORAR CONSEGUIR (NÃO)
DTTT-O-M) ONDE MORAR EU LEMBRAR CONSEGUIR (NÃO)
(WHERE DI(T-O-M) LIVE RE- MEMBER CAN (NO))
(WHERE DI(T-O-M) RREEEMEER LIVE CAN (NO))
(DI(T-O-M) WHERE LIVE RE- MEMBER CAN (NO))
Table 5: Example sentences where our based model performs better than the fine-tuned M2M model when compared with the reference translation.

---

RB
M2M
Reference
MEIO CAMINHO MULHER AVŌ
MEIO CAMINHO AVŌ MEU
MEIO CAMINHO AVŌ MEU
MEU
(HALF WAY WOMAN GRANDFA-
(HALP WAY GRANDMOTHEER MY)
(HALF WAY GRANDMOTHER MY)
THER MY)
(NOVO PREPARAR ACORDO
(NOVO ACORDO
OSLO
(NÓS NOVO ACORDO OSLO PREPARADO/Qq)
OSLOHQ)
PREPARAR Kq)
(NNWW PREPARE AGREEMENT
((NWW AGREEMENT OSLO PRE- PARE)())
((WE NEW AGREEMENT OSLO PREPARED)Q))
OSLOHQ))
FILME EMOÇÃO RUA POVO ENTERRO DIVT-A-N.-R.--..O))
FILME EMOÇÃO POVO RUA ENTERRO DI(T-A-N.C.R---D-O))) MOSTRAR
FILME EMOÇÃO POVO RUA ENTERRO DICT-ANN-C.....-O)) MOSTRAR
MOSTRAR
(MOVIE EMOTION STREET PEO- P.EE BURIAL DI(T-A-N-C.--F.D-
(MOVIE EMOTION PEOPLE
(MOVIE EMOTION PEOPLE
STREET BURIAL C.F... D-O) SHOW)
STREET BURIAL F.F... DO) SHOW)
O) SHOW)
Table 6: Example sentences where the fine-tuned M2M model performs better than our rule- based model when compared with the reference translation.
RB/M2M
Reference
VOCÊ AJUDAR
VOCÉ AJUDAR ME
(YOU HELP)
(YOU HELP ME)
FOLHA SPOTIFY EHAIA
FOI.HA SPOTIFY FLFIÇÃO CHAPA LANÇAR
LANÇAR
(FOLHA SPOTIFY
ELECTION
(FOLHA SPOTIFY ELECTION
CHAPA LAUNCH)
CHAPA LAUNCH)
PAÍS BENDEGÓ
NÓS PAÍS BENDEGÓ
(COUNTRY BENDEGÓ)
(WE COUNTRY BENDEGÓ)
Table 7: Example sentences where our rule-hased model and the fine-tuned M2M produce the same output, whether it correct or not.


Reviews:
[{"review_id": "MhwHJfmSer", "reviewer": "Reviewer_LV1i", "summary": "", "questions": "Question A: Did the authors consider pre-processing glosses for either lexicalisation or standardisation of what is annotated? (Related references shown below, related to text around line 55)\n\nQuestion B: How many individual signers are included in the COLIN dataset?\n\nQuestion C: What concrete future work is planned, and how will these models improve accessibility and communication to the LGP community?\n\nIn order to have the required specificity for reproducibility, the following shorter questions ask the following:\n\nQuestion D: what manual tasks are needed to create PE2LGP (line 43)? \n\nQuestion E: What linguistic information exactly is extracted from the corpus (line 145)? \n\nQuestion F: What exactly were the computational resources available, were there GPUs (line 194)? \n\nQuestion G: Which methods of calculating BLEU and ROUGE are used (and why)(should be described in Section 4)?\n\nQuestion H: How do you interpret the results of the evaluation metrics, and how do they link to your qualitative results (section 5)?\n\nQuestion I: How were sentences picked for the gold set (section 4.3)? Were they cherry-picked or chosen at random? This could create bias if they were selected with a certain motive in mind, and affect future testing of SLMT models", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for your feedback. \n\nWith respect to your main suggestion, and if the paper is accepted, we will use the extra page to add additional details about the dataset, model parameters, and evaluation metrics.\n\nWe will now proceed to answer your questions:\n- **Question A:** We have not pre-processed the glosses.\n- **Question B:** We do not know how many individual signers are in the COLIN dataset. That information is not available.\n- **Question C:** Concrete future work is to use the current models that generate the sequence of glosses to improve an existing avatar that reproduces the signs associated with the glosses.\n- **Question D:** The manual tasks that were needed to create the PE2LGP are:\n     1. Check if the rules created are grammatically sound (for example: rules that have a Portuguese structure but do not have an LGP one should be discarded);\n     2. Check if the alignment between Portuguese words and LGP glosses makes sense;\n     3. Identify glosses that need to be fingerspelled.\n- **Question E:** The linguistic information that is extracted from the corpus is composed of: the Portuguese sentences, the LGP sentences, and all the grammatical information such as part of speech tags, subjects, and objects.\n- **Question F:** Yes, there were GPUs. Two GPUs with 50GB of memory.\n- **Question G:** For calculating BLEU and ROUGE we used the metrics from the HuggingFace library. The BLEU score used is the BLEU-4 as it is usually the most common one to evaluate translation systems. The ROUGE used is the rougeL that scores based on the longest common subsequence. We will clarify this in the paper.\n- **Question H:** After obtaining the results of the automatic evaluation metrics, we manually compared the reference and the predicted translations and concluded what were the main problems, which we briefly describe in the paper. Although not mentioned in the paper, more specific metrics might be needed to effectively evaluate the system (for example: evaluate proper nouns and female gender not correctly identified).\n- **Question I:** The sentences for the gold set were randomly picked from the 5 domains described in Section 4.1. We will clarify this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a large body of data including a small gold-standard, expert annotated test set to facilitate future work on machine translation and computational linguistic studies into LGP. This is of benefit to the CL and NLP community, especially as LGP is a particularly low resource language and on behalf of a historically marginalised community.\n\nThe methodology is mostly described well - and though missing some specifics in the formulation of rules, model calibration, and evaluation metrics - it should be relatively straightforward to attempt to reproduce the findings in this paper. This is also in part due to the commitment of the authors to release data and code as open source.\n\nThe methodology itself is also robust and suitable for the resources available and novelty of the task. A rule-based approach is a relevant next step given the work it builds upon (in both LGP and San-Segundo's approach in LSE), and adding neural/hybrid experiment shows a commitment to test recent innovations on this data.\n\nThe authors also comment on the limitations of the approach taken and contents of the data, and it is hoped that future research will tackle these.\n\nIn my opinion, this short paper is suitable for publication with the integration of some of the suggestions and answering of some of the questions in this review.", "reasons_to_reject": "Overall, the layout of the paper and its contents are adequate. However, the most serious flaws are found in the experimental results sections.\n\nEvaluation metrics are briefly referred to in the text and are included in the Table, but they are not introduced or described in the text. It is necessary to know for example which type of BLEU (BLEU-4/sacreBLEU?/ROUGE-F1? see also references section) are used, and a more clear description of what higher/lower scores may mean or imply. The description of these results can be rather shallow in that certain settings and metrics are described as \"better\" than others. There is no further explanation or theoretical discussion about what these results imply and what could be done to mitigate them. \n\nMy main suggestion for improvement in this regard is to use the extra page allowed post-review to focus on being more specific in dataset, model parameter, and evaluation metric description. In addition, a lengthier discussion of results would ground this study in its research context in a helpful manner for the reader.\n\nAnother small comment is that the SoTA section (lines 122-136) does not mention many key works in the history of SLMT. A greater discussion of relevant technologies - such as Text2Gloss, RB vs. statistical vs. neural, or E2E vs. modular SLMT - where it references the present study would be of great benefit.", "questions_to_authors": "Question A: Did the authors consider pre-processing glosses for either lexicalisation or standardisation of what is annotated? (Related references shown below, related to text around line 55)\n\nQuestion B: How many individual signers are included in the COLIN dataset?\n\nQuestion C: What concrete future work is planned, and how will these models improve accessibility and communication to the LGP community?\n\nIn order to have the required specificity for reproducibility, the following shorter questions ask the following:\n\nQuestion D: what manual tasks are needed to create PE2LGP (line 43)? \n\nQuestion E: What linguistic information exactly is extracted from the corpus (line 145)? \n\nQuestion F: What exactly were the computational resources available, were there GPUs (line 194)? \n\nQuestion G: Which methods of calculating BLEU and ROUGE are used (and why)(should be described in Section 4)?\n\nQuestion H: How do you interpret the results of the evaluation metrics, and how do they link to your qualitative results (section 5)?\n\nQuestion I: How were sentences picked for the gold set (section 4.3)? Were they cherry-picked or chosen at random? This could create bias if they were selected with a certain motive in mind, and affect future testing of SLMT models", "paper_topic_and_main_contributions": "This short paper introduces new parallel resources for Portuguese Sign Language (which is an extremely low resource language) and Portuguese (Portugal): These include a considerable corpus as well as an expert-annotated gold-standard test set which could provide an evaluation benchmark for other work on LGP.\n\nIt also uses these resources to conduct experiments in Sign Language Machine Translation (SLMT), restricted to the text-to-sign language gloss direction. Two transformer-based neural models are tuned for the task, as well as a rule-based (RB) model which automates a process invented in a previous work (Gonçalves et al., 2021). The authors also compare this previous RB model on the same testing data in order to provide comparability between studies against a number of metrics.\n\nThe authors also select some qualitative output from each model to be shown in an Appendix.\n\nIt is envisaged that the dataset and code used to generate and evaluate the results will become available after the period of anonymity in order to allow reproducibility and verification of results/method.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Cite BLEU (whichever method was used), TER and ROUGE papers!\n\nOn historical progress in SLMT:\n\"A survey on Sign Language machine translation\" (Núñez-Marcos et al., 2023)\n\nT2G and G2T in the neural era:\n\"Sign Language Transformers: Joint end-to-end Sign Language Recognition and Translation\" (Camgöz et al., 2018)\n\"Approaching Sign Language Gloss Translation as a Low Resource Machine Translation Task\" (Zhang and Duh, 2021)\n\nA different hybrid RB/NMT approach:\n\"Translating Spanish into Spanish Sign Language: Combining Rules and Data-driven approaches\" (Chiruzzo et al., 2022)\n\nStandardisation of SL datasets and what is annotated:\n\"Digging into Signs: Emerging Annotation Standards for Sign Language Corpora\" (Cormier et al., 2016)\n\"Universal dependencies for Swedish Sign Language\" (Östling et al., 2017)", "Typos_Grammar_Style_and_Presentation_Improvements": "It should maybe be made more clear in the title and key summary sections that this is text-to-SL Gloss translation (and this unidirectional translation may be a limitation of use). This is to compare it with bidirectional systems, but also with E2E systems such as text->SL avatar or SL video->text which have very different implications in the real-world applications of this technology.\n\nPerhaps refer to European Portuguese by the more standard ISO reference \"pr-PT\" instead\n\nLine 113: Automatize -> Automate\n\nLine 141: Suggest \"to reconstruct the expert translator's grammatical rules\"\n\nNumbers <= twelve to be written in text in the main body of the text\n\n35000 -> 35,000\n\nLine 196: Sentence could be rephrased for clarity\n\nLine 217: (tone) \"famous Portuguese poet\" instead\n\nLine 267: Use \"fingerspelled\" instead\n\nTable 3: Upwards and downwards arrows next to the metric names to state whether a higher or lower figure is desirable\n\nTable 2/3: Make sure all numbers have either an equal number of digits after the decimal point/equal number of significant figures\n\nTable 3: Wrong TER is bolded in \"News\"\n\n\n"}}, {"review_id": "I0LU37B7Qa", "reviewer": "Reviewer_Ue8D", "summary": "", "questions": "Why was the gold collection created in such way (pre-annotated by rule-based system, separate annotation sets)? Isn't this method correlated with better results of the ruled based system?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for your feedback.\n\nThe gold collection was previously annotated by the rule-based system in order to facilitate the annotators' job. If they agree with the results they do not change them; otherwise, they will correct them. In such approaches to annotation, there is always some risk of \"bias\" towards the initial annotation. However, we do not think this has affected the results.\n\nIf the paper is accepted, we will use the extra page to add additional details about the evaluation methodology and metrics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important and challenging problem of translating between European Portuguese and Portuguese Sign Language. \n2. The paper's approach presents several new systems and their evaluation.\n3. The paper introduces the new dataset, comprising sentences from diverse domains.", "reasons_to_reject": "1. The major problem I have with this paper is its evaluation methodology on the newly created gold set. Having a gold collection annotated in parallel on a plain (not pre-annotated) text and calculating inter-annotator agreement would make it more reliable.\n2. The metrics used may not fully capture the quality of translation, especially for sign language. Additional metrics, such as fluency, adequacy, and subjective human evaluations, already started and presented in Appendix 1, could provide a more comprehensive assessment.\n3. The paper briefly mentions common errors made by the models but does not go into detail regarding the root causes or potential strategies to mitigate them.\n4. The paper mentions that the COLIN corpus includes signers ranging from 4 to 89 years old, but we don't know how it relates to the LGP corpus. ", "questions_to_authors": "Why was the gold collection created in such way (pre-annotated by rule-based system, separate annotation sets)? Isn't this method correlated with better results of the ruled based system?", "paper_topic_and_main_contributions": "The paper addresses the challenge of translating European Portuguese into Portuguese Sign Language. The authors create a rule-based translation system, a 5-domain parallel corpus between these languages and fine-tune two large multilingual neural machine translation models, mBART and M2M. All proposed models are evaluated on a previously available evaulation set and a new 5-domain newly annotated gold set. \n\nThe evaluation shows that the M2M-based method achieves the best overall performance on the original evaluation set while the rule-based system achieves the highest results on the new evaluation set. The qualitative evaluation shows some problems related to specific language constructs.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 039: from now on COLIN – COLIN should be explained here (is in line 104); moreover, it seems there is no citation/URL of COLIN anywhere\n- 053: evaluated in a gold collection > on a gold collection?\n- 055: we hope to contribute to benchmark this task > we hope to contribute to benchmarking this task \n- 056: To illustrate the task in hand > at hand\n- in footnote 2: Developed in project > Developed in the project \n- 094: afterward > afterwards\n- 165 and later: please consider using dashes instead of minuses when sequences of elements in syntactic structures are enumerated\n- the readability of the equations should be improved a bit (e.g. 'NP' and 'VP' should be separated a bit more than 'N' and 'P' in 'NP'; arrows should be arrows)\n- 166: a LGP sentence  > an LGP sentence \n- 202: 7500 times 5 domains gives 37500; why do we have 35k here?\n- 262: in a wrong order > in the wrong order\n- in Table 3: 49,74 > 49.74; 29.906 > 29.91\n- in the bibliography: please check the case of letters to avoid lowercase names (e.g. 'european portuguese').\n- please reformat all tables to conform to EMNLP 2023 style!\n- why aren't the tables in the appendix on the same page as its heading?\n\nIn Figures:\n- Figure 1 has a caption-ending dot while Fig. 2 and 3 don't – please make it consistent with EMNLP 2023 guidelines\n- maybe 'results' in the caption of Figure 2 should be lowercase?\n- please consider right- and dot-aligning the numbers, the results may read better"}}, {"review_id": "7d0C0FJH3C", "reviewer": "Reviewer_9TWC", "summary": "", "questions": "- For the sake of the paper, you *must* explain in more detail how the rules are automatically extracted. This process has to involve some sort of word aligment procedure between EP and LGP in order to extract the bilingual dictionary, right? How are the syntactic rules built? Do you have a set of hand-crafted templates?\n- It would strengthen the paper and make it more attractive to a general non-Portuguese audience if it became clear that the rule extraction mechanism is sufficiently general to be applied to other languages, as long as they can provide a suitably annotated text-sign parallel corpus. Can you say something about this?\n- As you state in 106-108, COLIN consists of 113 hours, only 20 of which are annotated. From those 20 hours, you were only able to use the 45 minutes that \"had all the needed syntactic annotations\". Can you briefly point out what you need and what's the annotation in the remaning 19h15?\n- It would be nice to have some discussion on how the rule-based system behaves if its input comes from automatic parsing, and not from what I presume are gold parses. In \"actual use\", a text will be automatically parsed before translation, without the parses being manually corrected.\n- I don't know what ELAN is (line 108). Either briefly explain what it is, or leave it out, as it isn't that relevant to know what is the particular annotation formalism you're using, only that it concerns syntax.\n- Example (3): What happens to ADJ1 and ADJ2? Where does N2 come from? From the example sentence you provide, the rule has to involve more transformations than what at first seems.\n- What is the TER metric? I'm assuming it's some sort of word error rate, but it would help to have a short sentence just quickly stating which metrics you're using.\n- DT() indicates that the word is to be spelled out, right? Isn't this redundant with the word being represented with characters separated by hyphens, e.g. \"A-N-A\", or is there a reason for the two notations?\n- Table 4, third example: Is \"Vc\" (short-form for \"Você\") really translated into \"VC\" in LGP, implying that there a different gesture than that for \"VOCÊ\"? Or is \"VC\" spelled out. In that case, shouldn't it be \"DT(V-C)\"?\n\n[these questions were mostly addressed in the rebuttal]", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for your feedback.\n\nWe will now proceed to answer your questions:\n- Questions regarding the rule-based system:\n     - **Q1:** The process to automatically extract the rules starts by extracting the Portuguese sentences, the LGP sentences, and the grammatical information such as part of speech tags, subjects, and objects from ELAN. The next step is to analyze the Portuguese sentences to obtain their grammatical information. Having the information regarding both languages, the alignment between the Portuguese words and the LGP signs is performed. This alignment is based on an algorithm of similarity measures, string matching, and semantic similarity. From the aligned word-sign pairs, the rules are created. We do not have a set of hand-crafted rules. We only have the information extracted from ELAN.\n     - **Q2:** The rule extraction mechanism is sufficiently general to be applied to other languages as long as a suitably annotated text-sign parallel corpus is provided.\n     - **Q3:** From the 20 hours, only 45 minutes have all the needed information to create the rules. This information consists of the Portuguese sentences, the LGP glosses, the type of sentence, the part of speech tags, and the syntactic analysis. The other 19h15 still do not have all this information.\n     - **Q4:** We are not sure that we understood the question. The parsing is always automatic. Even the one that is on the basis of the gold collection. Only the gold collection received manual correction/validation by the experts (in order to be created).\n- Questions regarding clarifications:\n     - **Q5:** ELAN is a software tool that enables the annotation of audio and video data. ELAN allows users to create multiple layers of annotations, which are synchronized and aligned with the corresponding audio or video segments.\n     - **Q7:** The TER metric returns the minimum amount of operations needed to convert the predicted translation into the reference one - includes insertion, deletion, and substitution of words and shifts within the sentence. The lower the value, the better.\n- Questions regarding annotations' clarifications:\n     - **Q6:** For the Portuguese sentence:\n                    \n         \"Há (V1) grandes (ADJ1) desenvolvimentos (N1) artísticos (ADJ2)\"                   \n          (There are (V1) great (ADJ1) aristic (ADJ2) developments (N1)),\n                      \n          the corresponding sentence in LGP glosses is:\n                      \n          \"TER-MUITO (V1) ARTE (N2) DESENVOLVIMENTO (N1)\"      \n          (HAVE-A-LOT (V1) ART (N2) DEVELOPMENT (N1)).\n                      \n          This occurs because “There are great”  is converted into only one sign “HAVE-A-LOT” which corresponds to the Verb (V1).  “artistic” is converted into “ART” which is a noun that was not present in the original sentence and is thus assigned a new number (2). Taking this into consideration, the ADJ1 and ADJ2 are removed from the sentence.\n     - **Q8:**  The annotation for the words that are fingerspelled is the annotation used by PE2LGP (defined by experts in LGP) and therefore used by us.\n     - **Q9:** The example \"Vc\" represents an error where the system cannot understand that it should generate the sign \"VOCÊ\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses a rarely seen but important task.\n- The rule extraction system might work for any text-sign parallel corpus (with \"sign\" understood as being glosses), as long as the text is syntactically annotated, which makes it more general than what is covered by this limited study.\n- Is enough content for a short paper.", "reasons_to_reject": "- There's very, very little information on how the fully automatic rule-based system works. While it's true that this is a short paper, the automatic rule extraction is the *most important* part of the paper in my view, as it enables all the rest, so not properly explaining how it works is a major weakness.\n\n[even with the rebuttal, much is still unclear regarding the workings of the rule-extraction procedure]", "questions_to_authors": "- For the sake of the paper, you *must* explain in more detail how the rules are automatically extracted. This process has to involve some sort of word aligment procedure between EP and LGP in order to extract the bilingual dictionary, right? How are the syntactic rules built? Do you have a set of hand-crafted templates?\n- It would strengthen the paper and make it more attractive to a general non-Portuguese audience if it became clear that the rule extraction mechanism is sufficiently general to be applied to other languages, as long as they can provide a suitably annotated text-sign parallel corpus. Can you say something about this?\n- As you state in 106-108, COLIN consists of 113 hours, only 20 of which are annotated. From those 20 hours, you were only able to use the 45 minutes that \"had all the needed syntactic annotations\". Can you briefly point out what you need and what's the annotation in the remaning 19h15?\n- It would be nice to have some discussion on how the rule-based system behaves if its input comes from automatic parsing, and not from what I presume are gold parses. In \"actual use\", a text will be automatically parsed before translation, without the parses being manually corrected.\n- I don't know what ELAN is (line 108). Either briefly explain what it is, or leave it out, as it isn't that relevant to know what is the particular annotation formalism you're using, only that it concerns syntax.\n- Example (3): What happens to ADJ1 and ADJ2? Where does N2 come from? From the example sentence you provide, the rule has to involve more transformations than what at first seems.\n- What is the TER metric? I'm assuming it's some sort of word error rate, but it would help to have a short sentence just quickly stating which metrics you're using.\n- DT() indicates that the word is to be spelled out, right? Isn't this redundant with the word being represented with characters separated by hyphens, e.g. \"A-N-A\", or is there a reason for the two notations?\n- Table 4, third example: Is \"Vc\" (short-form for \"Você\") really translated into \"VC\" in LGP, implying that there a different gesture than that for \"VOCÊ\"? Or is \"VC\" spelled out. In that case, shouldn't it be \"DT(V-C)\"?\n\n[these questions were mostly addressed in the rebuttal]", "paper_topic_and_main_contributions": "This paper presents a study on automatic translation between European Portuguese (EP) and Portuguese Sign Language (LGP).\n\nThe authors developed a fully automatic rule-based system that, given a parallel EP-LGP corpus, with syntactically annotated EP, extracts syntactic and morphosyntatic rule that translate syntactically annotated EP into LGP sentences (sequences of glosses, with some annotation for spelling-out words, signaling interrogatives, etc).\nThis rule-based system is used to automatically create a parallel EP-LGP corpus with 35k sentences, which is used to train two NMT systems (a mBART model and a M2M model).\nEvaluation of the rule-based and neural systems is performed on a held-out gold corpus of 200 sentences that has been manually validated by LGP experts.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- What's the purpose of the preliminary experiment you show at the start of Section 5 in Table 2? You've just ended Section 4 by describing the gold corpus you've created, but then start the evaluation with a totally different corpus, not introduced before, before moving on to the evaluation over your gold corpus (which is the one we're interested in). I'd leave this preliminary experiment out of the paper and use the space you gain to better explain the automatic rule extraction.\n- 030: \"attempts for developing\" -> \"attempts at developing\"\n- 056: \"task in hand\" -> \"task at hand\"\n- 094: \"afterward\" -> \"afterwards\"\n- 100: Add a linebreak as you change topic and start describing PE2LGP.\n- 180: \"Portuguese-to-Portuguese\" -> Perhaps \"Portuguese-to-LGP\" instead?\n- The paper would benefit from some reformating of the examples in the various tables, for the sake of readibility, as the all-caps characters and fully justified paragraphs aren't easy to read. Perhaps using a smaller font, small caps, or something else.\n- Table 7: Since you're showing cases where RB and M2M match, you don't really need three columns. Two are enough: one for RB/M2M and one for Reference."}}]

Author rebuttal:

Meta-review: This short paper introduces new parallel resources for Portuguese Sign Language (which is an extremely low resource language) and Portuguese (Portugal): These include a considerable corpus as well as an expert-annotated gold-standard test set which could provide an evaluation benchmark for other work on LGP.

Reasons To Accept:
- This paper presents a large body of data including a small gold-standard
- The methodology is mostly described well 
- The methodology itself is robust and suitable for the resources available and novelty of the task
- Addresses a rarely seen but important task

Reasons To Reject:
- Having a gold collection annotated in parallel on a plain (not pre-annotated) text and calculating inter-annotator agreement would make it more reliable.
- The metrics used may not fully capture the quality of translation, especially for sign language. Additional metrics, such as fluency, adequacy, and subjective human evaluations, already started and presented in Appendix 1, could provide a more comprehensive assessment.
- There's very, very little information on how the fully automatic rule-based system works. While it's true that this is a short paper, the automatic rule extraction is the most important part of the paper in my view, as it enables all the rest, so not properly explaining how it works is a major weakness.


In my humble opinion, this work is very important to develop tools for sign language. The golden dataset is very precious and this gift to the community is really significant.
Decision: Accept-Findings