BERTie Bott's Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician
Authors: Micaella Bruton
Meriem Beloucif
Venue: EMNLP 2023 Main (2023)
Keywords: semantic role labeling
semantic parsing
Galician
Spanish
srl
TL;DR: Introducing the first dataset and semantic role labeling model for Galician, and a new pre-processing method to improve semantic role labeling

Abstract: In this paper, we leverage existing corpora, WordNet, and dependency parsing to build the first Galician dataset for training semantic role labeling systems in an effort to expand available NLP resources. Additionally, we introduce verb indexing, a new pre-processing method, which helps increase the performance when semantically parsing highly-complex sentences. We use transfer-learning to test both the resource and the verb indexing method. Our results show that the effects of verb indexing were amplified in scenarios where the model was both pre-trained and fine-tuned on datasets utilizing the method, but improvements are also noticeable when only used during fine-tuning. The best-performing Galician SRL model achieved an f1 score of 0.74, introducing a baseline for future Galician SRL systems. We also tested our method on Spanish where we achieved an f1 score of 0.83, outperforming the baseline set by the 2009 CoNLL Shared Task by 0.025 showing the merits of our verb indexing method for pre-processing.

BERTie Bott's Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician
Micaella Bruton
Meriem Beloucif
Uppsala University
Uppsala University
micaella. bruton. 4363@student. uu. se meriem. www.oclilloloooiiiiiiiii
Abstract
participant with respect to the verbal root, while Arg/ represents that which undergoing a change of state or causally affected by another participant (Jurafsky and Martin, 2023).
In this paper, leverage existing resources, such as WordNet and dependency parsing, to build the first Galician dataset for training se- mantic role labeling systems in an effort to expand available NLP resources. Addition- ally, we introduce Verbal Indexing, new pre- processing method, which helps increase the performance when semantically parsing highly complex sentences. We use transfer learning to test both the resource and the Verbal Index- ing method. Our results show that the effects of Verbal Indexing were amplified in scenarios where the model was both pre-trained and fine- tuned on datasets utilizing the method, but im- provements are also noticeable when only used during fine-tuning. The best-performing Gali- cian SRL model achieved an score 0.74, introducing baseline for future Galician SRL systems. We also tested our method on Spanish where we achieved fl score of 0.83, outper- forming the baseline set by the 2009 CoNLL Shared Task by 0.025, showing the merits of our Verbal Indexing method for pre-processing.
argO
root
argl
glg eng
Touriño
dá
un
xiro
Touriño
takes
turn
Table 1: Introduction to SRL: Excerpt from Galician Dataset Sentence #66
Il glg Galician text eng- English text argo token assigned as Argument noo - token assigned as the verbal root argl token assigned as Argument
Table contains an excerpt from the Galician dataset introduced in this study alongside an En- glish translation to showcase the identification of roles, or arguments, within sentence. As can be seen in the table, the main verbal root in this ex- cerpt is dá or takes, Touriño is identified as Arg0 as they are performing the action described by the verb, and xiro turn is identified as Argl it is that which being affected by ArgO.
1 Introduction
Unfortunately, corpora including this shallow semantic information are extremely few and far between, with .1% of the languages of the world having the necessary language-specific information readily available and only .3% having access the translated version of English PropBank, the most extensive SRL resource currently available (Eberhard et al., 2023; Palmer et al., 2005; Jindal et al., 2022; Akbik et al., 2015). This lack of data makes it nearly impossible for SRL systems to be developed for the vast majority of languages.
The goal of semantic role labeling (SRL), also known as shallow semantic parsing, is to define the semantic roles of any given word in sentence in regard specific noun verb; the focus of this study specifically is SRL in regard verbal predicates (Jurafsky and Martin, 2023). Focus is placed on directly mapping a predicate to its role and recipients (Marquez et al., 2008). Computa- tional systems are then able to use these roles to establish a shallow semantic representation of a string of words, allowing for inferences which oth- erwise would be lost to the system (Jurafsky and Martin, 2023). As these roles seek to remove ambi- guity as much possible, they are specific each verb's sense meaning, though tendencies can be identified. In general, ArgO represents that which causes an event or a change of state in another
The main purpose of this work was to produce a dataset and initial baseline for Galician SRL, a low- resource Western Ibero-Romance language with ap- proximately 4 million speakers worldwide (Britan- nica, 2023). Despite its large number of speakers, NLP tools and resources for Galician are extremely limited and remains classified as a low-resource

---

Tokens (eng)
Touriñoo
takes
turn
in
his
campaign strategy
Tokens (glg)
Touriño
dá
un
xiro
a
súa
estratexia
de
campaña
ID Tags
4
0
0
0
0
0
0
Roles
rr:arg0
10:root
rO:arg
0
0
0
0
Table 2: SRL of Galician Dataset Sentence #66
eng- - English text glg Galician text r/idx]:[label] link [label] to [idx]th verb root token assigned as the
verbal root arg0 token assigned argument argl token assigned argument I token assigned as not directly related to root verbll
language for most NLP tasks; it is our hope that the release of this dataset will help to improve and expand resources for Galician (Language Archives Services, University of British Columbia, 2023; Agerri et al., 2018).
corpora exist outside of that produced by this work. Syntactic information contained in two of these resources, the UD Galician-CTT and UD Galician- TreeGal treebanks, were leveraged to create a sim- plc SRL datasct for Galician (Gómez-Rodrigeee et al., 2017; Alonso et al., 2016). The UD Galician- CTG treebank contains technical sentences of the medical, sociological, ecological, economical, and legal domains that have been pulled from the Gali- cian Technical Corpus and automatically parsed by the TALG NLP Research Group at the University of Vigo (Gómez-Rodrizee et al., 2017). The UD Dalician-TTeeCCC treebank was developed at the LyS Group at the Universidade de Coruña and was derived from subset of the XIADA corpus created at the Centro Ramón Piñeiro para Investigación Humanidades (Alonso al., 2016).
Several transfer-learning experiments were also performed to determine the efficacy of leveraging existing high-resource SRL systems in the devel- opment of languages with lesser amounts of data available as it has seen success previously (Oka- mura et al., 2018; Alimova et al., 2020; Oliveira et al., 2021; Daza Arévalo, 2022).
2 Related Work
Recently, methods such as Descriptive Semantic Role Labeling (DSRL) and the inclusion of syn- tactic information have been proposed in effort increase interpretability, flexibility, and overall performance of SRL systems (He al., 2018; Co- nia et al., 2022). DSRL seeks to shift away from traditional argument labeling in favor of labeling through descriptions using natural language (Conia et al., 2022). Both methods have seen great success, performing par with or beating current state-of- the-art results in high-resource contexts for English and Chinese (He et al., 2018; Conia et al., 2022). Despite their success, a choice utilize standard methods of argument labeling was made due to the extremely low-resource status held by Galician and the overall current lack of resources.
These datasets were downloaded from their re- spective project GitHub in CoNLL-U format and processed via a Python script. For each unique sentence in these corpora, the total number of root verbs were assigned an index which was then used to match arguments their respective verb. Using the NLTK library for Galician WordNet, a synset for each verbal lemma was identified and matched to its English PropBank role set (Bird et al., 2009; Guinovart and Portela, 2018; Palmer et al., 2005). This information was then used to assign one of five roles to each token: "roidx):root", "r[idx]:argO", "r[idx]:arg"" "r[idx]:arg" or "O"; where idx des- ignates the verb index and O designates involved token. An attempt to identify additional arguments or non-verbal roots was not made at this time, and verbal lemmas unable to be linked to a single, specific English PropBank role were not included.
3 Method
Development and training scripts can be found on the project G while the fully trained models and datasets can be accessed via HuggingFace".
Table 2 gives example of a sentence with root verb and its arguments; for each additional root appearing in a sentence, the index number will increase by one. Figure/ showcases how deci- sions were made throughout the automated argu- ment classification process; Upon completion, 25%
3.0.1 Galician
While Galician does have publically available WordNet and syntacticallly-paree corpora, no SRL
Thapss//hugggngffacccommmmbbbb

---

of sentences were randomly selected for manual validation. In cases where a clear distinction was not able to be made, the sentence was removed from the dataset. The final dataset contains 3,987 training sentences and 998 test sentences. Sen- tences in the dataset include up to 13 verbal roots, with arguments for each root varying between arg0, argl, and arg2.
The Verbal Indexing method proposed here dif- fers from basic direct mapping as it involves tally- ing the number of verbs present a given sentence (Marquez et al., 2008). This is done to enhance the ability of the model to understand the sentence s complexity and aid in the initial identification of verbs within the sentence. This method does not assign semantic roles or rulesets; the purpose of Verbal Indexing is to count all verbs within sen- tence, arranging them based on their position in the sentence. Importantly, this indexing process does not impact the PropBank label associated with each verb. For instance, even if a verb with the Prop- Bank label run.01 appears twice in a single sen- tence, each instance would be assigned a distinct index through the Verbal Indexing approach, as the index is directly linked to the verb's 'ss position in the sentence.
3.0.2 Spanish
The 2009 CoNLL Shared Task Spanish data was used to develop the Spanish SRL dataset; it con- tains 14,329 training sentences, 1,655 development sentences, and 1,725 test sentences (Hajič et al., 2009). Sentences were originally extracted from the AnCora-ES corpus, developed by the Centre de Llenguatge i Computació de la Universitat de Barcelona which were taken from newspaper ar- ticles (Rodríguez et al., 2004). Pre-processing of the data was done to implement Verbal Index- ing; additional thematic information included in the Spanish data was preserved. Sentences in the dataset include up to 16 verbal roots and the fol- lowing role labels: "rlidx]:root", "rrio I lagt, cau, exp, srel". "r[idx]:argl [ext, loc, pat, tem]", :arg2 [atr; ben, eft, exp, ext, ins, loc]", "rridx]:arg3 I [ben, ein, fin, ori]", "r[idx]:arg4 [des. effl", and "r[idx]:rrMM [adv, atr, cau, ext, fin, ins, loc, mar, tmp]".
Monolingual models were fine-tuned for the SRL task in the target language on the base ver- sion of the chosen language model. Transfer- learning models were previously trained on the SRL task in either English, Portuguese, Span- ish (for Galician) or some combination of these, then fine-tuned for the SRL task in the target lan- guage. Models are named using the following format, Lunget-Stt-lannuuge___peeee [prerraine--SRL- languagel
where the tar- get language is identified by a 3-letter identifier, the pre-trained language(s) are identified by a two- letter identifier, and the base language model is identified by its name.
3.1 Models
24 models in total were developed and tested, 16 Galician and 8 Spanish SRL models. To our knowl- edge, no SRL work on Galician has been published and our results introduce an initial baseline for fu- ture work. Due to this, Spanish models were de- veloped not only as an additional transfer-learning scenario for Galician but also to test our method against an established baseline. Results from the 2009 CoNLL Shared Task Spanish SRL-only mod- els are used as this baseline (Hajië al., 2009). Pre-trained language models mBERT and XLM-R were used as main architectures and fine-tuning followed the same basic structure for all models (Devlin et al., 2019; Conneau et al., 2020). As each sentence goes into model, it is tokenized and input all at once. The model is then expected make predictions as the role of each token within the sentence; additionally, the model is ex- pected to assign each verbal root an index and link each argument to its root via this index. Specific arguments vary by language as defined above.
To reduce computational and environmental costs, English and Portuguese pre-trained SRL models were adopted from HuggingFace and used as published by Oliveira et al. (2021). English models were pre-trained using the OntoNotes v5.0 English SRL corpus, including 316. 155 sentences from news, conversational telephone speech, we- blogs, Usenet newsgroups, broadcast, and talk shows while the Portuguese models used data from the PropBank-BR Brazilian Portuguese SRL cor- pus, including 13.665 sentences taken from news- papers (Duran and Aluísio, 2011; Weischedel et al., 2013; Oliveira et al., 2021). As these models were used as-is, they were not fine-tuned using our Ver- bal Indexing method, but rather (OB-sequencing in the format [,O,B]-[label]] (Oliveira et al., 2021).
After models trained to convergence, they were evaluated on their overall performance via fl score; their overall ability to identify verbal roots and their

---

Is the token verb?
"roidx]:roof"
Is the head of the token verb?
What is the deprel between token & head?
"0"
nsubj
obj Al possible argument of the root verb?
obl
iobj
other
AO possible argument of the root verb?
A2 possible argument of the root verb?
"O"
"rridx]:rr00" "0"
"r[idx]:argT"
"O"
"rrid "0"
What is the xpos of the token?
NCFP000/Zgms
NCMP000/Scfs
NCMS000
NCFS000/Inff
other
AO possible argument of the root verb?
AI possible argument of the root verb?
A2 possible argument of the root verb?
"O"
"rlidx]:arg0" "0"
"rril" "O"
"rlidx :arg2" "O"
A2 possible argument of the root verb?
"rlidx]:arg2"
AO possible argument of the root verb?
"rlidx]arrg(" "O"
Figure 1: Argument Classification Decision Tree
green line affirmative decision red line negative decision r/idx):/label] format of label as- signed to arguments, link [label] to fidx]th verbal root root assigned as the verbal arg0/A0 assigned argument 0 I //A/ assigned as argument 1 arg2/A2 assigned as argument 2 O assigned as not directly related to root deprel - dependency relation nsuhj - nominal subject ohj - object ob! oblique iobj indirect object xppss language specific part-of-speech tag

---

arguments viafl score: and their ability to index verbal roots and match them to their respective ar- guments fi score, using the Python framework seqeval (Nakayama, 2018). Scores are reported on a scale of 0.0-1.0.
these models all tie for best performing model over- all with score of 0.83. spa_sssaaa___s_o__eooooooo mBERT receives the top score of 0.51 when identifying roots and ties with the pre-trained Portuguese XLM-R model, spa_pt_Xip ppa_pt_XIMMR, with a top score of 0.39 when identifying argO. spa_pt_ spa_Pt_XLM-R XLM-R also produces top-level results when identifying argl, arg2, and argM with scores of 0.39, 0.36 and 0.30 respec- tively. The English pre-trained XLM-R model, spa_en_ spa_en_XLM-R, outperforms spa_pt_ spa_pt_XLM PL_MMM-RR in the identification of arg3 and arg4 with an increase of 0.04 and 0.07 correspondingly.
4 Results
Results are summarized across four tables, two for each language. Overall fl score and score by label are presented for Galician models in Table 3 and Spanish models in Table 6. fl score by Verbal Indexing set are shown in Table 4 for Galician and 7 for Spanish. Verbal Indexing sets are identified in the table as "r[idx]", where [idx] includes the average /l score for the identification of the verbal root and all arguments of the given index.
spa_en_ en_XLMMR also receives the most top scores for Verbal Indexing, achieving scores of 0.573, 0.499, 0.469, and 0.306 for the 1st, 2nd, 3rd, and 7th root sets. The monolingual XLM-R model outperformed all other identifications of the Oth root set by 0.006 or more, but had the worst per- formance for most sets past r3. All models but the duel pre-trained mBERT model, pia_entt_mBERT were able to identify up to 9 root sets, albeit poorly. No model was able to identify any part of root set r10 or greater. None of these models were pre- trained using the Verbal Indexing method, and such all models stop identifying at the 9th root set. Table 5 contains the fl scores for Spanish SRL models published by the 2009 CoNLL Shared Task, as well as the results from the top performing Span- ish model produced by this study, listed under Bru- ton.
4.1 Galician Models
In general, XLM-R models outperform mBERT models; even considering just the monolin- gual models, gal_ eal_XLM-R model outperforms gal MBERT model by 0.02. Transfer-eerning models pre-trained on Spanish tend to perform better, with the XLM-R model pre-trained on En- perform-
glish, then Spanish, pal_ensp_XLMM ing best overall; achieving a score of 0.74 for a total improvement of +0.06 over the worse per- forming monolingual model. Despite its top over- all score, it slightly underperforms in the identifi- cation of verbal roots, arg0 and argl; the XL.M- R model pre-trained on Portuguese, then Span- achieves the top score for identifying verbal roots and argl and the XLM-R model pre-trained solely on Spanish, gal_ val_sp_XLM XLM- R. achieves the top score for identifying argO.
5 Discussion
ish, gal_ptsp_XLLMee
Based on the performance of the monolingual mod- els, XLM-R clearly outperforms mBERT as base architecture in the low-resource scenario. As both architectures are pre- pre-trained on the same, unrelated task, have the same number of layers, hidden state, and attention heads, and include Galician and Span- ish unlabeled text in their pre-training languages, this performance boost appears be influenced by the difference in tokenization methods utilized by each architecture; mBERT uses WordPiece while XLM-R uses SentencePiece.
When considering identification by root set how- ever, the top performing model, gal__aggla Le__ee_s_________ scores just 0.01 under gal__tsp__L____ XLM-- at 0.79 when identifying the root and arguments for Oth root but maintains the top score for roots and argu- ments for the 1st through 5th sets; it is also the only model able to identify any portion of the 9th root set. In fact, all models not pre-trained on the Verbal Indexing method were unable identify anything past the 6th root set.
WordPiece is an algorithm that breaks down words into sub-words using a variation of Byte- Pair Encoding (BPE) able to use range of n-gram sizes, while SentencePiece is a re-implementation of sub-word units able to utilize both traditional BPE and unigram language modelling (Wu et al., 2016; Kudo and Richardson, 2018). Generally, WordPiece is seen to perform better in text classi-
4.2 Spanish Models
Again XLM-R models outperform mBERT, how- ever, differences in scores only range from 0.01-
0.02. Transfer-learnin XLM-R models also out-
perform the monolingual models, but only by 0.01;

---

Model [e2c]
tot avg
root
arg0
argl
arg2
gal mBERT[6]
0.68
0.39
0.32
0.32
0.30
gal_n_ en Ba_en_m_lll [
0.69
0.41
0.31
0.31
0.28
gal mal_t__nu po_mBERT[6]
0.68
0,41
0.35
0.30
0.28
gal_sp [___sp__ERTREE
0.72
0.53
0.40
0.36
0.33
val_enpt_mBERIIe
0.70
0.43
0.33
0.32
0.31
gal_ensp_mBERTTA)
0.70
0.45
0.38
0.34
0.30
gal_ptsp_mBBRRTV
0.69
0.46
0.38
0.34
0.30
gal_enttsp [impttp_mBERTTI4
0.70
0.52
0.36
0.37
0.31
gal_ XLM-R[6)
0.70
0.39
0.30
0.32
0.31
gal_en_ XLM-R(7)
0.70
0.45
0.30
0.33
0.31
gal pt_ XLM-R[6)
0.70
0.43
0.33
0.32
0.28
@al_sp_XMM__ 1PP_XLM-R[6)
0.73
0.52
0.47
0.40
0.34
gal_enpt____ XLM-R[6]
0.69
0.41
0.32
0.31
0.29
gal eal_ensp___ Bol_enss_XMMMae XLM-R[6]
0.74
0.53
0.42
0.41
0.40
gal_tsp__ (.P__XLM-R]6
0.73
0.56
0.41
0.42
0.38
gal enptsp XLM-R[6]
0.71
0,45
0.37
0.38
0.32
Table 3: Final fl Scores: Galician Models
e2c - number of epochs trained until convergence tot tot_avg - average fl score overall moo - average fl score identifying verbal roots arg0 average fl score identifying argument arg/ average score identifying argument arg2 - average fT score identifying argument 2
Model [e2c]
r0
r1
r2
r5
r6
r7
r8
r9
r10
gal mBERT [6]
0.78
0.69
0.62
0.54
0.50
0.29
0.11
0.00
0.00
0.00
0.00
gal_en mBERT [6]
0.78
0.69
0.61
0.52
0.52
0.27
0.08
0.00
0.00
0.00
0.00
gal pt .BERT [6]
0.77
0.69
0.60
0.53
0.50
0.29
0.16
0.00
0.00
0.00
0.00
gal gal_sp_____ sp_mBERY [4]
0.76
0.71
0.65
0.61
0.58
0.48
0.34
0.10
0.07
0.00
0.00
gal wept_nmppeee [6]
0.77
0.70
0.62
0.54
0.51
0.38
0.15
0.00
0.00
0.00
0.00
gal_ gal_ensp____ 1ess_ [4]
0.77
0.69
0.62
0.53
0.49
0.48
0.29
0.02
0.00
0.00
0.00
gal
[3]
0.75
0.68
0.63
0.57
0.49
0.44
0.26
0.00
0.07
0.00
0.00
gal _ enptsp mBERT [4]
0.78
0.69
0.62
0.57
0.52
0.46
0.27
0.19
0.04
0.00
0.00
gal XLM-R [6]
0.80
0.72
0.64
0.54
0.46
0.29
0.05
0.00
0.00
0.00
0.00
gal_en en XLM-R [7]
0.78
0.70
0.63
0.52
0.44
0.41
0.17
0.04
0.00
0.00
0.00
gal pt XLM-R [6]
0.79
0.70
0.63
0.53
0.51
0.35
0.09
0.00
0.00
0.00
0.00
gal sp XLM-R [6]
0.79
0.73
0.66
0.59
0.56
0.50
0.36
0.13
0.23
0.00
0.00
gal _al_enpt___ XLM-R [6]
0.77
0.71
0.62
0.52
0.48
0.33
0.06
0.00
0.00
0.00
0.00
gal gal_ensp_____
LIIIp_XXMM--M XLM-R [6]
0.79
0.73
0.68
0.61
0.62
0.59
0.37
0.25
0.08
0.07
0.00
gas_ XLM-R [6]
0.80
0.72
0.65
0.57
0.58
0.54
0.42
0.28
0.13
0.00
0.00
gal _al_enptsp___ XLM-R [4]
0.76
0.70
0.65
0.58
0.53
0.45
0.25
0.10
0.00
0.00
0.00
Table 4: Final f7 Scores by Verb Index: Galician Models
e2c the number of epochs trained until convergence r/idx) - average fl score identifying [idx]th root/arguments score of 0.00 signifies that no correct identifications of any kind were made by that model for the [idx]th root II

---

Model
fl
Despite Portuguese's similarities to Galician, models pre-trained in Spanish achieved the best results; scoring 0.04 over both the monolingual and Portuguese pre-trained models. As the Portuguese and Spanish training data are of comparable sizes and domains, the sole difference between the sets is the implementation of Verbal Indexing, suggesting this method has positive effect on final perfor- mance.
Bruton
0.83
Zhao
0.80
Meza-Ruiz
0.78
Nugues
0.77
Baoli Li
0.74
Moreau
0.64
Täckström
0.62
Lin
0.59
Only minor differences could be seen between the two architectures in the higher-resource Spanish scenario. with both monolingual models achieving score of 0.82. This indicates that the choice of base language model matters less the more target language training data exists. Pre-training on the SRL task also seemed to have little to no effect on the final fl score, further supporting that the most important variable toward accurate performance is the availability of data. As all scores were still able to outperform the baseline and maintained no differences in the training set other than our proposed method, this also suggests that Verbal Indexing is advantageous.
Table 5: 2009 CoNLL Shared Task Results for Spanish SRL-only Models (Hajič et al., 2009) and top results from this study (Bruton) f1 - average scores
fication models and those focusing on natural lan- guage understanding and is especially successful with highly morphological languages (Wu et al., 2016; Alimova al., 2020). SentencePiece is more suited to tasks such as sentiment analysis and named entity recognition (NER), and though is able to handle out-of-vocabulary words better than WordPiece, can struggle with morphology (Kudo and Richardson, 20. Though Galician is a highly inflectional language and the goal of SRL is to improve natural language understanding, in training process the labeling task was treated very much like NER, alongside indexing the verbal roots of sentence which could explain why the XLM-R model was able to perform so much better in this scenario (Galega, 2012).
5.1 Identification by Verbal Indexing
When measuring the score on the identification of Galician arguments associated with the Oth, 1st, and 2nd verbal root sets, the monolingual XLM-R model was able to outperform the mBER by 0.02, 0.03, and 0.02 respectively; root O tying for the highest score overall of 0.80. The mBERT model however was able to better identify arguments asso- ciated with the 4th and 6th verbal roots, by mark of 0.04 and 0.06 respectively. This could be due to the fact that mBERT is better able to general- ize, and as sentences including additional verbal roots occur far less frequently than those contain- ing 1-3, XLM-R simply was not able to adapt. The top performing model overall, gal_ensp_XL___ XLM-R, also achieved top fl scores for the 1st-5th root sets and was the only model able to identify any argu- ment of the 9th root set; receiving scores of 0.73, 0.68, 0.61, 0.62, 0.59, and 0.07 respectively. The Portuguese/Spanish pre-trained model ranks quite closely and outperforms the identification of the Oth, 6th, and 7th roots. Given the linguistic simi- larity between Galician and Portuguese, may be possible to achieve even better results if the Por- tuguese set is also pro-proceseed using the Verbal Indexing method in future work.
An example Galician sentence from the test dataset can be seen in Table 8 alongside the SRL results of both the gal_mBERT and gal XLM-R. mBERT's predisposition towards morphological processing can be seen in its partial error; while the correct verb index and argument were selected, it was inaccurately applied to only a portion of the correct token. Sexualidade split into two parts, sexual and idade, with only the first portion being identified as an argument. This can be seen through- out the dataset with the mBERT model; tokens are incorrectly parsed, with only portion of the to- ken being assigned role, rather than the entire token. The XLM-R model manages perfect pre- diction on this sentence, and generally throughout the dataset avoids mistakes of this type; its major struggles in being unable correctly identify roots and arguments more complex sentences; past the fourth root, its ability to identify roles drops sharply.
Looking at the Galician scoring overall, the ben-

---

Model [e2c]
tot jot_avg
root
arg0
argl
arg2
arg3
arg4
arg
spa mBERT [6]
0.82
0.51
0.39
0.38
0.33
0.13
0.26
0.27
spa_en mBERT [6]
0.81
0.49
0.37
0.37
0.32
0.09
0.22
0.26
spa sra_pt_miie Wot_BERTT [7]
0.82
0.44
0.37
0.38
0.32
0.20
0.27
0.29
spa enpt mBEETTEERTTT [6]
0.81
0.48
0.36
0.36
0.30
0.16
0.31
0.27
spa XLM-R [7]
0.82
0.44
0.36
0.37
0.34
0.18
0.28
0.24
spa_en XLM-R [7]
0.83
0.47
0.38
0.39
0.35
0.23
0.41
0.29
spa pt XLM-R [7]
0.83
0.49
0.39
0.39
0.36
0.19
0.34
0.30
spa_enpt XLMMR [6]
0.83
0.49
0.37
0.39
0.35
0.09
0.29
0.28
Table Final Scores: Spanish Models
ll e2c number of epochs trained until convergence; tot_avg avg -tt__aaggvvergee//eeee average score overall root average fl
identifying verbal roots arg0 average f/ score identifying argument arg/ average fl score identifying argument arg2 average fl score identifying argument arg3 average fl score identifying argument 3 arg4 average f7 score identifying argument 4 argM average fl score identifying argument M
Model [e2c]
r0
rl
r2
r3
r4
r5
r6
r7
r8
r10
spa WHL_MBRTT [6]
0.528
0.529
0.470
0.439
0,430
0.360
0.229
0.184
0.055
0.033
0.000
spa_en_mBBRl MBERT [6]
0.524
0.487
0.469
0.416
0.396
0.317
0.228
0.181
0.038
0.031
0.000
spa mBERT [7]
0.576
0.562
0.472
0.450
0.417
0.367
0.245
0.126
0.016
0.008
0.000
spa enpt mBERT [6]
0.543
0.540
0.459
0.417
0.423
0.376
0.194
0.131
0.033
0.000
0.000
spa XLM-R [7]
0.612
0.548
0.491
0.431
0.365
0.288
0.169
0.110
0.029
0.004
0.000
spa XLM-R [7]
0.606
0.573
0.499
0.469
0.447
0.324
0.245
0.306
0.070
0.010
0.000
spa_ pt_ 1LMMLM-R [7]
0.588
0.573
0.476
0.454
0.480
0,365
0.279
0.287
0.082
0.004
0.000
spa_enpt_XLLM-A XLM-R [6]
0.570
0.501
0.474
0.442
0.444
0.328
0.251
0.191
0.087
0.007
0.000
Table 7: Final fl Scores by Verb Index: Spanish Models
e2c number epochs trained until convergence rffid]] -fixx/aaverage fl score identifying [idx]th root/arguments rl r11-r14 scores not included always 0.00 score of 0.00 signifies that no correct identifications of any kind were made by that model for the [idx]tt Jth root II
rOzargO
O
r0:root
r1:root
rl:arg1
eng
Soedade
López
explains
how
educate on sexuality
glg
Soedade
López
explica
como
educar
para
a
sexualidade
#gall mBERT
rO:argO
O
r0:root
O
r1:root
O
O
rl:arg1 [sexual]
gal XLM-R
rO:arg0
O
r0:root
O
r1:rott
O
O
rl:arg1
Table 8: gal mBERT and gal XLM-R SRL Predictions: Galician Dataset Sentence #9
Il eng- English gloss of glg gig Galician text I gal mol_mBERTT predicted roles according gal mBERT gal RLM-R predicted roles according to gal_ XLM-R yellow target label green correct prediction I red- incorrect prediction orange partially correct prediction [error in brackets] r/idx):(label] - link [label] to [idx]th root root token assigned the verbal root arg0 token assigned argument 0 arg/ token assigned argument O - token assigned as not directly related to root worst fl scoring model(s)) best scoring model(s)

---

efits of Verbal Indexing are clear; models not pre- trained on a dataset using this method were unable to identify arguments of the 6th root or later ex- cept for a single model, val_en_XLMMRR achieving a score of 0.04 on the 7th root set. In compar- ison, the top performing model for the 7th root, gal_ptsp_XXLMa XLM-R which does pre-train on the Ver- bal Indexing method, achieves score of 0.28. This model was additionally able to identify ar- guments up through the 9th root set, full root sets past those not pre-trained on the Verbal Indexing method.
the dataset and performed by an individual with a non-native understanding of Galician.
Despite their linguistic similarities, the Por- ruguese dataset was also not pre-processed utilizing the Verbal Indexing method and it is recommended that this be explored in future work. Developing a reduced English dataset using this indexing method could also be quite helpful to other low-resource languages; as many pre-trained English models are available, it would be quite easy to fine-tune on a smaller English SRL dataset utilizing the indexing method, and then fine-tune the target low-resource language. The results reported here suggest this could have incredibly positive effects on the perfor- mance of SRL systems for low-resource languages, and potentially also higher-resouree languages, and future related work should focus on exploring this.
Looking at the Spanish models, the benefits are less clear but still visible. Despite the difficulties seen in argument identification by label, spa_XIM. XLM- R was still able to tie the monolingual mBERT model due to its increased performance in root set identification in less-complex sentences; scoring +0.084, +0.019, and +0.021 in the identification of the Oth-2nd root sets over mBERT. This could be due to the fact that no Spanish model was pre- trained on the Verbal Indexing method, or perhaps that the benefits of Verbal Indexing lessen as the availability of training data increases; it is recom- mended that this be explored in future research.
Ethics Statement
is important to specifyy that this dataset was de- veloped by an individual with a non-native under- standing of Galician and that native speakers were unable to be included in this initial production of the dataset.
Conclusion
References
In this work, we presented the first-ever SRL dataset for the low-resource Galician language, thus establishing a baseline for future SRL research in Galician. We also introduced a new prepro- cessing method for SRL systems, verb indexing, which shows positive effects on the final perfor- mance of systems when analyzing more complex sentences. The best performing SRL model for Galician scored 0.74 f7, increase of 0.06 over the Galician-only baseline model.
Rodrigo Agerri, Xavier Gomez Guinovart, German Rigau, and Miguel Anxo Solla Portela. 2018. De- veloping New Linguistic Resources and Tools for the Galician Language. In Proceedings of the 1 11th International Conference on Language Resources and Evaluation (LREC 2018), pages 28 2810-2817, Miyazaki, Japan. European Language Resources As- sociation (ELRA).
Alan Akbik, Laura Chiticariu, Marina Danilevsky, Yun- yao Li, Shivakumar Vaithyanathan, and Huaiyu Zhu. 2015. Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling. In Pro- ceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 397-407 Beijing, China. Association for Computational Lin- guistics (ACL).
Limitations
One limitation of this work is that the Galician dataset was created using a semi-automatic process, with only a small portion being manually validated. Ideally when developing corpora, manual annota- tions generated by native speakers are preferred as it ensures accuracy; a downside of this is the increased time, and likely financial, cost necessary recruit a team of annotators (King, 2015). Uti- lizing an automatic process which is then partially manually validated is one method reduce these costs. Due to external constraints, manual valida- tion in this study was limited to a small portion of
Isseyar Alimova, Elena Tutubalina, and Alexander Kir- illovich. 2020. Cross-lingual Transfer Learning for Semantic Role Labeling in Russian. In Proceed- ings of the 4th International Conference on Compu- tational Linguistics Bulgaria (CLIB 2020), pages 72-80, Sofia, Bulgaria. Department of Computational Linguistics, IBI.- BAS
Henrique Alonso, Carlos Rodríguez, and David Vilares. 2016. Galician UD Treehank.

---

Steven Bird, Edward Loper, and Ewan Klein. 2009, Nat- ural Language Processing with Python. "'iRelly Media, Inc.".
Nianwen Xue, and Yi Zhang. 2009. The CoNLL- 2009 Shared Task: Syntactic and Semantic Depen- dencies in Multiple Languages. In Proceedings of the 13th Conference on Computational Natural Lan- guage Learning (CoNLL 2009): Shared Task, pages 1-18, Boulder, Colorado. Association for Computa- tional Linguistics (ACL)..
Encyclopaedia Britannica. 2023. Galician Language.
Simone Conia, Edoardo Barba, Alessandro Scirè, and Roberto Navigli. 2022. Semantic role labeling meets definition modeling: Using natural language to de- scribe predicate-argument structures. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4253-4270, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Pupers), pages 2061-2071, Melbourne, Australia. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451. Association for Computational Linguistics (ACL).
Ishan Jindal, Alexandre Rademaker, Michaf Ulewicz, Ha Linh, Huyen Nguyen. Khoi-Nguyen Tran, Huaiyu Zhu, and Yunyao Li. 2022. Universal Proposition Bank 2.0. In Proceedings of the Language Resources and Evaluation Conference, pages 1700-1711, 1, Mar- seille, France. European Language Resources Asso- ciation (ELRA).
Dan Jurafsky and James H. Martin. 2023. Speech and Language Processing.
José Angel Daza Arévalo. 2022. Cross-lingudd Seman- tie Role Labeling through Translation and Multilin- gual Learning. Ph.D. thesis, Heidelberg University.
Benjamin Philip King. 2015. Pructical Natural Lan- guage Processing for Low-Resource Languages. Ph.D. thesis, University of Michigan.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and
Kristina Toutanova. 2019. BERT: Pre-training of
Taku Kudo and John Richardson. 2018. SentencePiece: A Simple and Language Independent Subword Tok- enizer and Detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguisties (ACL).
Deep Bidirectional Transformers for Language Un- derstanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), J, pages Minneapolis, Minnesota. Association for Computational Linguistics (ACL).
Language Archives Services, University of British Columbia. 2023. Olac Resources in and about the Galician Language.
Magali Sanches Duran and Sandra Maria Aluísio. 2011. Propbank-Br: Brazilian Portuguese Corpus Anno- tated with Semantic Role Labels. In Proceedings of the 8th Brazilian Symposium in Information and Human Language Technology.
Lluís Marquez, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. 2008. Special issue introduc- tion: Semantic role labeling: An introduction to the special issue. Computational Linguistics. 34(2):45- 159.
David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2023. Ethnologue: Languages of the World, 26th edition. SII International, Dallas, Texas.
Hiroki Nakayama. 2018. seqeval: A Python Framework for Sequence Labeling Evaluation.
Real Academia Galega. 2012. Normas ortográficas e morfolóxicas do idioma galego. Real Academia Galega, A Coruña.
Takuya Okamura, Koichi Takeuchi, Yasuhiro Ishihara, Masahiro Taguchi, Yoshihiko Inada, Masaya lizuka, Tatsuhiko Abo, and Hitoshi Ueda. 2018 Improving Japanese semantie-roee-lbbling performance with transfer learning case for limited resources of tagged corpora on aggregated language. Proceed- ings of the 32nd Pacific Asia Conference on Lan- guage, Information and Computation, Hong Kong. Association for Computational Linguisties (ACL).
Xavier Gómez Guinovart and Manuel Anxo Solla Portela. 2018. Building the Galician WordNet: Meth- ods and Applications. Language Resources and Eval- uation, 52(3):11-339.
Carlos Godrzzzod David Vilares, and Raquel Gómez-Díaz. 2017. Galician-CTG Treebank.
Jan Hajič, Massimiliano Ciaramita, Richard Johans- son, Daisuke Kawahara, Maria Antónia Martí, Lluis Márquez, Adam Meyers, Joakim Nivre, Sebastian Pad6, Jan Stěpánek, Pavel Strañák, Mihai Surdeanu,
Sofia Oliveira, Daniel Loureiro, and Alípio Jorge. 2021. Transformers and Transfer Learning for Improv- ing Portuguese Semantic Role Labeling. CoRR. abs/2101.01213.

---

Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An Annotated Cor- of Semantic Roles. Computational Linguistics, 31(1):71-106.
Héctor Rodríguez, Ana Villena, Mariona Martí, Xavier Carreras, Irene Castelló, Patricio Bartinez-BBrco, and Mariona Taulé. 2004. ANCORA A-ES: Multilevel annotated corpora for catalan and spanish. In Pro- ceedings of the Fourth International Conference on Language Resources and Evaluation (LREC-2004), pages 1575-1578. European Language Resources Association (ELRA).
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release 5.0. Philadelphia: Linguistic Data Consortium.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurya Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang. Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's Neural Machine Translation Sys- tem: Bridging the Gap between Human and Machine Translation.


Reviews:
[{"review_id": "4pb8cKKQCA", "reviewer": "Reviewer_x4WB", "summary": "", "questions": "Question A: Can you provide more details on the diversity of the Galician SRL dataset in terms of genres, styles, and sources? How do you ensure that it captures a broad spectrum of the Galician language?\n\nQuestion B: How does the \"verbal indexing\" method perform when applied to languages with significantly different linguistic structures than Galician or Spanish? Have you considered testing it on non-Romance languages?\n\nQuestion C: In your experiments, where did the models particularly struggle or fail? A deeper analysis of these challenges could provide insights for future improvements.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for providing feedback, and insightful suggestions. Below, we respond to the questions asked.\n\n A. The Galician dataset includes technical sentences of the medical, sociological, ecological, economic, and legal domains that have been pulled from the both Galician Technical and XIADA Corpus by the Galician-CTG and Galician-TreeGal treebanks. We will add that to the paper. \n\nB.  It would be interesting to branch out of romance languages to see if verbal indexing has the same effect on complex sentences in other language families and we do hope to do so in the future, however, this study was meant as a beginning and its main focus was increasing available resources for low-resource Galician. The inclusion of Spanish was a lesser focus and was mainly used as a method of comparing the results against an established baseline.\n\nC. mBERT models (using WordPiece tokenization) mainly struggled throughout with correctly tokenizing individual words, often breaking tokens down too far and assigning roles to only portions of a single token. \n\nXLM-R models (using SentencePiece tokenization) did not have this issue and instead mainly struggled with correctly identifying verbs/arguments in more complex sentences; for example in a sentence containing 6 verbs, a model may have only correctly identified the first four verbs and their arguments. The final performance was increased when pre-trained using the verbal indexing method, but the XLM-R models still struggled more with this than the mBERT models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 Addressing a Low-Resource Language: One of the paper's primary strengths is its focus on the Galician language, a low-resource language in the NLP domain. By introducing resources and methodologies for such a language, the paper contributes to the diversification and inclusivity of linguistic research in NLP.\n2 Creation of a Novel Dataset: The authors have successfully developed the first-ever dataset for training SRL systems in Galician. This dataset fills a significant gap in the NLP resources available for the language and provides a foundation for future research in this area.\n3 Introduction of \"Verbal Indexing\": The novel pre-processing method, \"verbal indexing,\" introduced in the paper, offers a fresh perspective on semantically parsing complex sentences. This technique can potentially be applied to other languages and tasks, making it a valuable contribution to the field.\n4 Use of Transfer Learning: The application of transfer learning to test the new resource and the verbal indexing method showcases a modern approach to NLP tasks, emphasizing the paper's relevance to current research trends.", "reasons_to_reject": "1 Dataset Size and Diversity: While the creation of a Galician SRL dataset is commendable, its size might be considered relatively small for deep learning models. The diversity of the dataset in terms of genres, styles, and sources could also be a concern, as it might not capture the full linguistic richness of the Galician language.\n2 Generalizability of \"Verbal Indexing\": The novel \"verbal indexing\" method is introduced and tested primarily on Galician and Spanish. Its effectiveness and applicability to other languages, especially those with different linguistic structures, remain unexplored.\n3 Detailed Analysis of Failures: While the paper presents the successes of its methods, a deeper dive into where the models failed or struggled would provide a more balanced view and insights for future improvements.", "questions_to_authors": "Question A: Can you provide more details on the diversity of the Galician SRL dataset in terms of genres, styles, and sources? How do you ensure that it captures a broad spectrum of the Galician language?\n\nQuestion B: How does the \"verbal indexing\" method perform when applied to languages with significantly different linguistic structures than Galician or Spanish? Have you considered testing it on non-Romance languages?\n\nQuestion C: In your experiments, where did the models particularly struggle or fail? A deeper analysis of these challenges could provide insights for future improvements.", "paper_topic_and_main_contributions": "The paper, \"BERTie Bott’s Every Flavor Labels: A Tasty Introduction to Semantic Role Labeling for Galician,\" delves into the domain of Semantic Role Labeling (SRL) with a specific focus on the Galician language. SRL is a computational linguistic task that identifies the semantic roles of words in a sentence concerning a particular noun or verb. The study emphasizes the importance of SRL in relation to verbal predicates.\n\nMain Contributions:\n1 Creation of a Galician SRL Dataset: The authors successfully leveraged existing resources, such as the UD Galician-CTG and UD Galician-TreeGal treebanks, to produce the first-ever dataset for training SRL systems in Galician. This dataset consists of 3,987 training sentences and 998 test sentences.\n2 Introduction of \"Verbal Indexing\": A novel pre-processing method named \"verbal indexing\" was introduced. This technique aids in semantically parsing complex sentences, enhancing the accuracy and efficiency of the SRL process.\n3 Transfer Learning Experiments: The paper employed transfer learning to test the newly created resource and the verbal indexing method. A total of 24 models were developed and tested, with 16 tailored for Galician and 8 for Spanish.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RyNUhzLMo9", "reviewer": "Reviewer_9UHD", "summary": "", "questions": "The verbal indexing looks very similar to to direct mapping in Lonneke, 2011. That approach was criticized in Akbik, 2015 (which you do cite) which showed filtered projection to work better. Can you explain the distinction better between Lonneke and you, and more clearly defend the novelty?\n\nI don't see a clear description how you matched the verbal lemma in Galician to its English PropBank roleset, which is an important part of the methodology. In any language there will be cases where you can not easily match the target language verb to its English roleset.  For example, phrasal verbs in English can map to a single verb in the target language and so on.  Please augment the paper by describing your matching algorithm.\n\nThere are other SoTA results on multilingual SRL that ought to be added to the experiments. For example Conia, 2021 (Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources) shows Spanish SoTA at 86.3, higher than this paper's 83 (interestingly, also using XLM-R). Please do additional lit review (or else explain why this work does not apply, as I believe it does) and expand your experimental section accordingly\n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the feedback from the reviewer and aim to address the concern raised by R2 regarding the lack of innovation in our model's verb indexing approach. \n\nOur approach to verb indexing differs from a direct mapping technique. Essentially, verb indexing involves tallying the number of verbs present in a given sentence. This is done to enhance the model's understanding of sentence complexity and aid in the initial identification of verbs within the sentence. It's important to note that this method does not play a role in assigning semantic roles or rolesets, unlike the direct projection method. The purpose of verbal indexing is to count all verbs within a sentence, arranging them based on their position in the sentence. Importantly, this indexing process does not impact the PropBank label associated with each verb. For instance, even if a verb with the PropBank label \"run.01\" appears twice in a single sentence, each instance would be assigned a distinct index through the verbal indexing approach, as the index is directly linked to the verb's position in the sentence. We will add all missing references and add a paragraph about the differences between verb indexing and direct mapping.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "More work in low resource languages is always good, we need Galician resources.", "reasons_to_reject": "-The contribution of the verbal indexing method does not seem to me to be new\n-There are some clarifications needed in the methodology section, see below in Questions\n-The experiments section does not appear to compare to SoTA, see below in Questions ", "questions_to_authors": "The verbal indexing looks very similar to to direct mapping in Lonneke, 2011. That approach was criticized in Akbik, 2015 (which you do cite) which showed filtered projection to work better. Can you explain the distinction better between Lonneke and you, and more clearly defend the novelty?\n\nI don't see a clear description how you matched the verbal lemma in Galician to its English PropBank roleset, which is an important part of the methodology. In any language there will be cases where you can not easily match the target language verb to its English roleset.  For example, phrasal verbs in English can map to a single verb in the target language and so on.  Please augment the paper by describing your matching algorithm.\n\nThere are other SoTA results on multilingual SRL that ought to be added to the experiments. For example Conia, 2021 (Unifying Cross-Lingual Semantic Role Labeling with Heterogeneous Linguistic Resources) shows Spanish SoTA at 86.3, higher than this paper's 83 (interestingly, also using XLM-R). Please do additional lit review (or else explain why this work does not apply, as I believe it does) and expand your experimental section accordingly\n\n\n", "paper_topic_and_main_contributions": "This paper claims two contributions: a new Galician SRL dataset and a new preprocessing method, verbal indexing. There are indeed not many Galician resources so this would benefit the community. On the other hand the verbal indexing method does not seem to me to be new. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "(see above Questions)\n\nLonneke 2011 \nConia 2021"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Overall the presentation is fairly clear.  I did find the captions extremely challenging to read in the way that they were formatted - I suggest fewer details and more description of what the takeaway point of the figure / table is."}}, {"review_id": "Wlc52Y77Gm", "reviewer": "Reviewer_Swd1", "summary": "", "questions": "- Question A: What are the grounds for choosing Spanish as a source for Galician? It is also a Romance language and their localisations are close, but having a few pieces of information about vocabulary and grammar similarities would help understand the method. A majority of speakers are probably bilingual, which may have an impact on the annotation.\n- Question B: Just one test or cross-validation? It could change the differences between the setups.\n- Question C: What quantity of data do you have? How did you divide your data into train, dev and test?\n- Question D: Are there more examples of different treatments by WordPiece and SentencePiece?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback and suggestions. Below, we offer responses to the posed inquiries:\nA. Few languages are categorized as high-resource for this task. Spanish stands out not only due to its abundant SRL data and published results that serve as a baseline but also because we had annotators who could speak both Spanish and Galician. \n\nB. Although this experiment did not involve cross-validation through alterations in train/test/dev groups, the inclusion of numerous models was deliberate. This approach provides a broader perspective on the results and accommodates potential variations.\n\nC. The Galician dataset comprises 3,987 training sentences and 998 test sentences, encompassing sentences with up to 13 verbal roots. In contrast, the Spanish dataset includes 14,329 training sentences, 1,655 development sentences, and 1,725 test sentences. Each of these sentences hosts up to 16 verbs. The train/test/dev split for Spanish adheres to the CoNLL release. For Galician, an 80/20 split was employed, with sentence allocation to sets being random yet with an endeavour to ensure a balanced distribution of simple (0-4 verbal roots) and more intricate (5+ verbal roots) sentences across sets, guided by insights gained from verbal indexing.\n\nD. A customary approach involves tracking word positions within a sentence. After reviewing the provided references, it seems this might align with your inquiry. Verbal indexing is specific to counting verbs within a sentence and assigning them indices. It does not pertain to sentence structure established by predicates and their arguments.\n\nWe appreciate the feedback and will add more statistics about our dataset to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- There was no such dataset for Galician. This work provides a new resource for a low-resource language and tries methods for this kind of language.\n- The authors have a sound approach: they evaluated their pre-processing method on a second language with more resources and speakers.\n- Comparing the the models' ability to find up to 10 semantic label in a sentence is an interesting and valuable evaluation.\n- Comparing WordPiece and SentencePiece for this task is valuable feedback for the community.", "reasons_to_reject": "- Not much has been explained about disambiguation of the annotation, although it is an important topic.\n- As far as I understand, the authors did not look into cross-validating their evaluations. A difference in score can be nuanced by changing train, dev and test distributions.\n- Verbal indexing seems to be a new name for what is common practice in the literature. The structure given by a predicate and its arguments is already the foundation of semantic role labeling (see missing references for examples).", "questions_to_authors": "- Question A: What are the grounds for choosing Spanish as a source for Galician? It is also a Romance language and their localisations are close, but having a few pieces of information about vocabulary and grammar similarities would help understand the method. A majority of speakers are probably bilingual, which may have an impact on the annotation.\n- Question B: Just one test or cross-validation? It could change the differences between the setups.\n- Question C: What quantity of data do you have? How did you divide your data into train, dev and test?\n- Question D: Are there more examples of different treatments by WordPiece and SentencePiece?", "paper_topic_and_main_contributions": "The authors created a dataset of Semantic Role Labeling for Galician, a low-resource language. They chose to focus on verbal roots and their arguments in order to identify semantic roles (2.0.1). Then they procede to prune the arguments and to attribute a specific label to specific kinds of arguments. For Spanish, they prepare the existing data for the verbal indexing step (2.0.2). They also evaluated the impact of verbal indexing as pre-processing for this task on Galician and Spanish.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Conia, Simone, Edoardo Barba, Alessandro Scirè, and Roberto Navigli. ‘Semantic Role Labeling Meets Definition Modeling: Using Natural Language to Describe Predicate-Argument Structures’, 2022. https://doi.org/10.48550/ARXIV.2212.01094.\n- He, Shexia, Zuchao Li, Hai Zhao, and Hongxiao Bai. ‘Syntax for Semantic Role Labeling, To Be, Or Not To Be’, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), pages 2061–2071. Melbourne, Australia, July 15 - 20, 2018.\n- Màrquez, Lluís, Xavier Carreras, Kenneth C. Litkowski, and Suzanne Stevenson. ‘Semantic Role Labeling: An Introduction to the Special Issue’. Computational Linguistics 34, no. 2 (June 2008): 145–59. https://doi.org/10.1162/coli.2008.34.2.145."}}, {"review_id": "Owj17fms0E", "reviewer": "Reviewer_Dkyz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your thoughtful feedback, and we are committed to addressing these points to improve the overall quality and relevance of our paper.\n\n´´It looks to me that the results here are suitable for a Findings paper, but there is not much to excite a general ACL audience. People with strong interests in less-resourced languages will be interested.''\nWe value your perspective on the potential interest of the ACL audience. While our primary focus is on providing insights for researchers working with less-resourced languages, we are also proposing a broader methodology as you mentioned. Your insight into the broader applicability of our methodology aligns with our intention. We aim to highlight not just the creation of a Galician corpus but also the potential relevance of our approach to languages facing similar resource challenges. In response to your suggestion, we will provide more explicit guidance on adapting our methodology to address the general language situation and challenges, offering a framework for researchers working with languages in comparable circumstances. We will take this feedback into account and work to enhance the discussion of general principles and methodology that could resonate with a wider audience within the ACL community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Nice work. All choices for methodology are reasonable. This is a good findings paper and will help researchers interested in building similar corpora for other less-resourced languages.\n\nWhat is missing is more discussion of why people from the ACL community should care, and what generalizable ideas have been uncovered as a result of the author's careful work.", "reasons_to_reject": "Since subjective judgments are involved in the manual verification of some of the data, it would be a good idea to have inter-rater reliability numbers. Their absence is a weakness.\n\nThe limitations section mentions that the author (described as \"an individual\" is not a native speaker of Galician). This is not necessarily a problem, but it needs to be discussed in the body of the paper, not just mentioned in passing. The rater's actual language background should be revealed. If necessary, native speaker help should be added. \n\nThe most interesting lessons of the paper are not about the availability of a corpus for Galician, but the availability of a methodology that might be used for similar language situations. It is of interest if a small-scale effort by a non-native speaker can produce something useful. in order to make this lesson available, the author needs to indicate which aspects of the general language situation are important for the design of the corpus. This requires some material explaining the language situation. Despite having worked on relationships between Romance languages, this reviewer was not aware of the existence of Galician or its relationships with Spanish and Portuguese. If I am typical, ACL readers need some introductory material. Space could be found for this by reducing the amount of material about the various models used. It is not clear why the paper needs to report on so many models. If the author has hypotheses or opinions about what SHOULD work, and why, and results on what DID work, and why, they do not come through clearly in the paper. More selectivity is needed, and more focus on interesting general principles.\n\nIf I understand correctly, Galician has official status in a region of north western Spain, and is widely spoken, with a literary tradition, but also a period in which the language was suppressed by Franco's regime. The revival of the language is therefore new, possibly paralleling the situation of other languages that used to be banned.\nThese facts are potentially relevant to the wider applicability of the methods used for the study. For example, potential re-users of the methods need to know that there is a Wikipedia and that there is just one active newspaper, published in Galician and Spanish.\n\nIt looks to me that the results here are suitable for a Findings paper, but there is not much to excite a general ACL audience. People with strong interests in less resourced languages will be interested.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This model describes a  first SRL  dataset of Galician.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There is no good reason for the Harry Potter reference in the title. An informative title is hugely preferable."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission introduces a training dataset for Galician Semantic Role Labeling, and a pre-processing method called verb indexing, to improve the accuracy of parsing complex sentences. The proposed method has been also evaluated on Spanish to measure its impact. 
After author rebuttal, the reviewers discussed it, and they agreed that a new resource for Galician is valid and worthy for the community. However, they also point out the need to improve the evaluation comparison with recent multilingual SoTA SRL works that provide better results (not included in the submission). Moreover, the paper would also benefit from a better explanation of the novelty of their proposed verbal indexing method.
Decision: Accept-Main