Learning Semantic Role Labeling from Compatible Label Sequences
Authors: Tao Li
Ghazaleh Kazeminejad
Susan Windisch Brown
Vivek Srikumar
Martha Palmer
Venue: EMNLP 2023 Findings (2023)
Keywords: SRL
TL;DR: We propose a joint model design to efficiently learn from disjoint but compatible label sequences for semantic role labeling.

Abstract: Semantic role labeling (SRL) has multiple disjoint label sets, e.g., VerbNet and PropBank. Creating these datasets is challenging, therefore a natural question is how to use each one to help the other. Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. A common issue with multi-task setup is that argument sequences are still separately decoded, running the risk of generating structurally inconsistent label sequences (as per lexicons like Semlink). In this paper, we eliminate such issue with a framework that jointly models VerbNet and PropBank labels as one sequence. In this setup, we show that enforcing Semlink constraints during decoding constantly improves the overall F1. With special input constructions, our joint model infers VerbNet arguments from given PropBank arguments with over 99 F1. For learning, we propose a constrained marginal model that learns with knowledge defined in Semlink to further benefit from the large amounts of PropBank-only data. On the joint benchmark based on CoNLL05, our models achieve state-of-the-art F1's, outperforming the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank). For out-of-domain generalization, our models surpass the prior best by 3.4 (VerbNet) and 0.2 (PropBank).

Learning Semantic Role Labeling from Compatible Label Sequences
Tao Li
Ghazaleh Kazeminejad
Susan W. Brown
Google Research tlinlp@google. c.meoooooocccommmmm.oomooooom com
University of Colorado Boulder
University of Colorado Boulder
ghia94468colorado. edu
susan. wrwaa.bron@cooorraae edu
Martha Palmer
Vivek Srikumar
University of Colorado Boulder
University of Utah
martha, pataaappameeeeer @wl.mee@@ooooddoo.oooooooooooooe edu
swivekecs.utal wtw...daa..uu.........
Abstract
can be coarsely divided into PropBank (Palmer et al., 2005) senses, each with core set of num- bered semantic arguments (c.g., ARGO-ARG5). There are also modifier arguments (e.g., ARGM- LOC) typically representing information such as the location, purpose, manner or time of an event. Alternatively, predicates can also be hierarchically clustered into VerbNet (Schuler, 2005) classes cording to similarities in their syntactic behav- ior. Each class admits a set of thematic roles (e.g. AGENT, THEME) whose interpretations interpretations are consis- tent with all predicates within the class.
Semantic role labeling (SRL) has multiple dis- joint label sets, e-g., VerbNet and PropBank. Creating these datasets is challenging, therefore a natural question is how to use each one to help the other. Prior work has shown that cross-task interaction helps, but only explored multitask learning so far. A common issue with multi- task setup is that argument sequences are still separately decoded, running the risk of gener- ating structurally inconsistent label sequences (as per lexicons like SEMLINK). In this paper, we eliminate such issue with a framework that jointly models VerbNet and PropBank labels as one sequence. In this setup, we show that en- forcing SEMLINK constraints during decoding constantly improves the overall F1. With spe- cial input constructions, our joint model infers VerbNet arguments from given PropBank argu- ments with over 99 F1. For learning, we pro- pose a constrained marginal model that learns with knowledge defined SEMLINK to further benefit from the large amounts of PropBank- only data. On the joint benchmark based on CONLLOS, our models achieve state-of-the-an Fl's, outperforming the prior best in-domain model by 3.5 (VerbNet) and 0.8 (PropBank), For out-of-domain generalization, surpass the prior best by 3.4 (VerbNet) and 0.2 (anopBank).
As a modeling problem, SRL requires associat- ing argument types and phrases with respect to an identified predicate. The two labeling tasks (i.e.. VerbNet SRL. and PropBank SRL) are closely re- lated; but they differ in their treatment of predicates and have disjoint label sets. Learning jointly improve data efficiency across the different label- ing SRL tasks.
A common formulation used to instantiate this idea in prior work is multitask learning (e-g. Strzyz et al., 2019; Gung and Palmer, 2021): each label set is treated as a separate labeling task, and sometimes also modeled with inter-task feature interaction or consistency losses. While multitask learning often works well in such cases, the loss formulation rep- resents conservative view over label compatibili- ties of different tasks. At prediction time, subtask modules still run independently and are not con- strained by each other. Consequently, decoded la- bels may violate structural constraints with respect to each other. In such settings, constrained infer- ence (e.g., Fürstenau and Lapata, 2012; Greenberg et al., 2018) 8) has been found helpful. However, this raises the question of how to involve such inference during learning for better data efficiency. Further- more, given the wider availability of PropBank- only data (e.g., Pradhan al., 2013, 2022), how to efficiently benefit from such data also remains a question.
models
1 Introduction
Semantic Role Labeling (SRL, Palmer et al., 2010) aims to understand the role of words or phrases in sentence. It has facilitated other natural language processing tasks including question an- swering (FitzGerald et al., 2018), sentiment analy- sis (Marasović and Frank, 2018), information ex- traction (Solawetz. and Larson, 2021), I), and machine translation (Rapp. 2022).
Semantic role labeling can take various forms, each associated with different datasets. Predicates
"Work done at the University of Utah.

---

In this paper, we argue that the two disjoint but compatible labeling tasks can be more effectively modeled as one task using their compatibility struc- tures that are already explicitly defined in the form of SEMLINK (Stowe et al., 2021). SEMLINK of- fers mappings between various semantic ontolo- gies including PropBank and VerbNet. Gung and Palmer (2021) devised a deterministic conversion from PropBank label sequences to VerbNet ones us- ing only the unambiguous mappings SEMLINK. This conversion gives a test bed that has half of the predicates CoNLLO5 SRL dataset (Carreras and Marquez, 2005) with both VerbNet and PropBank jointly labeled.
Admissible alignments (y"-y()
"
price-54.4
Agent-ArgO; Theme-Argl; Value-Arg2
price-54.4 02
Agent-ArgO; Theme-Argl; Value-Arg2
admire-31.2 02
Experiencer-ArgO: Stimulus-Argl
Table I: VN-PB alignments defined in SEMLINK for predi- cate value, by VN class, a: PB sense.
labeling as separate tasks and focus on the latter. Assuming predicates u and their associated VN classes 1) and PB senses are given along with æ, write the prediction problem as:
(a, u. 17,
(1)
Given this setting, we propose a simple and effective joint CRF model for the VerbNet SRL and PropBank SRL tasks. addition to the joint CRF, we propose an inference constraint that uses compatible label structures defined in SEMLINK, and show that our constrained inference achieves higher overall SRL Fh-the the average of VerbNet and ProbBank Fl scores-than the current state- of-the-art. Indeed, when PropBank labels ob- served, it achieves over 99 Fl on the VerbNet SRI,, suggesting the possibility of an automated annota- tion helper. We show that our formulation naturally extends to a constrained marginal model that learns from the more abundant PropBank semi-supervised setting. When learning and pre- dicting with constraints, it achieves even better SRL F1 in out-of-domain generalization.'
2.1 VerbNet Completion
There is a much larger amount of PropBank-only data (e.g.. Pradhan et al., 2013, 2022) than jointly labeled data. Inferring VerbNet labels from ob- served PropBank labels, therefore, is a realistic use case. This corresponds to the modeling problem:
(2, u. n, vy
(2)
We refer this scenario as completion mode. In this paper, we will focus on the joint task defined in Eq. while also generalizing our approach to address the completion task in Eq. 2.
data in a
2.2 Multitask learning and Its Limitations
When predicting multiple label sequences for SRL, a common approach is multitask learning using dedicated classifiers for each task that operate on a shared representation. The current state-of-the-art model (Gung and Palmer, 2021) used LSTM stacked on top of BERT (Devlin et al., 2019) model both PropBank and VerbNet. While cach set of the semantic roles is modeled jointly with VerbNet predicates, the argument labeling of the two subtasks still kept separate.
2 Joint Task of Semantic Role Labeling
We consider modeling VerbNet (VN) and Prop- Bank (PB) SRL as joint labeling task. Given a sentence we want to identify set of predicates (c.g., verbs), and for each predicate, generate two sequences of labels, one for VerbNet arguments v². the other for PropBank arguments y.. With respect to VN parsing, predicate is associated with a VerbNet class that represents a group of verbs with shared semantic and syntactic behav- ior, thereby scoping a set of thematic roles for the class. Similarly, the predicate is associated with a PropBank sense tag that defines set of PB core arguments along with their modifiers. A example is shown in Tab. 1.
Separate modeling of VerbNet SRL and Prop- Bank SRL has clear disadvantage: subtask ar- gument labels might disagree in three ways: 1) in terms of the BIO tagging scheme-e.g.. a word having a B_* VN label and 1-* PropBank label. or 2) assigning semantically invalid label e.g. an ARGM-LoC being called a THEME, or 3) vio- lating SEMLINK constraints. In Sec. 6, we show that model with separate task classifiers, while having a close to state-of-the-an F1, can have a fair
We treat predicate classification and argument
Prior work (c.g.. Täckström et al., 2015) has shown that the predicate disambiguation can be modeled with high accu- racy using standalone classifiers.
'The code for reproducing our experiments: https:// gi thub. hwmm/utainnll/mrrinnnsssss......

---

amount of argument assignment errors with respect SEMLINK, especially for out-of-domain inputs.
partition that sums exponentiated scores over all label sequences. The term y y denotes the label sequence that has VN and PB jointly labeled. We will refer to this model as the joint CRF, and the label sequence as the joint labels.
3 A Joint CRF Model
To eliminate the enrors discussed in Sec. 2.2, we propose to model the disjoint SRL tasks using joint set of labels. This involves converting multi- task modeling into single sequence labeling task whose labels are pairs of PB and VN labels. Doing so not only eliminates the BIO inconsistency, but also exposes an interface for injecting SEMLINK constraints.
Reduced Joint Label Space. We use the cross- product the two label sets, prefixed with a BIO prefix?. A brute-force cross product leads to a |YV] |Y label space. In practice, it is impor- tant to keep the joint label space at a small scale for efficient computation, especially for the CRF module. Therefore, we condense it by first disal- lowing pairs of the form (B-*, I-*) and predicate- to-argument pairs. The former enforces that the VerbNet arguments do not start within ProbBank ar- guments, while the latter ensures that the predicate is not part of any argument. Next, we observe the co-occurence pattern of VN and PB arguments, dis- abling semantically invalid pairs such as (THEME, ARGM-Locy+ This reduces the label space by order of magnitude (from 144 105 15, 120 685).
Our model uses ROBERTA (Liu et al., 2019) as the backbone to handle textual encoding, similarly to the SRL model of Li et al. (2020). At high level,
stack of linear layers with GELU activations (Hendryeks and Gimpel, 2016) en- code tokens be classified for a predicate. For the problem of predicting arguments of predicate u, we have an encoding vector Uu,i for the i-th word in the input text x.
E = map(RoBERTA(())
(3)
6u = (fua
(4)
Input Construction using Predicates. We take inspiration from prior work (e.g. Zhou and Xu, 2015; He al., 2017: Zhang et al., 2022) to ex- plicitly put predicate features as part of the input to augment textual information. At the same time, we also seek to maintain a simple construction that can be easily adapted to a semi-supervised setting (i.e. compatible with PropBank end, we propose simple solution that appends the PropBank senses of potential predicates to the original sentence x::
Here, map sums up word-piece embeddings to form a sequence of word-level embeddings, the func- tions fu and Sa are both linear layers, and fua de- notes a two-layer network with GELU activations in the hidden layer. We use dedicated module of the form in Eq. 4 for the VN and PB subtasks. This gives us a sequence of vectors dy for VN and a sequence of vectors dh for PB.
data). To this
Next, we project the VN and PB feature se- quences into |Y PY label space:
ZWP |CLS WIT SEP JNN SEP]
Z Lo (..)). V)E
(5)
where W1.7 denotes the input words, and T de- notes the senses of the N predicates. In practice, use the PropBank roleset IDs which consist of pair of (lemma, sense) rln.01. Our models only take the encodings for wI.T after the ROBERTA encoder and ignore the rest. We con- sider this design to be more efficient than prior work (e.g. Gung and Palmer, 2021: Zhang et al., 2022) that dedicated text feature for each predi- cate. In our setup, the argument labeling for dif- ferent predicates shares the same input, thus no
Here, g is another two-layer GELU network fol- lowed by linear projection that outputs [YV] x |YP| scores, corresponding to VN-PB label pairs. The final result 2a denotes a sequence of VN-PB label scores for a specific predicate u. In addition, we use CRF as a standard first-order sequence model over zu (treating it as emission scores), and use Viterbi decoding for inference. The training objective is to maximize:
log P(y s(y" S x) - Z(x)
(6)
where s(.) denotes the scoring function for la- bel sequence that adds up the emission and the transition scores, and the term Z(x) denotes the
'For example, the pair (B-THEME, B-ARGI). *VerhNet arguments typically align to PropBank core argu- ments, not modifiers; thus pairs like (ACTOR, ARGM-CAU) can he filtered out.

---

need run encoding multiple times for multiple predicates.
where y yh denotes potential joint label quence with only PropBank arguments observed for a predicate u. Scores of such label sequences are aggregated by the LSE operator. Note that the marginal CRF and the joint CRF (Eq. 6) use the same model architecture, just with different loss.
4 Semi-supervised Learning with PropBank-only Data
Compared to data with both VerbNet and Prop- Bank fully annotated, there is more data with only PropBank labeled. The SEMLINK corpus helps in unambiguously mapping ~ 56% of the CoNLLOS data (Gung and Palmer, 2021). Therefore. natu- ral question is: can use PropBank-only data to improve the joint task?
4.4 Marginal Model with SEMLINK (MarginalsEML)
The log marginal probability in Eq. 7 assumes uni- form distribution over large label space. It in- cluded any arbitrary VerbNet arguments paired the observed PropBank labels. In practice, we can narrow down to only legitimate VN-PB argument pairs defined in SEMLINK. Such legitimate space is uniquely determined by a VerbNet class 1j and PropBank sense o. We will refer to label sequences that comply with this space as y and apply it on Eq. 7:
Here, we explore model variants based on the joint CRF architecture described Sec. 3. We will focus on modeling for PropBank-only sentences.
4.1 Separate Classifiers for VN and PB
As first baseline, treat VN and PB as two separate label sequences during training. This is essentially multitask setup where VN and PB targets use separate classifiers. We let these two classifiers share the same RoBERTA encoder, and have their own learnable weights for Eq. 4-6.
LSE yevl ougee s(y) Lyyeengaa)s()))
(8)
4.2 Dedicated PropBank Classifier
Note that this formulation essentially changes the global optimization into a local version which im- plicitly requires using SEMLINK at inference time. We will present the details of yu in Sec. 5. Intu- itively, it zeros out losses associated with joint label traces that violate SEMLINK constraints. During training, found that it is important to apply this constraint to both B_* and |_* labels.
Another option is retain the joint CRF for the jointly labeled dataset and cated CRF for PB-only sentences, Note that this setup different from the model in Sec. 4.1. As before, we let these two to share the same encoder and, for Eq. 4-6, they have dedicated trainable weights.
additional dedi-
During inference, we rely on the Viterbi de- coding associated with the joint CRF module to make predictions. In preliminary experiments, the joint CRF and the dedicated PropBank CRF achieve similar on PropBank arguments.
Where to apply 1 SEMLL Technically, the sum- mation over reduced label space can be applied at different places, such as the partition in Eq. 6.
We will report performances on this setting Sec. 7.2. In short, plugging the label filter y SEMLL to the joint CRF (therefore jointly labeled data) has little impact on Fl scores, thus we reserve it for the PropBank-only data (as in Eq. 8).
4.3 Marginal CRF
For partially labeled sequences, we take inspira- tion from Greenberg et al. (2018) to maximize the marginal distribution of those observed labels. In our joint CRF, the marginalization assumes uniform distribution over VN arguments that are paired with observed PB arguments. The learning objective is to maximize the probabilities of such label sequences as a whole:
5 Inference with SEMLINK
Here we discuss the implementation of the ysem. Remember that each pair of VerbNet class 7 and PropBank sense a uniquely determines a set of joint argument labels for the predicate u. For brevity, let us denote this set as SEML(u) (c.g., Tab. I). Eventually, we want the Viterbi-decoded
LSEyeye (s(w) log Z(x)
(7)
where LSEy (+) log Σ exp(-)
y

---

label sequence comply with SEML(u). That is,
sets (including WSJ and Brown). We will refer this data as the joint data column in Table 2. For semi-supervised learning, we incorporate the rest of the TropBank-only predicates in the CoNLLO5 training split. For development and testing, we use the splits in the joint dataset for fair comparison with prior work.
A(I ) ESEML(U)
vi, (VussEnLia) = ,1, yu v
(VigsemI (w) 11) $ v(P))
6.2 Training and Evaluation
where i denotes the location in a sentence, Du YP is the joint label sequence, consisting of (M uf" pairs. The constraint in Eq. translates as: if VerbNet argument is present in the predicate n SEMLINK entry, we prevent it from aligning to any PropBank arguments not defined in SEMLINK; and the same applies to PropBank arguments.
We adopt the same fine-tuning strategy as in Li et al. (2020)-we fine-tune twice since this generally out- performs fine-tuning only once, even with the same number of total epochs. In the first round, we fine- tune our model for 20 epochs. In the second round, restart the optimizer and learning rate sched- uler, and fine-tune for epochs. In both rounds, checkpoints with the highest average VN/PB devel- opment Fl are saved. For different model variants, we report the average F1 from models trained with random seeds.
This constraint can be easily implemented by a masking operation on the emission scores of the joint CRF, thus can be used af both training and inference time. During inference, it effectively ig- nores those label sequences with SEMLINK viola- tions during Viterbi decoding:
For the SEMLINK constraint, we use the official mapping between VN/PB arguments When in- volved in constrained inference, we use the gold VerbNet classes 7) and PropBank senses For evaluation, in addition to the standard VN and PB FI scores, we also report the percent of predicates with predictions that are inconsistent with SEMLINK, denoted by p.
VP' arg max y
s(V)
(10)
SEM.
In Sec. 6, we will show that using Eq. 10 always improves the overall SRL scores.
VerbNet Label Completion. For models based our joint CRE, we mask out joint labels that are not defined in y during inference, similar Eq. 8. For models with separate VN and PB classifiers (in Sec. 4.1), we enforce the VN's Viterbi decoding to only search arguments that are compatible with the gold y in SEMLINK. Furthermore, we always use the constraint (Eq. 9) in the completion mode.
6.3 Performance on SEMLINK Extracted CoNLLO5
We want compare models trained on the joint dataset and variants (in Sec. 4) on the semi- supervised setup. Table presents their perfor- mances along with SEMLINK violation rates model predictions. Note that the ground truth joint data bears no violation at all (i.c., p 0).
6 Experiments
In this section, we aim verify whether the com- patibility structure between VerbNet and PropBank (in the form of SEMLINK) has a positive impact on their sequence laheling performances.
Multitask involves SEMLINK violations. Firstly, we show the limitations of multitask learning. While the architecture is simple, the testing scores mostly outperform the previous state-of-the--rt, except the Brown PropBank F1. However, there is fair percentage of predicates having structurally wrong predictions, especially in the Brown test set. With semi-supervised learning, VN and PB Fls are improved on the Brown set while slightly lowered on the WSSJ VN. This also comes with a degraded SEMLINK error rate (3.43 - 4.08 on WSJ and 8.71 - 10.48
6.1 Data
We follow the prior state-of-the-art (Gung and Palmer, 2021) in extracting VerbNet labels from the CoNLL05 dataset using the SEMLINK corpus. We use the same version of SEMLINK to extract the data for training and evaluation. Therefore, our Fl scores are directly comparable with theirs (denoted as IwC52021 in Table 2) The resulting dataset accounts for about 56% of the CoNLLO5 predicates, across training, development and test
htttps: thub. cow www.///uccceeeee wwwecueclaaeeeeee cww.r/ ink/blob/ master/instancee//sell ink-2

---

Train
Inf
Dev
WSJ
Brown
Data Fine-tune
SEML
VN
PB
P
VN
p
VN
PB
pl
IWCS2021 (gold & o)
88.2
88.7
83.0
82.8
Joint
Multitask
x
89.51
87.91
3.68 06
90.74 23
89.33,1s
3.43
83.69
81.73,98
8.71
Joint
90.20 22
87.58
89.12.16 91.42.11 11
85.90
82.56,52
Multitask
x
89.56
88.30
4.82 35
90.45 0M
89.34,7
4.08
84.28
82,51,70
10.48
Semi
Joint+CRFm
90.73 32
88.42.11
0
91.52.36 36
89.30 29
85.72.13 0
82.33.31
Marginal
90.91
88.68
0
91.74
89.51
0
885.39
82,461.06
Marginalsasu
90.87 32
88.59
24
91.55.28
89.49.
86.39 17
83.04.37
Table 2: Model performances SEMLINK extracted CoNLLO5 data. Each data point represents the meanad of 3 random runs. P: percentage of predicates with argument prediction that violates SEMLINK structures. IWCS2021 numbers reported in (Gung and Palmer, 2021).
Train
Inf
Dev
WSJ
Brown
Data
Fine-tunex2
SEML.
VN
PB
p
VN
PB
P
VN
PB
Joint
Joint
x
89.28
87.70
1.58
90.73
89.10
1.59
84.43
82.48
5.99
Joint
/
90.20
87.58
0
91.42
89.12
85.90
82.56
Joint+CRFfp
x
89.66
88.31
2.74
90.61
89.21
2.37
83.53
81.87
7.59
Joint+CRFra
90.73
88.42
0
91.52
89.30
85.72
82.33
Semi
Marginal
x 89.79
88.50
2.92
90.73
89.30
2.39
83.23
81.88
6.74
Marginal
90.91
88.68
91.74
89.51
85.39
82.46
Marginal
x
89.53
88.55
2.29
90.70
89.46
1.98
83.64
82.71 83.04
7.77
MarginalsEML
90.87
88.59
91.55
89.49
86.39
Table 3: Impact of SEMLINK at inference time. Each data point represents the average of 3 random runs. Improve- ments on VN Fls are both substantial and significant (p-value & 0.01).
on Brown). While SEMLINK inconsistency is not reported in (Gung and Palmer, 2021), we believe that, due the nature of multitask learning, SEMLINK errors are inevitable.
the decoding casier. Likely due to the higher gran- ularity of VerbNet argument types compared to PropBank, majority of the improvements the VN Fls.
Joint CRF outperforms multitask learning. A direct comparison between the Multitask v.s. Joint. Our joint CRF obtains higher overall SRL Fl across the WSJ and the Brown sets. A similar obser- vation applies to the semi-supervised setting where Multitask compares to Joint+CRFpp. pa- Most of such improvements from the use of inference-time SEMLINK constraints.
Does semi-supervised learning make a differ- ence? The answer is that it depends. For Multi- lask, using PropBank-only data traded off bit on the overall WSJ F1 but improved the out-of-domain performances. Accompanied with this tradc-off is the slightly higher inconsistency rate p. The Join++CRFpp model tells an opposite story that the partially labeled data is favorable on the in- domain test but not so in the out-of-domain test. This observation is also consistent with both the Marginal CRF and constrained Marginal model (Marginalsemn.) Furthermore, when performance improves, the margins on VN and PB fairly distributed. Finally, we should note that, neither the Coint+CRFrs nor the Marginal have better Brown Fl than the Joint, meaning that they did not use the PB-only data efficiently.
Inference with SEMLINK improves SRL. In Table 3, we do side-to-side comparison of using versus not using the SEMLINK structure during in- ference. We do so for cach modeling variant. With constrained inference, models longer have SEM- LINK structural violations (p 0). And this results in clear trend where using SEMLINK systemati- cally improves the Fl scores. We hypothesize this is due to the reduced search space which makes

---

Impact of marginal CRE. We compare the Joint+CRF pu Marginal to see how single CRF handles partially labeled data. The latter outper- forms the former consistently by 0.2 on the in- domain test set but performed slightly worse on the out-of-domain Brown set. Comparing to the Joint model, it seems that naively applying marginal CRF leads to even worse generalization.
6.4 VerbNet Label Completion from PropBank
As discussed in Sec. 1, we also aim to address the realistic use case of VerbNet completion. Ta- ble 4 summarizes the performances for VerbNet argument prediction when gold PropBank argu- ments are given. In the completion mode, the Joint model performs generally better than all the semi- supervised models. This phenomenon is likely be- cause the Joint model is optimized for the prob- ability P(y", yf x)), while the semi-supervised models, in one way or the other, have a term for End P(y", v² (r) on PB-only data. The latter term does not explicitly boost model's discrimina- tive capability on the unique ground truth.
Constrained marginal model improves general- ization. We want to if our constrained model can help learning. Modeling PropBank-only data with a separate classifier (i.e., the Multitask and Joins+ CRFaa failed to do so (although they indeed work better on the in-domain WSJ). In contrast, our constrained Marginalsemn. apparently learns from the PropBank-only data more efficiently, achiev- ing strong in-domain performance and substan- tially better out-of-domain generalization. This suggests that learning with constraints works better with partially labeled data. Interestingly though, seems that the constraint is optional for fully an- notated data since the MarginalsEMI. only enables the constraint on PB-only data. We verify this phc- nomenon in Sec. 7.2 with an ablation study.
In addition to the Zwp input construction in Sec. 3, we propose a special construction #ccore for the VN completion mode by using the PB argu- ments as text input.:
FCOMp [CLS wir SLP yf de yo Yot1 SEP]
where uf denotes the PropBank argument label for the i-th word. For the predicate word, predicate feature (i.e. lemma and sense). Com- pared Twr, this formulation makes the compu- tation less efficient as the input 3comp no longer shared across different predicates. However, it of- fers a more tailored input signal and delivers above 99 Fl on both WSJ and Brown.
the
Statistical Significance of Constrained Inference. We measure statistical significance using a t-test im- plemented by Dror et al. (2018) on predictions from models in Table 3. For each model, we compare inference with and without SEMLINK constraints (Sec. 5). For a fair comparison, we limit predic- tions from the model trained with the same random seed in each test, and apply the test for all random seeds (3 in total). We observe that the improve- ments VerbNet FI's universally significant. The p-values are far less than 0.01 across differ- ent testing data, models, and random seeds. This aligns to the observation in Table 3 that VN has a substantial FI boost while PB F1 improvements tend to be marginal.
Data
Model
WSJ VN
Brown VN
Zwe
Joint
Multitask
83..
86.26
Joint
99.83
98.12
Multitask
93.08
85.71
Semi
Joint+CRF
99.81
97.88
Marginal
99.76
97.45
Marginal
99.68
97.71
#CoMP
Joint
Joint
99.85
99.02
Table 4: VN completion with gold PB labels. Results averaged models trained with random seeds. Teom:: Input construction for the complction mode.
To look closer, we examined the predictions of a Joint model on the Dev set (1, 794 predicates), and found that, after using SEMLINK during inference, 51 wrongly predicted predicates in VN SRL were corrected (i.e., improved predicate-wise F1), and no predicates received degraded F1. However, for PropBank SRL, there were 12 predicates corrected by the constraint while 6 became errors.
7 Analysis
We report statistical metrics Sec. 7.1. In Sec. 7.2, we analyze the use of constrained learning on the jointly labeled data.

---

7.1 Variance of SRL Models
corruption (%)
10
A majority of Fis in Tab. 2 vary in a small range. Models trained on joint-only data show higher vari- ance on the out-of-domain Brown test set. Among the semi-supervised models, the Marginal models exhibit high Fl variance on the Brown set while the MarginalsemI. models work more stably.
VN
91.23
89,45
83.64
82.90
79.02
PB
88.82
88.23
87.32
85.56
84.66
corruption (%)
10
20
30
VN FI
91.23
90.35
89.55
87.18
84.94
PB FI
88.82
88.17
87.21
84.92
83.11
Table 6: Performance curve w.r.t. predicate label cor- ruption, measured using the Marginalsesa trained with random seed 1. ry: VN class. a: PB sense.
7.2 Impact of Learning with SEMLINK Constraint on Joint Data
In Table 5, we use the form of constrained learning in Eq. but apply it on the joint CRF loss over the jointly labeled training data. Note that the con- straint term only affects the denominator part in Eq. 6. Overall, the effeet of SEMLINK at train- ing time seems small. On the WSJ test set, both VN and PB Fls are fairly close. The Brown test Fis have drop, especially on VN, suggesting that constrained learning on the joint data is not needed.
work investigated inference with constraints (e.g. Punyakanok et al., 2004; Surdeanu et al., 2007; Punyakanok et al., 2008). Other work developed constrained models for learning (e.g. Chang et al.. 2012): incorporated constraints with emerging neural models (e.g. Riedel and Meza-Ruiz, Ruiz, 2008; Fürstenau and Lapata, 2012; Täckström et al., 2015; FitzGerald et al., 2015; Li et al., 2020).
VerbNet SRI., on the other hand, is often stud- ied as comparison or a helper for PropBank SRL (Kuznetsov and Gurevych. 2020). Yi et al. (2007) showed that the mapping between Verb- Net and PropBank can be used disambiguate PropBank labels. It has been shown that model performance on VerbNet SRL affected more by predicate features than PropBank SRL (Zapirain et al., 2008). In sense, our observation that Verb- Net Fl gains larger improvements from SEMLINK is also consistent with prior work. Beyond compar- ison work, Kazeminejad et al. (2021) explored the downstream impact of VerbNet SRL and showed promising uses in entity state tracking.
Joint
WSJ
Brown
SEMLINK
VN
PB
VB
PB
X
91.42
89.12
85.90
82.56
2
91.42
89.21
85.41
82.46
Table 5: Ablation of SEMLINK constraint during train- ing using the Joint CRF / indicates SEMLINK con- straint applied; and indicates not,
7.3 Reliance on Gold Predicate Labels
While we focused on experiments that use given gold predicate labels (i.e.. VN class and PB sense o), our models do not rely on them. Intu- itively, there will be a performance degradation argument labeling when predicate labels are not always accurate. To study this degradation curve, we experiment with randomly corrupted predicate classes/senses, to illustrate the perfor- mance dependency on predicate disambiguation. Specifically, we perturb the development split by randomly swapping a predicate's 's VN class (or PB sense) with another one from the dataset and ob- serve how well our best model performs. This setup essentially simulates real-world testing scenario where predicate labels on either VN/PB imper- fect. Results in Tab. suggest that Fl degradation happens in a smooth way due to error propagation.
Multitask Learning The closest work to this pa- per is Gung and Palmer (2021). Instead of model- ing argument labels and predicate classes via mul- titasking, we adopted simpler design and focused on joint SRL argument labeling. This comes with two benefits: 1) focused design that models joint labels; 2) easy extension for using marginal CRF for partially labeled data. Other technical differences include a generally better transformer (i.c., ROBERTA) instead of BERT (Vaswani et al., 2017), simpler input construction, and our proposal of the completion mode.
Marginal CRF
Greenberg et al. (2018) explored the use of marginal CRF on disjoint label sequences in the biomedical domain. Disjoint label sequences are concatenated into one, thus requiring dedicated decoding to reduce inconsistency w.r.t. various
8 Discussion
The use of constraints in SRL has a long history that mostly focuses on the PropBank SRL task. Earlier

---

structure patterns (e.g., aligned BIO pattern.. In this paper, we took step further by pairing label sequences to form a joint SRL task, allowing an easy interface for injecting decoding constraints.
Fl. In contrast, we assumed sentences in the corpus are independent.
Why does PropBank seem more difficult? We hypothesized the reason to be less granularity argument labels and more ambiguous label assign- ments, As mentioned in Sec. 3, prior work bene- fited from using dedicated label text/definition auxiliary input. We only used such features at the predicate level, implicitly trading off potential gains on PB Fl for more efficient computation.
Broader Impact Recent advantages in large lan- guage models (LLM) have shown promising per- formance in semantic parsing tasks (c.g. Drozdov et al., 2022; Mekala et al., 2022; Yang et al., 2022). A well-established approach is via iterative prompt- ing (c.g.. Chain-of-Thought (Wei al., 2022)) and potentially using in-domain examples for prompt construction. While such work bears many techni- cal differences from this work, there are advantages that can potentially be shared. For instance, our direct use of knowledge base (SEMLINK in our case) allowed for guarantee of O violations; and LLM--based work is less reliant on training data. Another scenario is when treating semantic struc- tures as explicit intermediate products, such as for language generation. Our joint modeling allows for e 99% accuracy in converting PropBank argu- ments to VN arguments. When using such labels for prompted inference, it can make fewer errors.
Marginal model's capacity at handling con- straints. In this paper, we focused on the SEM- LINK constraint for compatible label sequences. There is broad spectrum of SRL constraints in prior work (Punyakanok ct al., 2008), some of them do not easily fit in the marginalization formulation, such as the unique core role constraint.
Acknowledgement
We thank the reviewers for their helpful in- sights and comments. We gratefully acknowl- edge the support of NSF under grants #1801446 (SATC) and #1822877 (Cyberlearning), and of DARPA CwC (subcontracts from UIUC and SIFT), DARPA AIDA Award 1--220001---00016 (RAM- FIS), DARPA KAIROS 1AA8550-192--1 19-70-0--9921100-A22- 0047-5005 (RESIN, sub to RPI, Pl: Heng Ji),
Conclusions In this work, we presented model that learns from compatible label sequences for the SRL task. The proposal includes joint CRF de- sign, extension for learning from partially labeled data, and reasoning and learning with SEMLINK constraints. On the VerbNet and PropBank bench- mark based on CoNLLO5, our models achieved state-of-the-ar performance with especially strong out-of-domain generalization. For the newly pro- posed task of completing VerbNet arguments given PropBank labels, our models are near perfect, achieving over 99 Fl scores.
as well as DTRA HDTRA1-16-1- 16-00-------000//rojeee 1553695 (eTASC Empirical Evidence for a Theo-
retical Approach to Semantic Components).
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies of any government agency.
9 Limitations
References
Towards fully end-to-end parser. Our model ar- chitecture is on the track of end-to-end SRL. parser, but still assumes gold predicate positions and predicate attributes are given. fully end-to-end parser can take sentences in raw text and output disjoint label sequences. While doing so can make computation less efficient (e.g., requiring substan- tially larger memory for training), it can bring users convenience.
Xavier Carreras and Lluis Marquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role la- beling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLI- 2005), pages 152-164, Ann Arbor, Michigan. Asso- ciation for Computational Linguistics.
Ming-Wes Wei Chang, Lev Ratinov, and Dan Roth. 2012. Structured Learning with Constrained Conditional Models. Machine learning. 88(3):399 431.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and
Involving document context. Gung and Palmer (2021) showed that using neighboring sentence pre- diction with transformer positively impacts parsing
Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the

---

North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 4171- 4186.
(LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 123-132, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Ili Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on prob- ing results. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 171-182, Online. Association for Computational Linguistics.
Tao Li, Parth Anand Jawale, Martha Palmer, and Vivek Srikumar. 2020. Structured tuning for semantic role labeling. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Länguistics, pages 8402-8412, Online, Association for Computa- tional Linguistics.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. arXiv preprint ar 220: / 15003.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL pars- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2051-2060, Melbourne, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1
Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. Proceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 960-970.
Ana Marasović and Anette Frank. 2018. SRL4ORL: Im- proving opinion role labeling using multi-task learn- ing with semantic role labeling. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Techmologies. Volume I (Long Pa- pers), pages 583-594, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Hagen Fürstenau and Mirella Lapata. 2012. Semi- supervised semantic role labeling via structural align- ment. Computational Linguistics, 3851)1335-171
Nathan Greenberg, Trapit Bansal, Patrick Verga, and An- drew McCallum. 2018 Marginal likelihood training of BiLSTM-CRF for biomedical named entity recog- nition from disjoint label sets. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2824-2822 Brussels, Belgium. Association for Computational Linguistics.
Dheeraj Mekala, Jason Wolfe, and Subhro Roy. 2022. Zerotop: Zero-shot task-oriented semantic pars- ing using large language models. arXiv preprint 1081
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguisties, 31(1):71- 106.
James Gung and Martha Palmer. 2021. Predicate repre- sentations and polysemy in VerbNet semantic parsing. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 51-62, Groningen, The Netherlands (online). Association for Computational Linguistics.
Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic role labeling. Syuthesis Lectures on Human Language Technologies, 301)
Sameer Pradhan, Julia Bonn, Skatje Myers, Kathryn Conger. Tim O' g'gorman, James Gung, Kristin Wright- bettner, and Martha Palmer. 2022. PropBank comes of Lagger,, smarter, and more diverse. In Pro- ceedings of the 11th Joint Conference on Lexical and Computational Semanticx, pages 278-288, Seattle, Washington. Association for Computational Linguis- tics.
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettle- moyer. 2017. Deep semantic role labeling: What works and what's 's next. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 473-483, Vancouver, Canada. Association for Computational Linguistics.
Dan Hendrycks and Kevin Gimpel. 2016. Gaus- sian error linear units (gelus). arXiv preprint arXiv:1606.08411
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Seventeenth Conference on Computational Natural Language Learning. pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Ghazaleh Kazeminejad, Martha Palmer, Tao Li, and Vivek Srikumar. 2021. Automatic entity state annota- tion using the VerbNet semantic parser. In Proceed- ings of the Joint 15th Linguistic Annotation Workshop

---

Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34(2):257-287.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le. and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24437, Curran Associates, Inc.
Vasin Punyakanok, Dan Roth, Wen-tau Yih, and Day Zi- mak. 2004. Semantic role laheling via integer linear programming inference. In COLING 2004: Proceed- ings of the 20th International Conference on Com- putational Linguistics, pages 1346-1352, Geneva, Switzerland. COLING.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Fun- towicz, et al. 2019. Transformers: State-of-the- art natural language processing. arXiv preprint
Reinhard Rapp. 2022. Using semantic role labeling to improve neural machine translation. In Proceedings of the Thirteenth Language Resources and Evalua- tion Conference, pages 3079-3083, Marseille, France. European Language Resources Association.
Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin, and Diyi Yang. 2022. SEQZERO: Few-shot compositional semantic parsing with se- quential prompts and zero-shot models. In Findings of the Association for Computational Linguistics:
Sebastian Riedel and Ivan Meza-Ruiz. 2008. Collec- tive semantic role labelling with Markov logic. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning.
NAACL 2022. pages 49 60- Seattle, United States. Association for Computational Linguisties.
Szu-ting Yi, Edward Loper, and Martha Palmer. 2007. Can Semantic Roles Generalize Across Genres? Human Language Technologies 2007: The Confer- ence of the North American Chapter of the Associ- ation for Computational Linguisties; Proceedings of the Main Conference, pages 548-555, Rochester, New York. Association for Computational Linguis- tics.
KK Schuler. 2005. Verbnet: A broad-coverage. com- prehensive verb lexicon. Ph. D. Thesis, University of Pennsylvania.
Jacob Solawetz and Stefan Larson. 2021. LSOIE: A large-scale dataset for supervised open information extraction. In Proceedings of the 16th Conference of the European Chapter of the A Association for Compu- tational Linguistics: Main Volume. pages 2595 2600, Online. Association for Computational Linguistics.
Beñat Zapirain, Eneko Agirre, and Lluís Mårquez. 2008. Robustness and generalization of role sets: PropBank vs. VerbNet. In Proceedings of ACL-08: HLT, pages 550-558, Columbus, Ohio. Association for Compu- tational Linguistics.
Kevin Stowe, Jenette Preciado, Kathryn Conger, Su- san Windisch Brown, Ghazaleh Kazeminejad. James Gung, and Martha Palmer. 2021. Seml .ink 2.0: Chas- ing lexical resources. In Proceedings of the 14th In- lernational Conference on Computational Semantics (IWCS), pages 222-227, Groningen, The Netherlands (online). Association for Computational I. Linguistics.
Li Zhang, Ishan Jindal, and Yunyao Li. 2022. Label definitions improve semantic role labeling. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages Seattle, United States. Association for Computational Linguistics.
Michalina Strzyz, David Vilares, and Carlos Gómez- Rodríguez. 2019. Sequence labeling parsing by learn- ing across representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5350-5357, Florence, Italy. Asso- ciation for Computational Linguistics.
Jie Zhou and Wei Xu. 2015. End-to-end learning of se- mantic role labeling using recurrent neural networks. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers). pages 1127- 1137, Beijing, China. Association for Computational Linguistics.
Mihai Surdeanu, Lluís Mårquez, Xavier Carreras, and Pere R Comas. 2007. Combination strategies for semantic role labeling. Journal of Artificial Intelli- gence Research, 29: 105-151.
A Appendix
Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Efficient inference and structured learning for semantic role labeling. Transactions of the Associa- tion for Computational Linguistics.
A.1 Hyperparameters
As discussed in Sec. 6, we adopted a 2-stage fine- tuning for each of our models. In the first round, we fine-tune for 20 epochs with initial learning rate of 10 In the second round, we fine-tune for another epochs with fresh start on the opti- mizer. Across the two stages, we applied a dropout
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Pro- cessing Systems.

---

layer with rate 0.5 (which is preliminarily grid- searched from [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]) before each of the linear layers Sec. The ROBERTA transformer uses the built-in default configurations which implemented by Wolf et al. (2019).
The learning rate (A) in the second round of fine-tuning varies by model. We preliminarily grid- searched from [3 x 10 1 10 5 10 5, x 10-*). Specifically, for each learning rate, trained 3 random models and chose the one with the highest overall SRL Fl on the Dev set. We put their values in Table 7.
Data
Model
A
Joint
Multitask
1 10-4
Joint
1 10
Multitask
1 10-4
Semi
Joint+CRFpn
1 10
Marginal
10-5
Marginalsema.
1 10-5
Table 7: Learning rate in the second round of fine-tuning for each model in Table


Reviews:
[{"review_id": "gpvduTkDqT", "reviewer": "Reviewer_Evjp", "summary": "", "questions": "Question A：When constructing \"x_wp,\"  did you predict the predicates and senses first or did you directly use the correct answers? If the predicted answers were used, does that imply that both \"x\" and \"x_wp\" underwent separate encoding, with the encoding result of \"x\" being utilized for predicting predicates?\n\nQuestion B: In line 406, you mentioned that correct VerbNet classes and PropBank senses were used during constrained decoding. Have you attempted using predicted answers?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and suggestions. We have added more illustration and analysis. We will such these evidences to strengthen our work.\n\n>The writing of this paper requires a little improvement, as certain sections are difficult to comprehend. The foundational aspect of SEMLINK is inadequately introduced. For instance, in section 5, readers less familiar with SEMLINK might struggle to grasp concepts like \"SEML(u)\" and the associated inferences. I believe the authors could enhance understanding by incorporating more illustrative examples.\n- We will add examples to motivate and better illustrate the structures of SemLink. One such example can be the predicate “value”:\n| PB sense | VN class | Admissible alignments |\n|-----|-----|-----|\n| 01 | price-54.4 | Arg0-Agent; Arg1-Theme; Arg2-Value |\n| 02 | price-54.4 | Arg0-Agent; Arg1-Theme; Arg2-Value |\n| 02 | admire-31.2 | Arg0-Experiencer; Arg1-Stimulus |\n\n- With a given predicate, each PB sense can be associated with multiple VN classes; and each VN class can also be associated with multiple PB senses. The PB sense and VN class jointly determine the set of admissible core arguments. Our models are therefore built to use such structure for improved parsing.\n\n> Additionally, as mentioned by the authors in the Limitation section, the model in this study relies on gold predicate positions and predicate attributes. This limitation diminishes the practicality of the model in real-world scenarios.\n- We want to clarify that our work focuses on experiments that use gold predicate attributes, but **does not rely on** them. Indeed, our experiments focused on scenario where such gold predicate features are given.\n- We chose such focus since prior works (e.g., Gung and Palmer IWCS21) have shown that predicate sense disambiguation tends to give high accuracy (e.g., >97% on WSJ and >91% on Brown in their paper).\n- Below, we show how our best model performs when predicate attributes are noisy. We experimented with randomly corrupted predicate attributes, to illustrate the performance dependency on predicate sense disambiguation. Specifically, we perturb the validation dataset by randomly swapping a predicate’s VN class (or PB sense) with another one from the dataset and observe how well our trained model performs on the validation set.\n  - VN class corruption\n| VN class corruption percent | 0% | 5% | 10% | 20% | 30% |\n|-----|-----|-----|-----|-----|-----|\n| VN F1 | 91.23 | 89.45 | 87.64 | 82.90 | 79.02 |\n| PB F1 | 88.82 | 88.23 | 87.32 | 85.56 | 84.66 |\n  - PB sense corruption\n| PB sense corruption percent | 0% | 5% | 10% | 20% | 30% |\n|-----|-----|-----|-----|-----|-----|\n| VN F1 | 91.23 | 90.35 | 89.55 | 87.18 | 84.94 |\n| PB F1 | 88.82 | 88.17 | 87.21 | 84.92 | 83.11 |\n  - This setup essentially simulates a real-world application scenario where predicate attributes are 100%, 95%, 90%, 80%, or 70% accurate accordingly. This study suggests that our model can use non-perfect predicate attributes. The resulting performance degradation happened in a smooth way and is expected due to error propagation.\n\n> Question A: When constructing \"x_wp,\" did you predict the predicates and senses first or did you directly use the correct answers?\n- When constructing the wp, we used the gold predicate attributes.\n\n> Question A: If the predicted answers were used, does that imply that both \"x\" and \"x_wp\" underwent separate encoding, with the encoding result of \"x\" being utilized for predicting predicates?\n- One option is to predict predicate senses from “x” and then use such senses to construct “x_wp”. In this case, yes, they undergo separate encodings.\n- Another option is to build an end-to-end differentiable pipeline that handles predicate senses as well as the SemLink structures in a “soft” way. This option would require substantial model changes.\n\n> Question B: In line 406, you mentioned that correct VerbNet classes and PropBank senses were used during constrained decoding. Have you attempted using predicted answers?\n- We experiment with predicting using corrupted predicate attributes (in the above table). The randomly corrupted predicate attributes affect the construction of x_wp and the associated argument alignment constraints. We expect to see both VN and PB performances will be dragged by such corruption in a smooth manner. This is indeed the case as shown in the table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiments conducted within this study are comprehensive, accompanied by a thorough analysis.\n\nThe proposed approach can be employed to facilitate mutual transformation between SRL data annotated under different standards, which holds significance for the automated construction of annotated datasets.", "reasons_to_reject": "The writing of this paper requires a little improvement, as certain sections are difficult to comprehend. The foundational aspect of SEMLINK is inadequately introduced. For instance, in section 5, readers less familiar with SEMLINK might struggle to grasp concepts like \"SEML(u)\" and the associated inferences. I believe the authors could enhance understanding by incorporating more illustrative examples.\n\nAdditionally, as mentioned by the authors in the Limitation section, the model in this study relies on gold predicate positions and predicate attributes. This limitation diminishes the practicality of the model in real-world scenarios.", "questions_to_authors": "Question A：When constructing \"x_wp,\"  did you predict the predicates and senses first or did you directly use the correct answers? If the predicted answers were used, does that imply that both \"x\" and \"x_wp\" underwent separate encoding, with the encoding result of \"x\" being utilized for predicting predicates?\n\nQuestion B: In line 406, you mentioned that correct VerbNet classes and PropBank senses were used during constrained decoding. Have you attempted using predicted answers?", "paper_topic_and_main_contributions": "This paper presents a joint model that simultaneously addresses two distinct SRL tasks with different formats, namely VerbNet SRL and PropBank SRL.\n \nBy incorporating SEMLINK (Stowe et al. , 2021) during decoding, the authors successfully prevent the generation of conflicting argument labels. \n\nThe experimental results indicate that the method proposed by the authors outperforms previous multitasking learning approaches. \nAnd when provided with the gold standard PropBank SRL labels, this method achieves an F1 score of 99 for predicting VerbNet SRL labels.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There is a missing period at line 383."}, "extra_text": {}}, {"review_id": "s1sxbnlOPT", "reviewer": "Reviewer_bCiH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our contributions. We will add the discussion to strengthen our work.\n\n> The existence of compatible verbnet and PropBank labels is an accident of the history of the field There are not many situations parallel to this, so the result is probably of interest only to a small subset of ACL researchers. The authors make little attempt to draw general lessons from their work.\n- The fuzzy alignment between PropBank and VerbNet is only one of the many bonus features offered by SemLink. SemLink also includes mappings between VerbNet and FrameNet, as well as direct mappings between PropBank and FrameNet.  Future work will explore the adaptation of this approach to FrameNet parsing, to determine how well it generalizes.  It could also be applied to Abstract Meaning Representations, which rely on the same PropBank frame files as SRL, but with different assumptions about applicability.\n- Here are some insights our work could offer (in addition to those in the Broader Impact section):\n  - The mutual benefit of VN and PB promotes a unified design of semantic structure/resource.\n  - Our use of consistency between label sequences to improve tagging accuracy conceptually connects to the idea of using consistency to improve LLM's prompted inference, although the modeling strategies are very different. We believe our work offers a positive signal to upcoming SRL models that are based on prompted inference of LLM: labeling consistency is of great potential.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results are state of the art, the methods sensible and the discussion useful.", "reasons_to_reject": "The existence of compatible verbnet and PropBank labels is an accident of the history of the field There are not many situations parallel to this, so the result is probably of interest only to a small subset of ACL researchers. The authors make little attempt to draw general lessons from their work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores various joint traiing decoding strategies for Propbank and VerbNet based semantic role labeling.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ufHVWDcPw4", "reviewer": "Reviewer_wA9b", "summary": "", "questions": "- A: Although the joint label space is pruned, it seems still pretty large, how would this affect training and testing efficiency? How slower would it be compared to the methods that predict separately?\n- B: How effective is CRF in these settings? With strong neural models, maybe simple token-wise labelers would obtain good results (maybe adding BIO constraint enforcing at testing time)?\n- C: The main experimenting dataset only covers 56% of the CoNLL05 predicates, are there any biases for these convertible predicates? For example, are they more frequent ones? Also, it would be better if more analysis can be provided based on predicate frequencies (my intuition is that probably low-frequent predicates would be the ones that benefit more from joint learning).\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments and insights on broadening the impact of our work. We will add the discussions and additional analysis to strengthen our work.\n\n> The experiments only involve joint prediction of PB and VN, where there are abundant joint data and high-coverage mapping resources. The paper could be much stronger and more interesting if it could be extended to scenarios where there are less such joint resources (like PropBank/VerbNet & FrameNet).\n- This is a good question. There are two challenges to joint predictions. One is how to max out model performances with existing joint data at hand. The other one is how to make it generalize to resources with less or no alignments. Our work focused on the first question.\n\n> The performance gaps between different methods seem small.\n- We genuinely believe a 3.4 (VN) improvement over the prior state-of-the-art is not a small one for parsing task. The momentum is from 83.0 to 86.4. Granted that our improvement on the PB axis is relatively smaller (0.8 over baseline and 0.2 over prior best).\n\n> Though this is not quite surprising giving strong neural models, it would be more interesting to explore scenarios where the proposed methods could be more helpful (such as low-resource cases and less-frequent predicates).\n- We believe this is indeed an interesting space to explore. It will possibly lead to similar observations in [1] that external structures bring a larger performance boost when there are very few annotated sentences.\n- We want to point out that our semi-supervised models are in a sense on such low-resource settings. There is a wider availability of PB-annotated sentences but much fewer VN-annotated ones. Our semi-supervised models benefit from those PB-only annotations to improve performances on the lower-resourced VN task.\n- We present analysis on performance over different predicate class frequency in our response to your Question C.\n\n[1]: Structured Tuning for Semantic Role Labeling. Li et. al. ACL 2020\n\n> Question A: Although the joint label space is pruned, it seems still pretty large, how would this affect training and testing efficiency? How slower would it be compared to the methods that predict separately?\n- The most noticeable part is the GPU memory consumption. In our multitask model, we have separate CRFs for VN and PB, each side has 100+ labels. In comparison, our joint label space is 685. With the same transformer backbone and the same CRF architecture, GPU memory is almost doubled for training. Accordingly, training time is also slowed by the relative larger label space. Our rough estimation is about 50% more training time when moving from multitask to joint model.\n- Inference speed is actually not noticeably slower, even though the complexity of Viterbi algorithm seems to suggest otherwise (O(Length * Label^2). This is likely due to the Viterbi decoding algorithm is already vectorized on the label dimension.\n\n> Question B: How effective is CRF in these settings? With strong neural models, maybe simple token-wise labelers would obtain good results (maybe adding BIO constraint enforcing at testing time)?\n- To the best of our knowledge, CRF and Viterbi is still common component for sequence labeling task, especially for BIO schemes. The BIO consistency is managed by the Viterbi decoding during testing, and is managed by CRF during training; since both of them are global optimizations sharing similar objectives.\n- Removing the CRF module, in our preliminary experience, always hurt the labeling performance by a large margin. The detailed ablation dates back to the introduction of LSTM+CRF[2] and such design is carried over onto BERT and RoBERTa encoders. The prior state-of-the-art (Gung and Palmer IWCS21) also followed this design.\n\n[2]: Bidirectional LSTM-CRF Models for Sequence Tagging. Huang et. al. 2015\n\n> Question C: The main experimenting dataset only covers 56% of the CoNLL05 predicates, are there any biases for these convertible predicates? For example, are they more frequent ones?\n- One potential “bias” could be the long-tailed distribution of predicates. This is indeed the case in our analysis below.\n\n> Question C: Also, it would be better if more analysis can be provided based on predicate frequencies (my intuition is that probably low-frequent predicates would be the ones that benefit more from joint learning).\n- We did an analysis on the validation split. Our finding (below) indicates that the average improvements tend to be \"uniform\" over predicate attribute frequencies. But both VN and PB improvements indeed tend to have higher variance at lower predicate frequency.\n- [vn_accuracy_diff](https://anonymous.4open.science/r/emnlp23_rebuttal-CCC6/vn_f1_diff.png) shows the VN accuracy gain of our best model over the joint multitask baseline. The x-axis indicates VN class distribution.The red line indicates the mean at each frequency.\n- [srl_accuracy_diff](https://anonymous.4open.science/r/emnlp23_rebuttal-CCC6/srl_f1_diff.png) shows the PB accuracy gain of our best model over the joint multitask baseline. The x-axis indicates PB sense distribution. The red line indicates the mean at each frequency.\n- Please note that the averaged improvements are relatively small on the validation set, thus margins are not always substantial in the above plots.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I think the direction explored in this work is interesting and important for the future development of unifying semantic resources.\n- The paper is well-written and easy to follow, and the experiments are well-conducted.\n", "reasons_to_reject": "- The experiments only involve joint prediction of PB and VN, where there are abundant joint data and high-coverage mapping resources. The paper could be much stronger and more interesting if it could be extended to scenarios where there are less such joint resources (like PropBank/VerbNet & FrameNet).\n- The performance gaps between different methods seem small. Though this is not quite surprising giving strong neural models, it would be more interesting to explore scenarios where the proposed methods could be more helpful (such as low-resource cases and less-frequent predicates).\n", "questions_to_authors": "- A: Although the joint label space is pruned, it seems still pretty large, how would this affect training and testing efficiency? How slower would it be compared to the methods that predict separately?\n- B: How effective is CRF in these settings? With strong neural models, maybe simple token-wise labelers would obtain good results (maybe adding BIO constraint enforcing at testing time)?\n- C: The main experimenting dataset only covers 56% of the CoNLL05 predicates, are there any biases for these convertible predicates? For example, are they more frequent ones? Also, it would be better if more analysis can be provided based on predicate frequencies (my intuition is that probably low-frequent predicates would be the ones that benefit more from joint learning).\n", "paper_topic_and_main_contributions": "This paper explores outputting multiple SRL label sequences with compatible labels, more specifically, jointly outputting PropBank (PB) and VerbNet (VN) labels. The authors propose a joint CRF model that outputs joint PB-VN labels, which is shown to be generally better than a multi-task baseline, especially on the out-of-domain Brown test set. Furthermore, semi-supervised learning with PB-only data and inferencing with SemLink constraints are explored, which could effectively bring benefits.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes joint training and decoding strategies for Propbank and VerbNet-based semantic role labeling through the incorporation of SemLink. Experiments show SRL predictions to be on par or better than previous approaches. 

Soundness:
All reviewers agree on their "soundness" score of 4, i.e. that the presented experiments are comprehensive, the discussion is informative, and the presented results support the conclusions of the paper. This points to the potential of using two different SRL formalisms to support each other in decoding. 

However, it is also pointed out that some improvements are rather marginal. While Tables 1 and 2 present experiments averaged over three runs, they do not present a standard deviation score. Some of the bolded results in these tables are most certainly not statistically significant (for instance WSJ Joint decoding in Table 1 for PB increasing from 89.10 to 89.12). Still, as authors point out in their response, depending on the dataset and SRL type, some improvements are larger. 

Another challenge to soundness is that only a single dataset is considered that only covers 56% of the CoNLL05 predicates. The authors discuss in their response how this might bias the results. 

Excitement: 
All reviewers agree on an "excitement" score of 3. The main challenge to excitement is the narrow focus of the paper on two particular SRL formalisms linked by SemLink, the focus on a single dataset with derived VN predicates, and the moderate improvements in decoding scores. The authors argue in their response that their results in principle may transfer to other SRL resources like FrameNet, but no supporting experiments are presented in the paper. 

Clarity:
Reviewer agree that the paper is interesting to read, but could be improved for readers who are not familiar with details on the formalisms (more examples, better explanation of SemLink, concepts like "SEML(u)", etc.) The reviewers provide some clarification in their response. Should the paper be accepted, I would recommend clarifying these aspects (and perhaps even working with a Figure to better illustrate the overlap between the two formalisms.)
Decision: Accept-Findings