Dependency or Span, End-to-End Uniform Semantic Role Labeling

Abstract: Semantic role labeling (SRL) aims to discover the predicateargument structure of a sentence. End-to-end SRL without syntactic input has received great attention. However, most of them focus on either span-based or dependency-based semantic representation form and only show specific model optimization respectively. Meanwhile, handling these two SRL tasks uniformly was less successful. This paper presents an end-to-end model for both dependency and span SRL with a unified argument representation to deal with two different types of argument annotations in a uniform fashion. Furthermore, we jointly predict all predicates and arguments, especially including long-term ignored predicate identification subtask. Our single model achieves new state-of-the-art results on both span (CoNLL 2005, 2012) and dependency (CoNLL 2008, 2009) SRL benchmarks.

The purpose of semantic role labeling (SRL) is to derive the meaning representation for a sentence, which is beneficial to a wide range of natural language processing (NLP) tasks BIBREF0 , BIBREF1 . SRL can be formed as four subtasks, including predicate detection, predicate disambiguation, argument identification and argument classification. For argument annotation, there are two formulizations. One is based on text spans, namely span-based SRL. The other is dependency-based SRL, which annotates the syntactic head of argument rather than entire argument span. Figure FIGREF1 shows example annotations.
Great progress has been made in syntactic parsing BIBREF2 , BIBREF3 , BIBREF4 . Most traditional SRL methods rely heavily on syntactic features. To alleviate the inconvenience, recent works BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 propose end-to-end models for SRL, putting syntax aside and still achieving favorable results. However, these systems focus on either span or dependency SRL, which motivates us to explore a uniform approach.
Both span and dependency are effective formal representations for semantics, though for a long time it has been kept unknown which form, span or dependency, would be better for the convenience and effectiveness of semantic machine learning and later applications. Furthermore, researchers are interested in two forms of SRL models that may benefit from each other rather than their separated development. This topic has been roughly discussed in BIBREF19 , who concluded that the (best) dependency SRL system at then clearly outperformed the span-based (best) system through gold syntactic structure transformation. However, BIBREF19 johansson2008EMNLP like all other traditional SRL models themselves had to adopt rich syntactic features, and their comparison was done between two systems in quite different building styles. Instead, this work will develop full syntax-agnostic SRL systems with the same fashion for both span and dependency representation, so that we can revisit this issue under a more solid empirical basis.
In addition, most efforts focus on argument identification and classification since span and dependency SRL corpora have already marked predicate positions. Although no predicate identification is needed, it is not available in many downstream applications. Therefore, predicate identification should be carefully handled in a complete practical SRL system. To address this problem, BIBREF9 he2018jointly proposed an end-to-end approach for jointly predicting predicates and arguments for span SRL. Likewise, BIBREF11 cai2018full introduced an end-to-end model to naturally cover all predicate/argument identification and classification subtasks for dependency SRL.
To jointly predict predicates and arguments, we present an end-to-end framework for both span and dependency SRL. Our model extends the span SRL model of BIBREF9 he2018jointly, directly regarding all words in a sentence as possible predicates, considering all spans or words as potential arguments and learning distributions over possible predicates. However, we differ by (1) introducing unified argument representation to handle two different types of SRL tasks, and (2) employing biaffine scorer to make decisions for predicate-argument relationship.
The proposed models are evaluated on span SRL datasets: CoNLL 2005 and 2012 data, as well as the dependency SRL dataset of CoNLL 2008 and 2009 shared tasks. For span SRL, our single model outperforms the previous best results by 0.3% and 0.5% F INLINEFORM0 -score on CoNLL 2005 and 2012 test sets respectively. For dependency SRL, we achieve new state-of-the-art of 85.3% F INLINEFORM1 and 90.4% F INLINEFORM2 on CoNLL 2008 and 2009 benchmarks respectively.
SRL is pioneered by BIBREF20 gildea2002, which uses the PropBank conventions BIBREF21 . Conventionally, span SRL consists of two subtasks, argument identification and classification. The former identifies the arguments of a predicate, and the latter assigns them semantic role labels, namely, determining the relation between arguments and predicates. The PropBank defines a set of semantic roles to label arguments, falling into two categories: core and non-core roles. The core roles (A0-A5 and AA) indicate different semantics in predicate-argument structure, while the non-core roles are modifiers (AM-adj) where adj specifies the adjunct type, such as temporal (AM-TMP) and locative (AM-LOC) adjuncts. For example shown in Figure FIGREF1 , A0 is a proto-agent, representing the borrower.
Slightly different from span SRL in argument annotation, dependency SRL labels the syntactic heads of arguments rather than phrasal arguments, which was popularized by CoNLL-2008 and CoNLL-2009 shared tasks BIBREF22 , BIBREF23 . Furthermore, when no predicate is given, two other indispensable subtasks of dependency SRL are predicate identification and disambiguation. One is to identify all predicates in a sentence, and the other is to determine the senses of predicates. As the example shown in Figure FIGREF1 , 01 indicates the first sense from the PropBank sense repository for predicate borrowed in the sentence.
The traditional approaches on SRL were mostly about designing hand-crafted feature templates and then employ linear classifiers such as BIBREF24 , BIBREF25 , BIBREF12 . Even though neural models were introduced, early work still paid more attention on syntactic features. For example, BIBREF14 Fitzgerald2015 integrated syntactic information into neural networks with embedded lexicalized features, while BIBREF15 roth2016 embedded syntactic dependency paths between predicates and arguments. Similarly, BIBREF16 marcheggianiEMNLP2017 leveraged the graph convolutional network to encode syntax for dependency SRL. Recently, BIBREF17 Strubell2018 presented a multi-task neural model to incorporate auxiliary syntactic information for SRL, BIBREF18 li2018unified adopted several kinds of syntactic encoder for syntax encoding while BIBREF10 he:2018Syntax used syntactic tree for argument pruning.
However, using syntax may be quite inconvenient sometimes, recent studies thus have attempted to build SRL systems without or with little syntactic guideline. BIBREF5 zhou-xu2015 proposed the first syntax-agnostic model for span SRL using LSTM sequence labeling, while BIBREF7 he-acl2017 further enhanced their model using highway bidirectional LSTMs with constrained decoding. Later, BIBREF8 selfatt2018 presented a deep attentional neural network for applying self-attention to span SRL task. Likewise for dependency SRL, BIBREF6 marcheggiani2017 proposed a syntax-agnostic model with effective word representation and obtained favorable results. BIBREF11 cai2018full built a full end-to-end model with biaffine attention and outperformed the previous state-of-the-art.
More recently, joint predicting both predicates and arguments has attracted extensive interest on account of the importance of predicate identification, including BIBREF7 , BIBREF17 , BIBREF9 , BIBREF11 and this work. In our preliminary experiments, we tried to integrate the self-attention into our model, but it does not provide any significant performance gain on span or dependency SRL, which is not consistent with the conclusion in BIBREF8 and lets us exclude it from this work.
Generally, the above work is summarized in Table TABREF2 . Considering motivation, our work is most closely related to the work of BIBREF14 Fitzgerald2015, which also tackles span and dependency SRL in a uniform fashion. The essential difference is that their model employs the syntactic features and takes pre-identified predicates as inputs, while our model puts syntax aside and jointly learns and predicts predicates and arguments.
Given a sentence INLINEFORM0 , we attempt to predict a set of predicate-argument-relation tuples INLINEFORM1 , where INLINEFORM2 is the set of all possible predicate tokens, INLINEFORM3 includes all the candidate argument spans or dependencies, and INLINEFORM6 is the set of the semantic roles. To simplify the task, we introduce a null label INLINEFORM7 to indicate no relation between arbitrary predicate-argument pair following BIBREF9 he2018jointly. As shown in Figure FIGREF5 , our uniform SRL model includes four main modules:
 INLINEFORM0 token representation component to build token representation INLINEFORM1 from word INLINEFORM2 ,
 INLINEFORM0 a BiHLSTM encoder that directly takes sequential inputs,
 INLINEFORM0 predicate and argument representation module to learn candidate representations,
 INLINEFORM0 a biaffine scorer which takes the candidate representations as input and predicts semantic roles.
We follow the bi-directional LSTM-CNN architecture BIBREF26 , where convolutional neural networks (CNNs) encode characters inside a word INLINEFORM0 into character-level representation INLINEFORM1 then concatenated with its word-level INLINEFORM2 into context-independent representation. To further enhance the word representation, we leverage an external representation INLINEFORM3 from pretrained ELMo (Embeddings from Language Models) layers according to BIBREF27 ELMo. Eventually, the resulting token representation is concatenated as DISPLAYFORM0 
The encoder in our model adopts the bidirectional LSTM with highway connections (BiHLSTM) to contextualize the representation into task-specific representation: INLINEFORM0 , where the gated highway connections is used to alleviate the vanishing gradient problem when training very deep BiLSTMs.
We employ contextualized representations for all candidate arguments and predicates. As referred in BIBREF2 , applying a multi-layer perceptron (MLP) to the recurrent output states before the classifier has the advantage of stripping away irrelevant information for the current decision. Therefore, to distinguish the currently considered predicate from its candidate arguments in SRL context, we add an MLP layer to contextualized representations for argument INLINEFORM0 and predicate INLINEFORM1 candidates specific representations respectively with ReLU BIBREF28 as its activation function: INLINEFORM2 INLINEFORM3 
To perform uniform SRL, we introduce unified argument representation. For dependency SRL, we assume single word argument span by limiting the length of candidate argument to be 1, so our model uses the INLINEFORM0 as the final argument representation INLINEFORM1 directly. While for span SRL, we utilize the approach of span representation from BIBREF29 lee2017end. Each candidate span representation INLINEFORM2 is built by DISPLAYFORM0 
where INLINEFORM0 and INLINEFORM1 are boundary representations, INLINEFORM2 indicates a span, INLINEFORM3 is a feature vector encoding the size of span, and INLINEFORM4 is the specific notion of headedness which is learned by attention mechanism BIBREF30 over words in each span (where INLINEFORM5 is the position inside span) as follows : INLINEFORM6 INLINEFORM7 
For predicate and arguments, we introduce two unary scores on their candidates: INLINEFORM0 INLINEFORM1 
For semantic role, we adopt a relation scorer with biaffine attention BIBREF2 : DISPLAYFORM0 
 where INLINEFORM0 and INLINEFORM1 respectively denote the weight matrix of the bi-linear and the linear terms and INLINEFORM2 is the bias item.
The biaffine scorer differs from feed-forward networks scorer in bilinear transformation. Since SRL can be regarded as a classification task, the distribution of classes is uneven and the problem comes worse after the null labels are introduced. The output layer of the model normally includes a bias term designed to capture the prior probability of each class, with the rest of the model focusing on learning the likelihood of every classes occurring in data. The biaffine attention as Dozat and Manning (2017) in our model directly assigns a score for each specific semantic role and would be helpful for semantic role prediction. Actually, (He et al., 2018a) used a scorer as Equation (2), which is only a part of our scorer including both Equations ( EQREF14 ) and (). Therefore, our scorer would be more informative than previous models such as BIBREF9 .
The model is trained to optimize the probability INLINEFORM0 of the predicate-argument-relation tuples INLINEFORM1 given the sentence INLINEFORM2 , which can be factorized as: DISPLAYFORM0 
 where INLINEFORM0 represents the model parameters, and INLINEFORM1 , is the score for the predicate-argument-relation tuple, including predicate score INLINEFORM2 , argument score INLINEFORM3 and relation score INLINEFORM4 .
Our model adopts a biaffine scorer for semantic role label prediction, which is implemented as cross-entropy loss. Moreover, our model is trained to minimize the negative likehood of the golden structure INLINEFORM0 : INLINEFORM1 . The score of null labels are enforced into INLINEFORM2 . For predicates and arguments prediction, we train separated scorers ( INLINEFORM3 and INLINEFORM4 ) in parallel fed to the biaffine scorer for predicate and argument predication respectively, which helps to reduce the chance of error propagation.
The number of candidate arguments for a sentence of length INLINEFORM0 is INLINEFORM1 for span SRL, and INLINEFORM2 for dependency. As the model deals with INLINEFORM3 possible predicates, the computational complexity is INLINEFORM4 for span, INLINEFORM5 for dependency, which is too computationally expensive. To address this issue, we attempt to prune candidates using two beams for storing the candidate arguments and predicates with size INLINEFORM6 and INLINEFORM7 inspired by BIBREF9 he2018jointly, where INLINEFORM8 and INLINEFORM9 are two manually setting thresholds. First, the predicate and argument candidates are ranked according to their predicted score ( INLINEFORM10 and INLINEFORM11 ) respectively, and then we reduce the predicate and argument candidates with defined beams. Finally, we take the candidates from the beams to participate the label prediction. Such pruning will reduce the overall number of candidate tuples to INLINEFORM12 for both types of tasks. Furthermore, for span SRL, we set the maximum length of candidate arguments to INLINEFORM13 , which may decrease the number of candidate arguments to INLINEFORM14 .
According to PropBank semantic convention, predicate-argument structure has to follow a few of global constraints BIBREF25 , BIBREF7 , we thus incorporate constraints on the output structure with a dynamic programing decoder during inference. These constraints are described as follows:
 INLINEFORM0 Unique core roles (U): Each core role (A0-A5, AA) should appear at most once for each predicate.
 INLINEFORM0 Continuation roles (C): A continuation role C-X can exist only when its base role X is realized before it.
 INLINEFORM0 Reference roles (R): A reference role R-X can exist only when its base role X is realized (not necessarily before R-X).
 INLINEFORM0 Non-overlapping (O): The semantic arguments for the same predicate do not overlap in span SRL.
As C and R constraints lead to worse performance in our models from our preliminary experiments, we only enforce U and O constraints on span SRL and U constraints on dependency SRL.
Our models are evaluated on two PropBank-style SRL tasks: span and dependency. For span SRL, we test model on the common span SRL datasets from CoNLL-2005 BIBREF32 and CoNLL-2012 BIBREF31 shared tasks. For dependency SRL, we experiment on CoNLL 2008 BIBREF22 and 2009 BIBREF23 benchmarks. As for the predicate disambiguation in dependency SRL task, we follow the previous work BIBREF15 .
We consider two SRL setups: end-to-end and pre-identified predicates. For the former setup, our system jointly predicts all the predicates and their arguments in one shot, which turns into CoNLL-2008 setting for dependency SRL. In order to compare with previous models, we also report results with pre-identified predicates, where predicates have been beforehand identified in corpora. Therefore, the experimental results fall into two categories: end-to-end results and results with pre-identified predicates.
CoNLL 2005 and 2012 The CoNLL-2005 shared task focused on verbal predicates only for English. The CoNLL-2005 dataset takes section 2-21 of Wall Street Journal (WSJ) data as training set, and section 24 as development set. The test set consists of section 23 of WSJ for in-domain evaluation together with 3 sections from Brown corpus for out-of-domain evaluation. The larger CoNLL-2012 dataset is extracted from OntoNotes v5.0 corpus, which contains both verbal and nominal predicates.
CoNLL 2008 and 2009 CoNLL-2008 and the English part of CoNLL-2009 shared tasks use the same English corpus, which merges two treebanks, PropBank and NomBank. NomBank is a complement to PropBank with similar semantic convention for nominal predicate-argument structure annotation. Besides, the training, development and test splits of English data are identical to that of CoNLL-2005.
In our experiments, the word embeddings are 300-dimensional GloVe vectors BIBREF33 . The character representations with dimension 8 randomly initialized. In the character CNN, the convolutions have window sizes of 3, 4, and 5, each consisting of 50 filters. Moreover, we use 3 stacked bidirectional LSTMs with 200 dimensional hidden states. The outputs of BiLSTM employs two 300-dimensional MLP layers with the ReLU as activation function. Besides, we use two 150-dimensional hidden MLP layers with ReLU to score predicates and arguments respectively. For candidates pruning, we follow the settings of BIBREF9 he2018jointly, modeling spans up to length INLINEFORM0 for span SRL and INLINEFORM1 for dependency SRL, using INLINEFORM2 for pruning predicates and INLINEFORM3 for pruning arguments.
Training Details During training, we use the categorical cross-entropy as objective, with Adam optimizer BIBREF34 initial learning rate 0.001. We apply 0.5 dropout to the word embeddings and character CNN outputs and 0.2 dropout to all hidden layers and feature embeddings. In the LSTMs, we employ variational dropout masks that are shared across timesteps BIBREF35 , with 0.4 dropout rate. All models are trained for up to 600 epochs with batch size 40 on a single NVIDIA GeForce GTX 1080Ti GPU, which occupies 8 GB graphic memory and takes 12 to 36 hours.
We present all results using the official evaluation script from the CoNLL-2005 and CoNLL-2009 shared tasks, and compare our model with previous state-of-the-art models.
Span SRL Table TABREF15 shows results on CoNLL-2005 in-domain (WSJ) and out-of-domain (Brown) test sets, as well as the CoNLL-2012 test set (OntoNotes). The upper part of table presents results from single models. Our model outperforms the previous models with absolute improvements in F INLINEFORM0 -score of 0.3% on CoNLL-2005 benchmark. Besides, our single model performs even much better than all previous ensemble systems.
Dependency SRL Table TABREF19 presents the results on CoNLL-2008. J & N (2008b) BIBREF36 was the highest ranked system in CoNLL-2008 shared task. We obtain comparable results with the recent state-of-the-art method BIBREF11 , and our model surpasses the model BIBREF10 by 2% in F INLINEFORM0 -score.
To compare with to previous systems with pre-identified predicates, we report results from our models as well.
Span SRL Table TABREF22 shows that our model outperforms all published systems, even the ensemble model BIBREF8 , achieving the best results of 87.7%, 80.5% and 86.0% in F INLINEFORM0 -score respectively.
Dependency SRL Table TABREF29 compares the results of dependency SRL on CoNLL-2009 English data. Our single model gives a new state-of-the-art result of 90.4% F INLINEFORM0 on WSJ. For Brown data, the proposed syntax-agnostic model yields a performance gain of 1.7% F INLINEFORM1 over the syntax-aware model BIBREF18 .
To investigate the contributions of ELMo representations and biaffine scorer in our end-to-end model, we conduct a series of ablation studies on the CoNLL-2005 and CoNLL-2008 WSJ test sets, unless otherwise stated.
Table TABREF31 compares F INLINEFORM0 scores of BIBREF9 he2018jointly and our model without ELMo representations. We observe that effect of ELMo is somewhat surprising, where removal of the ELMo dramatically declines the performance by 3.3-3.5 F INLINEFORM1 on CoNLL-2005 WSJ. However, our model gives quite stable performance for dependency SRL regardless of whether ELMo is concatenated or not. The results indicate that ELMo is more beneficial to span SRL.
In order to better understand how the biaffine scorer influences our model performance, we train our model with different scoring functions. To ensure a fair comparison with the model BIBREF9 , we replace the biaffine scorer with their scoring functions implemented with feed-forward networks, and the results of removing biaffine scorer are also presented in Table TABREF31 . We can see 0.5% and 1.6% F INLINEFORM0 performance degradation on CoNLL 2005 and 2008 WSJ respectively. The comparison shows that the biaffine scorer is more effective for scoring the relations between predicates and arguments. Furthermore, these results show that biaffine attention mechanism is applicable to span SRL.
It is very hard to say which style of semantic formal representation, dependency or span, would be more convenient for machine learning as they adopt incomparable evaluation metric. Recent researches BIBREF37 have proposed to learn semantic parsers from multiple datasets in Framenet style semantics, while our goal is to compare the quality of different models in the span and dependency SRL for Propbank style semantics. Following BIBREF19 johansson2008EMNLP, we choose to directly compare their performance in terms of dependency-style metric through a transformation way. Using the head-finding algorithm in BIBREF19 which used gold-standard syntax, we may determine a set of head nodes for each span. This process will output an upper bound performance measure about the span conversion due to the use of gold syntax.
We do not train new models for the conversion and the resulted comparison. Instead, we do the job on span-style CoNLL 2005 test set and dependency-style CoNLL 2009 test set (WSJ and Brown), considering these two test sets share the same text content. As the former only contains verbal predicate-argument structures, for the latter, we discard all nomial predicate-argument related results and predicate disambiguation results during performance statistics. Table TABREF33 shows the comparison.
On a more strict setting basis, the results from our same model for span and dependency SRL verify the same conclusion of BIBREF19 johansson2008EMNLP, namely, dependency form is in a favor of machine learning effectiveness for SRL even compared to the conversion upper bound of span form.
This paper presents an end-to-end neural model for both span and dependency SRL, which may jointly learn and predict all predicates and arguments. We extend existing model and introduce unified argument representation with biaffine scorer to the uniform SRL for both span and dependency representation forms. Our model achieves new state-of-the-art results on the CoNLL 2005, 2012 and CoNLL 2008, 2009 benchmarks. Our results show that span and dependency SRL can be effectively handled in a uniform fashion, which for the first time enables us to conveniently explore the useful connection between two types of semantic representation forms.

Figure and table captions:
Figure 1: Examples of annotations in span (above) and dependency (below) SRL.
Table 1: A chronicle of related work for span and dependency SRL. SA represents syntax-aware system (no + indicates syntaxagnostic system) and ST indicates sequence tagging model. F1 is the result of single model on official test set.
Figure 2: The framework of our end-to-end model for uniform SRL.
Table 2: End-to-end span SRL results on CoNLL-2005 and CoNLL-2012 data, compared with previous systems in terms of precision (P), recall (R), F1-score. The CoNLL-2005 contains two test sets: WSJ (in-domain) and Brown (out-of-domain).
Table 3: Dependency SRL results on CoNLL-2008 test sets.
Table 4: Span SRL results with pre-identified predicates on CoNLL-2005 and CoNLL-2012 test sets.
Table 5: Dependency SRL results with pre-identified predicates on CoNLL-2009 English benchmark.
Table 6: Effectiveness of ELMo representations and biaffine scorer on the CoNLL 2005 and 2008 WSJ sets.
Table 7: Dependency vs. Span-converted Dependency on CoNLL 2005, 2009 test sets with dependency evaluation.

Questions about this paper:
what were the baselines?

Answers:
2008 Punyakanok et al. 
2009 Zhao et al. + ME 
2008 Toutanova et al. 
2010 Bjorkelund et al.  
2015 FitzGerald et al. 
2015 Zhou and Xu 
2016 Roth and Lapata 
2017 He et al. 
2017 Marcheggiani et al.
2017 Marcheggiani and Titov 
2018 Tan et al. 
2018 He et al. 
2018 Strubell et al. 
2018 Cai et al. 
2018 He et al. 
2018 Li et al.