Venue: EMNLP 2017

# Encoding Sentences with Graph Convolutional Networks

for Semantic Role Labeling

Diego Marcheggiani\({}^{1}\)&Ivan Titov\({}^{1,2}\)

\({}^{1}\)ILLC, University of Amsterdam

\({}^{2}\)ILCC, School of Informatics, University of Edinburgh

marcheggiani@uva.nl ititov@inf.ed.ac.uk

###### Abstract

Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard NLP pipeline. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of neural networks operating on graphs, suited to model syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark [CoNLL-2009] both for Chinese and English.

## 1 Introduction

Semantic role labeling (SRL) [53] can be informally described as the task of discovering _who_ did _what_ to _whom_. For example, consider an SRL dependency graph shown above the sentence in Figure 1. Formally, the task includes (1) detection of predicates (e.g., _makes_); (2) labeling the predicates with a sense from a sense inventory (e.g., _make_.01); (3) identifying and assigning arguments to _semantic roles_ (e.g., _Sequa_ is A0, i.e., an agent / 'door' for the corresponding predicate, and _engines_ is A1, i.e., a patient / 'an affected entity'). SRL is often regarded as an important step in the standard NLP pipeline, providing information to downstream tasks such as information extraction and question answering.

The semantic representations are closely related to syntactic ones, even though the syntax-semantics interface is far from trivial [8]. For example, one can observe that many arcs in the syntactic dependency graph (shown in black below the sentence in Figure 1) are mirrored in the semantic dependency graph. Given these similarities and also because of availability of accurate syntactic parsers for many languages, it seems natural to exploit syntactic information when predicting semantics. Though historically most SRL approaches did rely on syntax [67, 43, 46, 45], the last generation of SRL models put syntax aside in favor of neural sequence models, namely LSTMs [46, 47], and outperformed syntactically-driven methods on standard benchmarks. We believe that one of the reasons for this radical choice is the lack of simple and effective methods for incorporating syntactic information into sequential neural networks (namely, at the level of words). In this paper we

Figure 1: An example sentence annotated with semantic (top) and syntactic dependencies (bottom).



propose one way how to address this limitation.

Specifically, we rely on graph convolutional networks (GCNs) Duvenaud et al. (2015); Kipf and Welling (2017); Kearnes et al. (2016), a recent class of multilayer neural networks operating on graphs. For every node in the graph (in our case a word in a sentence), GCN encodes relevant information about its neighborhood as a real-valued feature vector. GCNs have been studied largely in the context of undirected unlabeled graphs. We introduce a version of GCNs for modeling syntactic dependency structures and generally applicable to labeled directed graphs.

One layer GCN encodes only information about immediate neighbors and \(K\) layers are needed to encode \(K\)-order neighborhoods (i.e., information about nodes at most \(K\) hops away). This contrasts with recurrent and recursive neural networks Elman (1990); Socher et al. (2013) which, at least in theory, can capture statistical dependencies across unbounded paths in a trees or in a sequence. However, as we will further discuss in Section 3.3, this is not a serious limitation when GCNs are used in combination with encoders based on recurrent networks (LSTMs). When we stack GCNs on top of LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark CoNLL-2009), both for English and Chinese.1

Footnote 1: The code is available at [https://github.com/diegma/neural-dep-srl](https://github.com/diegma/neural-dep-srl).

Interestingly, again unlike recursive neural networks, GCNs do not constrain the graph to be a tree. We believe that there are many applications in NLP, where GCN-based encoders of sentences or even documents can be used to incorporate knowledge about linguistic structures (e.g., representations of syntax, semantics or discourse). For example, GCNs can take as input combined syntactic-semantic graphs (e.g., the entire graph from Figure 1) and be used within downstream tasks such as machine translation or question answering. However, we leave this for future work and here solely focus on SRL.

The contributions of this paper can be summarized as follows:

* we are the first to show that GCNs are effective for NLP;
* we propose a generalization of GCNs suited to encoding syntactic information at word level;
* we propose a GCN-based SRL model and obtain state-of-the-art results on English and Chinese portions of the CoNLL-2009 dataset;
* we show that bidirectional LSTMs and syntax-based GCNs have complementary modeling power.

## 2 Graph Convolutional Networks

In this section we describe GCNs of Kipf and Welling (2017). Please refer to Gilmer et al. (2017) for a comprehensive overview of GCN versions.

GCNs are neural networks operating on graphs and inducing features of nodes (i.e., real-valued vectors / embeddings) based on properties of their neighborhoods. In Kipf and Welling (2017), they were shown to be very effective for the node classification task: the classifier was estimated jointly with a GCN, so that the induced node features were informative for the node classification problem. Depending on how many layers of convolution are used, GCNs can capture information only about immediate neighbors (with one layer of convolution) or any nodes at most \(K\) hops ways (if \(K\) layers are stacked on top of each other).

More formally, consider an undirected graph \({\cal G}=({\mathcal{V}},{\cal E})\), where \({\mathcal{V}}\) (\(|V|=n\)) and \({\cal E}\) are sets of nodes and edges, respectively. Kipf and Welling (2017) assume that edges contain all the self-loops, i.e., \((v,v)\in{\cal E}\) for any \(v\). We can define a matrix \(X\in\mathbb{R}^{m\times n}\) with each its column \(x_{v}\in\mathbb{R}^{m}\) (\(v\in{\mathcal{V}}\)) encoding node features. The vectors can either encode genuine features (e.g., this vector can encode the title of a paper if citation graphs are considered) or be a one-hot vector. The node representation, encoding information about its immediate neighbors, is computed as

\[h_{v}=ReLU\left(\sum_{u\in{\mathcal{N}}(v)}(Wx_{u}+b)\right), \tag{1}\]

where \(W\in\mathbb{R}^{m\times m}\) and \(b\in\mathbb{R}^{m}\) are a weight matrix and a bias, respectively; \({\mathcal{N}}(v)\) are neighbors of \(v\); \(ReLU\) is the rectifier linear unit activation function.2 Note that \(v\in{\mathcal{N}}(v)\) (because of self-loops), so the input feature representation of \(v\) (i.e. \(x_{v}\)) affects its induced representation \(h_{v}\).

 

As in standard convolutional networks (LeCun et al., 2001), by stacking GCN layers one can incorporate higher degree neighborhoods:

\[h_{v}^{(k+1)}=ReLU\left(\sum_{u\in\mathcal{N}(v)}W^{(k)}h_{u}^{(k)}+b^{(k)}\right)\]

where \(k\) denotes the layer number and \(h_{v}^{(1)}=x_{v}\).

## 3 Syntactic GCNs

As syntactic dependency trees are directed and labeled (we refer to the dependency labels as _syntactic functions_), we first need to modify the computation in order to incorporate label information (Section 3.1). In the subsequent section, we incorporate gates in GCNs, so that the model can decide which edges are more relevant to the task in question. Having gates is also important as we rely on automatically predicted syntactic representations, and the gates can detect and downweight potentially erroneous edges.

### Incorporating directions and labels

Now, we introduce a generalization of GCNs appropriate for syntactic dependency trees, and in general, for directed labeled graphs. First note that there is no reason to assume that information flows only along the syntactic dependency arcs (e.g., from _makes_ to _Sequa_), so we allow it to flow in the opposite direction as well (i.e., from dependents to heads). We use a graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\), where the edge set contains all pairs of nodes (i.e., words) adjacent in the dependency tree. In our example, both (_Sequa_, _makes_) and (_makes_, _Sequa_) belong to the edge set. The graph is labeled, and the label \(L(u,v)\) for \((u,v)\in\mathcal{E}\) contains both information about the syntactic function and indicates whether the edge is in the same or opposite direction as the syntactic dependency arc. For example, the label for \((\textit{makes},\textit{Sequa})\) is \(subj\), whereas the label for \((\textit{Sequa},\textit{ makes})\) is \(subj^{\prime}\), with the apostrophe indicating that the edge is in the direction opposite to the corresponding syntactic arc. Similarly, self-loops will have label \(self\). Consequently, we can simply assume that the GCN parameters are label-specific, resulting in the following computation, also illustrated in Figure 2:

\[h_{v}^{(k+1)}=ReLU\left(\sum_{u\in\mathcal{N}(v)}W_{L(u,v)}^{(k)}h_{u}^{(k)}+ b_{L(u,v)}^{(k)}\right).\]

This model is over-parameterized,3 especially given that SRL datasets are moderately sized, by deep learning standards. So instead of learning the GCN parameters directly, we define them as

Footnote 3: Chinese and English CoNLL-2009 datasets used 41 and 48 different syntactic functions, which would result in having 83 and 97 different matrices in every layer, respectively.

\[W_{L(u,v)}^{(k)}=V_{dir(u,v)}^{(k)}, \tag{2}\]

where \(dir(u,v)\) indicates whether the edge \((u,v)\) is directed (1) along, (2) in the opposite direction to the syntactic dependency arc, or (3) is a self-loop; \(V_{dir(u,v)}^{(k)}\in\mathbb{R}^{n\times m}\). Our simplification captures the intuition that information should be propagated differently along edges depending whether this is a head-to-dependent or dependent-to-head edge (i.e., along or opposite the corresponding syntactic arc) and whether it is a self-loop. So we do not share any parameters between these three very different edge types. Syntactic functions are important, but perhaps less crucial, so they are encoded only in the feature vectors \(b_{L(u,v)}\).

### Edge-wise gating

Uniformly accepting information from all neighboring nodes may not be appropriate for the SRL

Figure 2: A simplified syntactic GCN (bias terms and gates are omitted); the syntactic graph of the sentence is shown with dashed lines at the bottom. Parameter matrices are sub-indexed with syntactic functions, and apostrophes (e.g., _subj’_) signify that information flows in the direction opposite of the dependency arcs (i.e., from dependents to heads).

 

setting. For example, we see in Figure 1 that many semantic arcs just mirror their syntactic counterparts, so they may need to be up-weighted. Moreover, we rely on automatically predicted syntactic structures, and, even for English, syntactic parsers are far from being perfect, especially when used out-of-domain. It is risky for a downstream application to rely on a potentially wrong syntactic edge, so the corresponding message in the neural network may need to be down-weighted.

In order to address the above issues, inspired by recent literature van den Oord et al. (2016); Dauphin et al. (2016), we calculate for each edge node pair a scalar gate of the form

\[g_{u,v}^{(k)}=\sigma\left(h_{u}^{(k)}\cdot\hat{v}_{dir(u,v)}^{(k)}+\hat{b}_{L( u,v)}^{(k)}\right), \tag{3}\]

where \(\sigma\) is the logistic sigmoid function, \(\hat{v}_{dir(u,v)}^{(k)}\in\mathbb{R}^{m}\) and \(\hat{b}_{L(u,v)}^{(k)}\in\mathbb{R}\) are weights and a bias for the gate. With this additional gating mechanism, the final syntactic GCN computation is formulated as

\[h_{v}^{(k+1)} =ReLU( \tag{4}\] \[\sum_{u\in\mathcal{N}(v)}g_{v,i}^{(k)}(V_{dir(u,v)}^{(k)}h_{u}^{( k)}+b_{L(u,v)}^{(k)})).\]

### Complementarity of GCNs and LSTMs

The inability of GCNs to capture dependencies between nodes far away from each other in the graph may seem like a serious problem, especially in the context of SRL: paths between predicates and arguments often include many dependency arcs Roth and Lapata (2016). However, when graph convolution is performed on top of LSTM states (i.e., LSTM states serve as input \(x_{v}=h_{v}^{(1)}\) to GCN) rather than static word embeddings, GCN may not need to capture more than a couple of hops.

To elaborate on this, let us speculate what role GCNs would play when used in combinations with LSTMs, given that LSTMs have already been shown very effective for SRL Zhou and Xu (2015); Marcheggiani et al. (2017). Though LSTMs are capable of capturing at least some degree of syntax Linzen et al. (2016) without explicit syntactic supervision, SRL datasets are moderately sized, so LSTM models may still struggle with harder cases. Typically, harder cases for SRL involve arguments far away from their predicates. In fact, 20% and 30% of arguments are more than 5 tokens away from their predicate, in our English and Chinese collections, respectively. However, if we imagine that we can 'teleport' even over a single (longest) syntactic dependency edge, the 'distance' would shrink: only 9% and 13% arguments will now be more than 5 LSTM steps away (again for English and Chinese, respectively). GCNs provide this 'teleportation' capability. These observations suggest that LSTMs and GCNs may be complementary, and we will see that empirical results support this intuition.

## 4 Syntax-Aware Neural SRL Encoder

In this work, we build our semantic role labeler on top of the syntax-agnostic LSTM-based SRL model of Marcheggiani et al. (2017), which already achieves state-of-the-art results on the CoNLL-2009 English dataset. Following their approach we employ the same bidirectional (BiLSTM) encoder and enrich it with a syntactic GCN.

The CoNLL-2009 benchmark assumes that predicate positions are already marked in the test set (e.g., we would know that _makes_, _repairs_ and _engines_ in Figure 1 are predicates), so no predicate identification is needed. Also, as we focus here solely on identifying arguments and labeling them with semantic roles, for predicate disambiguation

Figure 3: Predicting an argument and its label with an LSTM + GCN encoder.

 

(i.e., marking _makes_ as _make.01_) we use of an off-the-shelf disambiguation model Roth and Lapata (2016); Bjorkelund et al. (2009). As in Marcheggiani et al. (2017) and in most previous work, we process individual predicates in isolation, so for each predicate, our task reduces to a sequence labeling problem. That is, given a predicate (e.g., _disputed_ in Figure 3) one needs to identify and label all its arguments (e.g., label _estimates_ as A1 and label _those_ as 'NULL', indicating that _those_ is not an argument of _disputed_).

The semantic role labeler we propose is composed of four components (see Figure 3):

* look-ups of word embeddings;
* a BiLSTM encoder that takes as input the word representation of each word in a sentence;
* a syntax-based GCN encoder that re-encodes the BiLSTM representation based on the automatically predicted syntactic structure of the sentence;
* a role classifier that takes as input the GCN representation of the candidate argument and the representation of the predicate to predict the role associated with the candidate word.

### Word representations

For each word \(w_{i}\) in the considered sentence, we create a sentence-specific word representation \(x_{i}\). We represent each word \(w\) as the concatenation of four vectors:4 a randomly initialized word embedding \(x^{re}\in\mathbb{R}^{d_{w}}\), a pre-trained word embedding \(x^{pe}\in\mathbb{R}^{d_{w}}\) estimated on an external text collection, a randomly initialized part-of-speech tag embedding \(x^{pos}\in\mathbb{R}^{d_{p}}\) and a randomly initialized lemma embedding \(x^{le}\in\mathbb{R}^{d_{l}}\) (active only if the word is a predicate). The randomly initialized embeddings \(x^{re}\), \(x^{pos}\), and \(x^{le}\) are fine-tuned during training, while the pre-trained ones are kept fixed. The final word representation is given by \(x=x^{re}\circ x^{pe}\circ x^{pos}\circ x^{le}\), where \(\circ\) represents the concatenation operator. Footnote 4: We drop the index \(i\) from the notation for the sake of brevity.

### Bidirectional LSTM layer

One of the most popular and effective ways to represent sequences, such as sentences Mikolov et al. (2010), is to use recurrent neural networks RNN Elman (1990). In particular their gated versions, Long Short-Term Memory (LSTM) networks Hochreiter and Schmidhuber (1997) and Gated Recurrent Units (GRU) Cho et al. (2014), have proven effective in modeling long sequences Chiu and Nichols (2016); Sutskever et al. (2014).

Formally, an LSTM can be defined as a function \(LSTM_{\theta}(x_{1:i})\) that takes as input the sequence \(x_{1:i}\) and returns a hidden state \(h_{i}\in\mathbb{R}^{d_{h}}\). This state can be regarded as a representation of the sentence from the start to the position \(i\), or, in other words, it encodes the word at position \(i\) along with its left context. However, the right context is also important, so Bidirectional LSTMs Graves (2008) use two LSTMs: one for the forward pass, and another for the backward pass, \(LSTM_{F}\) and \(LSTM_{B}\), respectively. By concatenating the states of both LSTMs, we create a complete context-aware representation of a word \(BiLSTM(x_{1:n},i)=LSTM_{F}(x_{1:i})\circ LSTM_{B}(x_{n:i})\). We follow Marcheggiani et al. (2017) and stack \(J\) layers of bidirectional LSTMs, where each layer takes the lower layer as its input.

### Graph convolutional layer

The representation calculated with the BiLSTM encoder is fed as input to a GCN of the form defined in Equation (4). The neighboring nodes of a node \(v\), namely \(\mathcal{N}(v)\), and their relations to \(v\) are predicted by an external syntactic parser.

### Semantic role classifier

The classifier predicts semantic roles of words given the predicate while relying on word representations provided by GCN; we concatenate hidden states of the candidate argument word and the predicate word and use them as input to a classifier (Figure 3, top). The softmax classifier computes the probability of the role (including special 'NULL' role):

\[p(r|t_{i},t_{p},l)\propto\exp(W_{l,r}(t_{i}\circ t_{p})), \tag{5}\]

where \(t_{i}\) and \(t_{p}\) are representations produced by the graph convolutional encoder, \(l\) is the lemma of predicate \(p\), and the symbol \(\propto\) signifies proportionality.5As FitzGerald et al. (2015) and Marcheggiani et al. (2017), instead of using a fixed matrix \(W_{l,r}\) or simply assuming that \(W_{l,r}=W_{r}\) 

we jointly embed the role \(r\) and predicate lemma \(l\) using a non-linear transformation:

\[W_{l,r}=ReLU(U(q_{l}\circ q_{r})), \tag{6}\]

where \(U\) is a parameter matrix, whereas \(q_{l}\in\mathbb{R}^{d_{t}}\) and \(q_{r}\in\mathbb{R}^{d_{r}}\) are randomly initialized embeddings of predicate lemmas and roles. In this way each role prediction is predicate-specific, and, at the same time, we expect to learn a good representation for roles associated with infrequent predicates. As our training objective we use the categorical cross-entropy.

## 5 Experiments

### Datasets and parameters

We tested the proposed SRL model on the English and Chinese CoNLL-2009 dataset with standard splits into training, test and development sets. The predicted POS tags for both languages were provided by the CoNLL-2009 shared-task organizers. For the predicate disambiguator we used the ones from Roth and Lapata (2016) for English and from Bjorkelund et al. (2009) for Chinese. We parsed English sentences with the BIST Parser Kiperwasser and Goldberg (2016), whereas for Chinese we used automatically predicted parses provided by the CoNLL-2009 shared-task organizers.

For English, we used external embeddings of Dyer et al. (2015), learned using the structured skip n-gram approach of Ling et al. (2015). For Chinese we used external embeddings produced with the neural language model of Bengio et al. (2003). We used _edge dropout_ in GCN: when computing \(h_{v}^{(k)}\), we ignore each node \(v\in\mathcal{N}(v)\) with probability \(\beta\). Adam Kingma and Ba (2015) was used as an optimizer. The hyperparameter tuning and all model selection were performed on the English development set; the chosen values are shown in Appendix.

### Results and discussion

In order to show that GCN layers are effective, we first compare our model against its version which lacks GCN layers (i.e. essentially the model of Marcheggiani et al. (2017)). Importantly, to measure the genuine contribution of GCNs, we first tuned this syntax-agnostic model (e.g., the number of LSTM layers) to get best possible performance on the development set.6

Footnote 6: For example, if we would have used only one layer of LSTMs, gains from using GCNs would be even larger.

We compare the syntax-agnostic model with 3 syntax-aware versions: one GCN layer over syntax (\(K=1\)), one layer GCN without gates and two GCN layers (\(K=2\)). As we rely on the same

Figure 4: F\({}_{1}\) as function of word distance. The distance starts from zero, since nominal predicates can be arguments of themselves.

\begin{table}
\begin{tabular}{p{85.4pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline System (English) & P & R & F\({}_{1}\) \\ \hline LSTMs & 84.3 & 81.1 & 82.7 \\ LSTMs + GCNs (K=1) & 85.2 & 81.6 & 83.3 \\ LSTMs + GCNs (K=2) & 84.1 & 81.4 & 82.7 \\ LSTMs + GCNs (K=1), no gates & 84.7 & 81.4 & 83.0 \\ \hline GCNs (no LSTMs), K=1 & 79.9 & 70.4 & 74.9 \\ GCNs (no LSTMs), K=2 & 83.4 & 74.6 & 78.7 \\ GCNs (no LSTMs), K=3 & 83.6 & 75.8 & 79.5 \\ GCNs (no LSTMs), K=4 & 82.7 & 76.0 & 79.2 \\ \hline \hline \end{tabular}
\end{table}
Table 1: SRL results without predicate disambiguation on the English development set.

Figure 5: Performance with dependency arcs of given type dropped, on Chinese development set.

 

off-the-shelf disambiguator for all versions of the model, in Table 1 and 2 we report SRL-only scores (i.e., predicate disambiguation is not evaluated) on the English and Chinese development sets. For both datasets, the syntax-aware model with one GCN layers (\(K=1\)) performs the best, outperforming the LSTM version by 1.9% and 0.6% for Chinese and English, respectively. The reasons why the improvements on Chinese are much larger are not entirely clear (e.g., both languages are relative fixed word order ones, and the syntactic parses for Chinese are considerably less accurate), this may be attributed to a higher proportion of long-distance dependencies between predicates and arguments in Chinese (see Section 3.3). Edge-wise gating (Section 3.2) also appears important: removing gates leads to a drop of 0.3% F\({}_{1}\) for English and 0.6% F\({}_{1}\) for Chinese.

Stacking two GCN layers does not give any benefit. When BiLSTM layers are dropped altogether, stacking two layers (\(K=2\)) of GCNs greatly improves the performance, resulting in a 3.8% jump in F\({}_{1}\) for English and a 3.0% jump in F\({}_{1}\) for Chinese. Adding a 3rd layer of GCN (\(K=3\)) further improves the performance.7 This suggests that extra GCN layers are effective but largely redundant with respect to what LSTMs already capture.

Footnote 7: Note that GCN layers are computationally cheaper than LSTM ones, even in our non-optimized implementation.

In Figure 4, we show the \(F_{1}\) scores results on the English development set as a function of the distance, in terms of tokens, between a candidate argument and its predicate. As expected, GCNs appear to be more beneficial for long distance dependencies, as shorter ones are already accurately captured by the LSTM encoder.

We looked closer in contribution of specific dependency relations for Chinese. In order to assess this without retraining the model multiple times, we drop all dependencies of a given type at test time (one type at a time, only for types appearing over 300 times in the development set) and observe changes in performance. In Figure 5, we see that the most informative dependency is COMP (complement). Relative clauses in Chinese are very frequent and typically marked with particle (de) (de). The relative clause will syntactically depend on as COMP, so COMP encodes important information about predicate-argument structure. These are often long-distance dependencies and may not be accurately captured by LSTMs. Although TMP (temporal) dependencies are not as frequent (\(\sim\)2% of all dependencies), they are also important: temporal information is mirrored in semantic roles.

In order to compare to previous work, in Table 3 we report test results on the English in-domain (WSJ) evaluation data. Our model is _local_, as all the argument detection and labeling decisions are conditionally independent: their interaction is captured solely by the LSTM+GCN encoder. This makes our model fast and simple, though, as shown in previous work, _global_ modeling of the structured output is beneficial.8 We leave this extension fo

\begin{table}
\begin{tabular}{l c c c} \hline System & P & R & F\({}_{1}\) \\ \hline Zhao et al. (2009) (global) & 80.4 & 75.2 & 77.7 \\ Björkelund et al. (2009) (global) & 82.4 & 75.1 & 78.6 \\ Roth and Lapata (2016) (global) & 83.2 & 75.9 & 79.4 \\
**Ours (local)** & **84.6** & **80.4** & **82.5** \\ \hline \end{tabular}
\end{table}
Table 4: Results on the Chinese test set.

\begin{table}
\begin{tabular}{l c c c} \hline System & P & R & F\({}_{1}\) \\ \hline Lei et al. (2015) (local) & - & - & 86.6 \\ FitzGerald et al. (2015) (local) & - & - & 86.7 \\ Roth and Lapata (2016) (local) & 88.1 & 85.3 & 86.7 \\ Marcheggiani et al. (2017) (local) & 88.7 & 86.8 & 87.7 \\

- & 87.3 \\ Foland and Martin (2015) (global) &
- & 86.0 \\ Swayamdipta et al. (2016) (global) &
- & 85.0 \\ Roth and Lapata (2016) (global) & 90.0 & 85.5 & 87.7 \\ \hline FitzGerald et al. (2015) (ensemble) &
- & 87.7 \\ Roth and Lapata (2016) (ensemble) & 90.3 & 85.7 & 87.9 \\
**Ours (ensemble 3x)** & **90.5** & **87.7** & **89.1** \\ \hline \end{tabular}
\end{table}
Table 3: Results on the test set for English.

 

we outperform even the best global model and the best ensemble of global models, without using global modeling or ensembles. When we create an ensemble of 3 models with the product-of-expert combination rule, we improve by 1.2% over the best previous result, achieving 89.1% F\({}_{1}\).9

Footnote 9: To compare to previous work, we report combined scores which also include predicate disambiguation. As we use disambiguators from previous work (see Section 5.1), actual gains in argument identification and labeling are even larger.

For Chinese (Table 4), our best model outperforms the state-of-the-art model of Roth and Lapata (2016) by even larger margin of 3.1%.

For English, in the CoNLL shared task, systems are also evaluated on the out-of-domain dataset. Statistical models are typically less accurate when they are applied to out-of-domain data. Consequently, the predicted syntax for the out-of-domain test set is of lower quality, which negatively affects the quality of GCN embeddings. However, our model works surprisingly well on out-of-domain data (Table 5), substantially outperforming all the previous syntax-aware models. This suggests that our model is fairly robust to mistakes in syntax. As expected though, our model does not outperform the syntax-agnostic model of Marcheggiani et al. (2017).

## 6 Related Work

Perhaps the earliest methods modeling syntax-semantics interface with RNNs are due to Henderson et al. (2008); Titov et al. (2009); Gesmundo et al. (2009), they used shift-reduce parsers for joint SRL and syntactic parsing, and relied on RNNs to model statistical dependencies across syntactic and semantic parsing actions. A more modern (e.g., based on LSTMs) and effective reincarnation of this line of research has been proposed in Swayamdipta et al. (2016). Other recent work which considered incorporation of syntactic information in neural SRL models include: FitzGerald et al. (2015) who use standard syntactic features within an MLP calculating potentials of a CRF model; Roth and Lapata (2016) who enriched standard features for SRL with LSTM representations of syntactic paths between arguments and predicates; Lei et al. (2015) who relied on low-rank tensor factorizations for modeling syntax. Also Foland and Martin (2015) used (nongraph) convolutional networks and provided syntactic features as input. A very different line of research, but with similar goals to ours (i.e. integrating syntax with minimal feature engineering), used tree kernels Moschitti et al. (2008).

Beyond SRL, there have been many proposals on how to incorporate syntactic information in RNN models, for example, in the context of neural machine translation Eriguchi et al. (2017); Sennrich and Haddow (2016). One of the most popular and attractive approaches is to use tree-structured recursive neural networks Socher et al. (2013); Le and Zuidema (2014); Dyer et al. (2015), including stacking them on top of a sequential BiLSTM Miwa and Bansal (2016). An approach of Mou et al. (2015) to sentiment analysis and question classification, introduced even before GCNs became popular in the machine learning community, is related to graph convolution. However, it is inherently single-layer and tree-specific, uses bottom-up computations, does not share parameters across syntactic functions and does not use gates. Gates have been previously used in GCNs Li et al. (2016) but between GCN layers rather than for individual edges.

Previous approaches to integrating syntactic information in neural models are mainly designed to induce representations of sentences or syntactic constituents. In contrast, the approach we presented incorporates syntactic information at word level. This may be attractive from the engineering perspective, as it can be used, as we have shown, instead or along with RNN models.

## 7 Conclusions and Future Work

We demonstrated how GCNs can be used to incorporate syntactic information in neural models and specifically to construct a syntax-aware SRL

\begin{table}
\begin{tabular}{l c c c} \hline \hline System & P & R & F\({}_{1}\) \\ \hline Lei et al. (2015) (local) & - & - & 75.6 \\ FitzGerald et al. (2015) (local) & - & - & 75.2 \\ Roth and Lapata (2016) (local) & 76.9 & 73.8 & 75.3 \\ Marcheggiani et al. (2017) (local) & 79.4 & 76.2 & 77.7 \\

- & 75.2 \\ Foland and Martin (2015) (global) &
- & 75.9 \\ Roth and Lapata (2016) (global) & 78.6 & 73.8 & 76.1 \\ \hline FitzGerald et al. (2015) (ensemble) &
- & 75.5 \\ Roth and Lapata (2016) (ensemble) & 79.7 & 73.6 & 76.5 \\
**Ours (ensemble 3x)** & **80.8** & **77.1** & **78.9** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on the out-of-domain test set.

 

model, resulting in state-of-the-art results for Chinese and English. There are relatively straightforward steps which can further improve the SRL results. For example, we relied on labeling arguments independently, whereas using a joint model is likely to significantly improve the performance. Also, in this paper we consider the dependency version of the SRL task, however the model can be generalized to the span-based version of the task (i.e. labeling argument spans with roles rather that syntactic heads of arguments) in a relatively straightforward fashion.

More generally, given simplicity of GCNs and their applicability to general graph structures (not necessarily trees), we believe that there are many NLP tasks where GCNs can be used to incorporate linguistic structures (e.g., syntactic and semantic representations of sentences and discourse parses or co-reference graphs for documents).

## Acknowledgements

We would thank Anton Frolov, Michael Schlichtkrull, Thomas Kipf, Michael Roth, Max Welling, Yi Zhang, and Wilker Aziz for their suggestions and comments. The project was supported by the European Research Council (ERC StG BroadSem 678254), the Dutch National Science Foundation (NWO VIDI 639.022.518) and an Amazon Web Services (AWS) grant.

## References

* Bengio et al. (2003) Yoshua Bengio, Rejean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. _Journal of Machine Learning Research_, 3:1137-1155.
* Bjorkelund et al. (2010) Anders Bjorkelund, Bernd Bohnet, Love Haffdell, and Pierre Nugues. 2010. A high-performance syntactic and semantic dependency parser. In _Proceedings of COLING: Demonstrations_.
* Bjorkelund et al. (2009) Anders Bjorkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In _Proceedings of CoNLL_.
* Chiu and Nichols (2016) Jason P. C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. _TACL_, 4:357-370.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of EMNLP_.
* Dauphin et al. (2016) Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2016. Language modeling with gated convolutional networks. _arXiv preprint arXiv:1612.08083_.
* Duvenaud et al. (2015) David K. Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P. Adams. 2015. Convolutional networks on graphs for learning molecular fingerprints. In _Proceedings of NIPS_.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of ACL_.
* Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. _Cognitive Science_, 14(2):179-211.
* Eriguchi et al. (2017) Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. _arXiv preprint arXiv:1702.03525_.
* FitzGerald et al. (2015) Nicholas FitzGerald, Oscar Tackstrom, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In _Proceedings of EMNLP_.
* Foland and Martin (2015) William Foland and James Martin. 2015. Dependency-based semantic role labeling using convolutional neural networks. In _Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics (*SEM)_.
* Gesmundo et al. (2009) Andrea Gesmundo, James Henderson, Paola Merlo, and Ivan Titov. 2009. Latent variable model of synchronous syntactic-semantic parsing for multiple languages. In _Proceedings of CoNLL_.
* Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. _Computational linguistics_, 28(3):245-288.
* Gilmer et al. (2017) Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. _arXiv preprint arXiv:1704.01212_.
* Graves (2008) Alex Graves. 2008. _Supervised sequence labelling with recurrent neural networks_. Ph.D. thesis, Munchen, Techn. Univ., Diss., 2008.
* Henderson et al. (2008) James Henderson, Paola Merlo, Gabriele Musillo, and Ivan Titov. 2008. A latent variable model of synchronous parsing for syntactic and semantic dependencies. In _Proceedings of CoNLL_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Johansson and Nqueses (2008) Richard Johansson and Pierre Nqueses. 2008. The effect of syntactic representation on semantic role labeling. In _Proceedings of COLING_.

 

Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, and Patrick Riley. 2016. Molecular graph convolutions: moving beyond fingerprints. _Journal of computer-aided molecular design_, 30(8):595-608.
* Kingma and Ba [Ba2015] Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of ICLR_.
* Kiperwasser and Goldberg [Simple and Goldberg2016] Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _TACL_, 4:313-327.
* Kipf and Welling [Kipf and Welling2017] Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In _Proceedings of ICLR_.
* Le and Zuidema [Zuidema2014] Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In _Proceedings of EMNLP_.
* LeCun et al. [2001] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. 2001. Gradient-based learning applied to document recognition. In _Proceedings of Intelligent Signal Processing_.
* Lei et al. [2015] Tao Lei, Yuan Zhang, Lluis Marquez, Alessandro Moschitti, and Regina Barzilay. 2015. High-order low-rank tensors for semantic role labeling. In _Proceedings of NAACL_.
* Levin [1993] Beth Levin. 1993. _English verb classes and alternations: A preliminary investigation_. University of Chicago press.
* Li et al. [2016] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel. 2016. Gated graph sequence neural networks. In _Proceedings of ICLR_.
* Ling et al. [2015] Wang Ling, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Two/too simple adaptations of word2vec for syntax problems. In _Proceedings of NAACL_.
* Linzen et al. [2016] Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _TACL_, 4:521-535.
* Marcheggiani et al. [2017] Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In _Proceedings of CoNLL_.
* Mikolov et al. [2010] Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In _Proceedings of INTERSPEECH_.
* Miwa and Bansal [2016] Makoto Miwa and Mohit Bansal. 2016. End-to-end relation extraction using lstms on sequences and tree structures. In _Proceedings of ACL_.
* Moschitti et al. [2008] Alessandro Moschitti, Daniele Pighin, and Roberto Basili. 2008. Tree kernels for semantic role labelling. _Computational Linguistics_, 34(2):193-224.
* Mou et al. [2015] Lili Mou, Hao Peng, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. 2015. Discriminative neural sentence modeling by tree-based convolution. In _Proceedings of EMNLP_.
* Oord et al. [2016] Aaron van den Oord, Nak Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. 2016. Conditional image generation with PixelCNN decoders. In _Proceedings of NIPS_.
* Pradhan et al. [2005] Sameer Pradhan, Kadri Hacioglu, Wayne H. Ward, James H. Martin, and Daniel Jurafsky. 2005. Semantic role chunking combining complementary syntactic views. In _Proceedings of CoNLL_.
* Punyakanok et al. [2008] Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008. The importance of syntactic parsing and inference in semantic role labeling. _Computational Linguistics_, 34(2):257-287.
* Roth and Lapata [2016] Michael Roth and Mirella Lapata. 2016. Neural semantic role labeling with dependency path embeddings. In _Proceedings of ACL_.
* Sennrich and Hadow [2016] Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In _Proceedings of WMT_.
* Socher et al. [2013] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of EMNLP_.
* Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of NIPS_.
* Swayamdipta et al. [2016] Swabha Swayamdipta, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Greedy, joint syntactic-semantic parsing with stack LSTMs. In _Proceedings of CoNLL_.
* Thompson et al. [2003] Cynthia A. Thompson, Roger Levy, and Christopher D. Manning. 2003. A generative model for semantic role labeling. In _Proceedings of ECML_.
* Titov et al. [2009] Ivan Titov, James Henderson, Paola Merlo, and Gabriele Musillo. 2009. Online projectivisation for synchronous parsing of semantic and syntactic dependencies. In _Proceedings of IJCAI_.
* Zhao et al. [2009] Hai Zhao, Wenliang Chen, Jun'ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Multilingual dependency learning: Exploiting rich features for tagging syntactic and semantic dependencies. In _Proceedings of CoNLL_.
* Zhou and Xu [2015] Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In _Proceedings of ACL_