Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling
Year: 2023

Abstract:
Recent models have shown that incorporating syntactic knowledge into the semantic role labelling (SRL) task leads to a significant improvement. In this paper, we propose Syntaxaware Graph-to-Graph Transformer (SynG2GTr) model, which encodes the syntactic structure using a novel way to input graph relations as embeddings, directly into the self-attention mechanism of Transformer. This approach adds a soft bias towards attention patterns that follow the syntactic structure but also allows the model to use this information to learn alternative patterns. We evaluate our model on both spanbased and dependency-based SRL datasets, and outperform previous alternative methods in both in-domain and out-of-domain settings, on CoNLL 2005 and CoNLL 2009 datasets.1

heading: 1 Introduction
text: The task of semantic role labelling (SRL) provides a shallow representation of the semantics in a sentence, and constructs event properties and relations among relevant words. Traditionally, a syntactic structure was considered a prerequisite for SRL models (Punyakanok et al., 2008; Gildea and Palmer, 2002), but, newer models that leverage deep neural network architectures (Cai et al., 2018; Tan et al., 2017; He et al., 2017; Marcheggiani et al., 2017) have outperformed syntax-aware architectures, without the need for explicit encoding of syntactic structure. However, recent studies (Zhou et al., 2020a; Strubell et al., 2018; He et al., 2017; Marcheggiani and Titov, 2017) have proposed that deep neural network models could benefit from using syntactic information, rather than disregarding it. These studies suggest that incorporating syntax into the model can improve SRL prediction by jointly learning both syntactic and semantic structures (Zhou et al., 2020a), training a self-attention
1The implementation is publicly available at https://github.com/alirezamshi/SynG2GTr-SRL.
head in Transformer (Vaswani et al., 2017a) to attend to each token’s syntactic parent (Strubell et al., 2018), or encoding the syntactic structure using graph convolutional networks (Fei et al., 2021; Marcheggiani and Titov, 2017). In this paper, we propose a novel method for encoding syntactic knowledge by introducing Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr) architecture. The model conditions on the sentence’s dependency structure and jointly predicts both span-based and dependency-based SRL structures. Inspired by Mohammadshahi and Henderson (2021, 2020), our model inputs graph relations as embeddings incorporated into the self-attention mechanism of Transformer (Vaswani et al., 2017b). Different from the original Graph-to-Graph Transformer, our self-attention function models the interaction of the graph relations with both the query and key vectors of self-attention mechanism, instead of just the query. We also find that excluding the interaction of graph structure with the value vectors of self-attention does not harm the performance. Furthermore, compared to the previous work on Graph-to-Graph Transformers (Mohammadshahi and Henderson, 2021, 2020), our architecture uses different types of graphs as the input and output. We show empirically that our model outperforms previous comparable models. In the in-domain setting, SynG2G-Tr model achieves 88.93 (87.57) F1 score on the CoNLL 2005 dataset, given the predicate (end-to-end), and 91.23 (88.05) F1 on the CoNLL 2009 dataset, given the predicate (endto-end). In the out-of-domain setting, our model reaches 83.21 (80.53) F1 score on the CoNLL 2005 dataset, given predicate (end-to-end), and 86.43 (81.93) F1 scores on the CoNLL 2009 dataset, given predicate (end-to-end). Our contributions are:
• We propose SynG2G-Tr model for encoding the dependency parsing graph in the SRL task.
174
• We evaluate our model on CoNLL 2005 and CoNLL 2009 datasets and outperform previous comparable models in most cases of both in-domain and out-of-domain sets.

heading: 2 Syntax-aware Graph-to-Graph Transformer
text: The architecture of the SynG2G-Tr model is illustrated in Figure 1. The input to the model is the tokenised text (W = (w1,w2,...,wN )), which are the nodes of the input and output graphs, and N is the length of tokenised input. The outputs are the dependency-based (Gdep) and span-based (Gspan) SRL graphs. The SynG2G-Tr model can be formalised in terms of an encoder Esg2g and decoder Dsg2g:
{ Z=Esg2g(W,P,Gsyn)
Gspan,Gdep=D sg2g(Z)
(1)
Initially, a syntactic parser predicts the dependency graph (Gsyn), and Part-of-Speech (PoS) tags (P = (p1, p2, ..., pN )). Then the encoder of SynG2G-Tr (Esg2g) encodes both sequences (W, P ) and the dependency graph (Gsyn) into contextualised representations of graph nodes (Z). This representation (Z) is then used by the decoder (Dsg2g) to jointly predict SRL graphs. For the decoder, we follow the same unified scorer and decoder as defined in Zhou et al. (2020a). Further explanation of SRL scorer and decoding mechanism is provided in Appendix A.
The encoder employs an enhanced way of inputting graph relations into the self-attention mechanism of Transformer (Vaswani et al., 2017b). Unlike the previously proposed version of Graph-toGraph Transformer (Mohammadshahi and Henderson, 2021), we modify the self-attention mechanism to have a more comprehensive interaction between graph relations, queries and keys. We also find that excluding the interaction of graph relations with value vectors retains good performance.
Specifically, given the output of an intermediate embedding layer X = (x1,...,xN ), we define the attention mechanism of each head in each layer to take the dependency graph as input. These attention scores (αij) are calculated as a Softmax function over eij values:
(2)
eij= 1√ d
[ xiW Q(xjW K)T+xiW Q(rijW R)T
+rijW R(xjW K)T ]
where WQ,WK ∈Rdx×d are learned query and key matrices. rij∈R is a one-hot vector specifying both the label and direction of the dependency relation between token i and token j. R is the matrix of graph relations, derived from the syntactic graph (Gsyn). Figure 2 illustrates a sample computation of R matrix, where rij = idlabel if i→ j, idlabel + |Lsyn| if j ← i , or NONE (|Lsyn| is the size of syntactic label set). WR∈R(2|Lsyn|+1)×d is a matrix of learned relation embeddings. d is the attention head size, and dx is the hidden size. The second and third terms in Equation 2 incorporate the graph information into the self-attention mechanism of Transformer with a soft bias, while the model can still learn other structures, using this encoded graph information. For better efficiency, we share the relation embeddings across multiple attention heads in each layer. Additionally, the computation complexity of both the second and third terms is O(N), as we ignore the NONE graph relation, and the syntactic dependency graph is a tree. The output of the attention function is the
value embedding (vi), which is calculated as: vi= ∑
j
αij(xjW V ) (3)
which, in our model, does not use the graph, and W V ∈Rdx×d is the learned value matrix. Syntactic Parser. The parser jointly predicts PoS tags and the dependency graph. We apply the parser defined in Zhou et al. (2020a), which uses a joint scorer and decoder for dependency and constituency graphs based on Head-driven Phrase Structure Grammar (Zhou and Zhao, 2019). This method has achieved state-of-the-art results in the dependency parsing task.

heading: 3 Related Work
text: Several approaches have been proposed to use syntax for the SRL task. Roth and Lapata (2016) embed dependency paths, while some researchers (Fei et al., 2021; Munir et al., 2021; Marcheggiani and Titov, 2017) use graph convolutional networks to encode the syntactic structure. Strubell et al. (2018) incorporates a dependency graph by training one attention head of Transformer to attend to syntactic parents for each token, in a multi-task setting. He et al. (2019, 2018b) use syntactic information to guide the argument pruning. Xia et al. (2019) exploit different alternatives e.g. tree-structured GRU and graph features of dependency tree to encode syntactic knowledge. Kasai et al. (2019) apply BiLSTM to tag the text with supertags extracted from dependency parses and feed them into SRL models. Xia et al. (2020) showed that incorporating heterogeneous syntactic knowledge results in significant improvement. Some other work focus on joint learning of both SRL and syntax (Zhou et al., 2020a,b; Cai and Lapata, 2019a,b). Additionally, some approaches discarded the syntax, but achieve impressive results (Shi and Lin, 2019; Peters et al., 2018; He et al., 2018a; Marcheggiani et al., 2017; He et al., 2017; Tan et al., 2017; Zhou and Xu, 2015). Our work is different from previous work since we encode the syntactic graph by directly inputting it as embeddings into the attention mechanism of Transformer, which provides a soft bias. Moreover, both sequences and syntactic graph can be encoded in one general model.

heading: 4 Results and Discussion
text: Experimental Setup. Our models are evaluated on CoNLL 2005 (Carreras and Màrquez, 2005)
Model SA WSJ (in-domain) Brown (out-of-domain) P R F1 P R F1 end-to-end He et al. (2017) ✗ 85.0 84.3 84.6 74.9 72.4 73.6 He et al. (2018a) ✗ 81.2 83.9 82.5 69.7 71.9 70.8 Strubell et al. (2018) ✓ 85.53 84.45 84.99 75.8 73.54 74.66 Li et al. (2019) ✗ - - 83.0 - - - Xia et al. (2019) ✓ 84.3 83.8 84.1 73.7 72.0 72.9 Xia et al. (2020) ✓ 83.05 84.49 84.49 73.47 74.92 74.19 SynG2G-Tr (w/o BERT) ✓ 84.48 86.46 85.45 73.92 76.65 75.26 +pre-training He et al. (2018a) ✗ 84.8 87.2 86.0 73.9 78.4 76.1 Strubell et al. (2018)† ✓ 87.13 86.67 86.9 79.02 77.49 78.25 Li et al. (2019) ✗ 85.2 87.5 86.3 74.7 78.1 76.4 SynG2G-Tr ✓ 86.86 88.3 87.57 80.01 81.07 80.53 given predicate Tan et al. (2017) ✗ 84.5 85.2 84.8 73.5 74.6 74.1 He et al. (2018a) ✗ - - 83.9 - - 73.7 Strubell et al. (2018)† ✓ 86.02 86.05 86.04 76.65 76.44 76.54 Ouchi et al. (2018) ✗ 84.7 82.3 83.5 76.0 70.4 73.1 Xia et al. (2020) ✓ 85.12 85.0 85.06 76.3 75.42 75.86 SynG2G-Tr (w/o BERT) ✓ 86.46 86.56 86.50 77.73 77.18 77.45 +pre-training He et al. (2018a) ✗ - - 87.4 - - 80.4 Ouchi et al. (2018) ✗ 88.2 87.0 87.6 79.9 77.5 78.7 Li et al. (2019) ✗ 87.9 87.5 87.7 80.6 80.4 80.5 Jindal et al. (2020) ✗ 87.70 88.15 87.93 81.52 81.36 81.44 Zhang et al. (2021) ✗ 88.70 88.00 87.90 80.30 80.10 80.20 Jia et al. (2022) ✗ - - 88.25 - - 81.90 SynG2G-Tr ✓ 89.11 88.74 88.93 83.93 82.50 83.21
ROOT Mr John plays soccer . ROOT
Mr
John
plays
soccer
.
idroot
idcomp
idnsubj idobj idpunct
0
0
0
0
0
0
idnsubj+|Lsyn|
idcomp+|Lsyn|
idobj+|Lsyn|
idpunct+|Lsyn|
0 0 0 0
0 0 0 0
0 0 0
0 0
0 0 0 0
0 0 0 0
SynG2G-Tr
M r J
oh n
pl ay
s so
cc er
.
Syntactic Parser ROOT Mr John plays soccer
nsubj obj root comp
.
punct
Figure 2: A sample computation of R matrix for the sentence "Mr John plays soccer.".
initialisation, our SynG2G-Tr model outperforms Strubell et al. (2018) (the second best model) in both end-to-end and given-predicate settings. This highlights the benefit of injecting the graph information into the self-attention mechanism using a soft bias, instead of hard-coding one attention head to attend to the syntactic parent of each token, as used in Strubell et al. (2018). The main reason for this improvement is that the model can still learn other attention patterns in combination with the graph information, which will be described later in this section. When adding BERT initialisation, our SynG2G-Tr model outperforms best previous work by 5.4%/8.8% F1 relative error reduction (RER) on average in both in-domain and out-of-domain evaluation sets, which demonstrates the compatibility of the modified self-attention mechanism of SynG2GTr with BERT (Devlin et al., 2019) initialisation.
CoNLL 2009 Results.6 Table 3 illustrates the results of dependency-based SRL on the test set of CoNLL 2009 dataset. Without BERT initialisation, SynG2G-Tr significantly outperforms previous work in in-domain and out-of-domain settings. With BERT initialisation, our model significantly outperforms previous work in end-to-end setting with 3.2%/10.4% F1 RER in both in-domain and out-of-domain evaluation sets, while having competitive performance in given-predicate setting. For a better comparison with Fei et al. (2021) (last setting of Table 3), we also employ the gold dependency tree for training and use the predicted dependency graph at inference time. Our model
graph additional to the dependency tree. Also, to better understand the effect of syntactic information, we exclude Fernández-González (2023); Zhou et al. (2022); Conia and Navigli (2020), as they exploited different scorer and training mechanism for SRL graphs. However, the best setting of SynG2G-Tr model still shows competitive or better results when compared to aforementioned excluded works.
6Scores are calculated with CoNLL 2009 shared task script (https://ufal.mff.cuni.cz/conll2009-st/).
significantly outperforms Fei et al. (2021), especially on the out-of-domain dataset. This shows the benefit of encoding the dependency graph by modifying the self-attention mechanism of Transformer (Vaswani et al., 2017b) compared to using graph convolutional network, as in Fei et al. (2021).
Further Analysis. We also analyse the selfattention matrix of SynG2G-Tr model for different heads and layers. Figure 3 in Appendix D demonstrates that the self-attention mechanism of SynG2G-Tr ignores the dependency graph information in the first few layers, and only uses the context-dependent information. However, as it progresses to upper layers, it begins to utilise the graph relation information, as shown in the atten-
tion matrix. This highlights the benefit of encoding the dependency graph with a soft bias as the model can still learn different structures in different layers, given this encoded graph information. Furthermore, in Appendix E, we show that removing the interaction of graph embeddings with key vectors results in a performance drop. Additionally, ignoring the interaction of graph relations with both key and query vectors 7 results in a significant drop as well. However, integrating the graph information into Equation 3 as stated in Mohammadshahi and Henderson (2021) does not improve the performance, and we remove it for better efficiency.

heading: 5 Conclusion
text: In this paper, we propose the Syntax-aware Graph-to-Graph Transformer architecture, which effectively incorporates syntactic information by inputting the syntactic dependency graph into the self-attention mechanism of Transformer. Our mechanism for inputting graph relation embeddings differs from the original Graph-toGraph Transformer in that it models the complete interaction between the dependency relation, query vector and key vector. It also excludes the graph interaction with value vectors while maintaining good performance. We have evaluated our model on CoNLL 2005 and CoNLL 2009 SRL datasets and outperformed previous comparable models. Future studies can apply our model to any NLP task which might benefit from conditioning on the syntactic structure or other graphs.
Limitations
SynG2G-Tr encodes the syntactic dependency graph because the nodes of input and output graphs should be similar. Future work could include investigating the use of constituency graphs in the self-attention mechanism of Transformer (Vaswani et al., 2017b), where the nodes of the input graph (constituency graph) are different from those of the SRL output graph. In this paper, we initialise our model with the pre-trained BERT (Devlin et al., 2019) model. As future study, larger and better pretrained language models will be used for the initialisation of SynG2G-Tr models, to achieve better performance. Additionally, future studies can easily extend our work to multilingual SRL benchmarks.
7This leads to a BERT-based syntax-agnostic model, similar to Shi and Lin (2019).

heading: Acknowledgement
text: We are grateful to the Swiss National Science Foundation (SNSF), grant CRSII5-180320, for funding this work. We also thank members of the IDIAP NLU group for helpful discussions and suggestions. We are grateful to anonymous reviewers for their fruitful comments and corrections.

heading: Appendix A SRL Scorer and Decoder Details
text: Scorer. Inspired by Zhou et al. (2020a), we first define span representation (sij) as the difference between right and left end-points of the span:
sij= s⃗rj− ⃗sli (4)
where s⃗rj is defined as [ ⃗zj+1;z⃗j ], and ⃗sli is calculated as [ ⃗zi; ⃗zi+1]. ⃗zi is computed by dividing the output representation of Transformer (zi) in half. Argument (aij) and predicate (vk) representations are defined as:
aij=ReLU(W 1 srlsij+b 1 srl) vk=zk (5)
where W 1srl and b 1 srl are learned parameters and ReLU(.) is the Rectified Linear Unit (Nair and Hinton, 2010) function. We predict semantic roles as defined in Zhou et al. (2020a):
Φl(v,a)=W 3 srl(LN(W 2 srl[aij ;vk]+b 2 srl))+b 3 srl (6)
where LN(.) is the layer normalisation (Ba et al., 2016) function, and W 2srl, W 3 srl, b 2 srl, and b 3 srl are learned parameters. The semantic role score for a specific label lout is defined as:
Φl(v,a,lout)=[Φl(v,a)]lout (7)
Since the number of predicate-argument pairs is O(n3), we apply the pruning method proposed in Li et al. (2019); He et al. (2018a) by defining separate scorers for argument and predicate candidates (Φa and Φv), and pruning all but the top-ranked arguments and predicates based on their corresponding scores.
Training. The model is trained to optimise the probability P (ŷ|W,P,Gsyn) of predicate-argument pairs, conditioned on input sequence (W ), PoS tags (P ), and predicated dependency graph (Gsyn). This objective can be factorised as:
J(θ)= ∑
y∈Γ −logPθ(y|W,P,Gsyn)
= ∑
⟨v,a,lout⟩∈Γ −log exp(Φ(v,a,lout))∑ l̂∈Lsrlexp(Φ(v,a,l̂))
(8)
where Φ(v,a,lout) is defined as Φv(v)+Φa(a)+Φl(v,a,lout), and θ is model parameters. Γ is the set of predicate-argument-relation tuples for all possible predicate-argument pairs and either the correct relation or NONE.
Decoders. Following Zhou et al. (2020a), we apply a single dynamic programming decoder according to the uniform score following the non-overlapping constraints (Punyakanok et al., 2008).

heading: Appendix B Implementation Details and Pre-processing Steps
text: CoNLL 2005: In this shared task (Carreras and Màrquez, 2005) (under LDC license), the focus was on verbal predicates in English. The training data includes sections 2-21 of the Wall Street Journal (WSJ) dataset. Section 24 is considered as the development set, while section 23 is used as the in-domain test set. Three sections of the Brown corpus are used for the out-of-domain dataset. The dataset can be downloaded from here, and pre-processing steps are provided in here.
CoNLL 2009: This shared task (Hajič et al., 2009) (under LDC license) focused on dependency-based SRL and was created by merging PropBank and NomBank treebanks. We evaluate our models on the English dataset with the same split as the CoNLL 2005 dataset. The dataset and pre-processing steps can be found at here and here. The number of sentences in train and evaluation sets is as follows:

heading: Appendix C Hyper-parameters Setting
text: We use bert-large-whole-word-masking8 (345M parameters) for the initialisation of encoder in SynG2G-Tr model. We use Adam optimiser (Kingma and Ba, 2014) and apply separate optimisers for pre-trained parameters and randomly initialised ones. We use bucket batching, grouping sentences by their lengths to the same batch to speed up the model. Early stopping is used to mitigate over-fitting, as in previous work (Mohammadshahi et al., 2022b,a, 2019). In a pre-defined predicate setting, we use different dynamic programming decoders to find SRL graphs, since predicates are not necessarily the same in dependency-based and span-based SRL graphs. For choosing the best hyper-parameters, we use manual tuning to find the base learning rate and BERT learning rate. For other hyper-parameters, we follow previous work (Zhou et al., 2020a). The base learning rate is selected from {1e−2,1e−3,1.5e−3}, and the BERT learning rate is chosen from {1e−5,1.5e−5,2e−5}. So, we train our models with 9 different learning rates to find the best performing model based on the summation of F1 scores of span-based and dependency-based SRL graphs. We use NVIDIA GeForce GTX 1080 Ti for training and evaluating our models. 9 For the dependency parser, we apply the same hyper-parameters as Zhou et al. (2020a). We use the base learning rate of 2e−3, and the BERT learning rate of 1.5e−5. Here is the list of hyper-parameters for the SynG2G-Tr model:

heading: Appendix D Attention Visualisation
text: Figure 3 shows the attention weights for different layers of self-attention in the SynG2G-Tr model (Figure 3b-3d), alongside the dependency relation matrix (Figure 3a). The self-attention matrix includes four patterns. The first layer of the SynG2G-Tr model (Figure 3b) ignores the graph relations and learns string-local context information. For the middle layer (Figure 3c), attention weights partially use the graph relation pattern. Then, in the last layer (Figure 3d), the dependency graph relations are evident in the attention pattern. This demonstrates the benefit of adding the graph information with a soft bias, allowing the model to learn different structures using both local context and graph information. Furthermore, it can be inferred that the last layers of the self-attention mechanism require a global view and between-edge information, while the first few layers learn local context information. More examples are provided in Figures 4-5-6.
8https://github.com/google-research/bert. Apache License 2.0. 9The training time of SynG2G-Tr model is 0h20m40s, and the evaluation time is 0h02m24s.

heading: Appendix E Ablation Study
text: In Table 6, we analyse the interaction of the dependency graph with key and query vectors in the attention mechanism, as defined in Equation 2. Excluding the key interaction results in a similar attention mechanism as defined in Mohammadshahi and Henderson (2021). This SynG2G-Tr-key model achieves similar results compared to the SynG2G-Tr model on the WSJ test dataset given the predicate, but the SynG2G-Tr model outperforms it on all other settings, including both types of out-of-domain datasets, confirming that key interaction is a critical part of the SynG2G-Tr model. When both key and query interactions are excluded from the SynG2G-Tr model (SynG2G-Tr-key-query), it has significantly lower performance than the SynG2G-Tr model in all settings. This demonstrates the impact of encoding the graph relation embeddings in the self-attention mechanism of Transformer (Vaswani et al., 2017b) model. We also evaluate adding the interaction of graph relations with value vectors to the SynG2G-Tr model, as defined in Espinosa Anke et al. (2022); Mohammadshahi and Henderson (2020). The SynG2G-Tr+value model achieves similar or worse results compared to the SynG2G-Tr model. So, we exclude this interaction to speed up the modified attention mechanism.

heading: Future Work
text: Future studies can apply our model to any NLP task which might benefit from conditioning on the syntactic structure or other graphs. Limitations
SynG2G-Tr encodes the syntactic dependency graph because the nodes of input and output graphs should be similar. Future work could include investigating the use of constituency graphs in the self-attention mechanism of Transformer (Vaswani et al. , 2017b), where the nodes of the input graph (constituency graph) are different from those of the SRL output graph. In this paper, we initialise our model with the pre-trained BERT (Devlin et al. , 2019) model. As future study, larger and better pretrained language models will be used for the initialisation of SynG2G-Tr models, to achieve better performance. Additionally, future studies can easily extend our work to multilingual SRL benchmarks. 7This leads to a BERT-based syntax-agnostic model, similar to Shi and Lin (2019)