SlowGuess/ACL_2020_OCR huggingface dataset:
Transition-based parsers implemented with Pointer Networks have become the new state of the art in dependency parsing, excelling in producing labelled syntactic trees and outperforming graph-based models in this task. In order to further test the capabilities of these powerful neural networks on a harder NLP problem, we propose a transition system that, thanks to Pointer Networks, can straightforwardly produce labelled directed acyclic graphs and perform semantic dependency parsing. In addition, we enhance our approach with deep contextualized word embeddings extracted from BERT. The resulting system not only outperforms all existing transition-based models, but also matches the best fully-supervised accuracy to date on the SemEval 2015 Task 18 English datasets among previous state-of-the-art graph-based parsers.
In dependency parsing, the syntactic structure of a sentence is represented by means of a labelled tree, where each word is forced to be attached exclusively to another that acts as its head. In contrast, semantic dependency parsing (SDP) (Oepen et al., 2014) aims to represent binary predicate-argument relations between words of a sentence, which requires producing a labelled directed acyclic graph (DAG): not only semantic predicates can have multiple or zero arguments, but words from the sentence can be attached as arguments to more than one head word (predicate), or they can be outside the SDP graph (being neither a predicate nor an argument) as shown in the examples in Figure 1. Since existing dependency parsers cannot be directly applied, most SDP research has focused on adapting them to deal with the absence of single-head and connectedness constraints and to produce an SDP graph instead.
Figure 1: Sentence from the SemEval 2015 Task 18 development set parsed with semantic dependencies following the DM, PAS and PSD formalisms.
As in dependency parsing, we can find two main families of approaches to efficiently generate accurate SDP graphs. On the one hand, graph-based algorithms have drawn more attention since adapting them to this task is relatively straightforward. In particular, these globally optimized methods independently score arcs (or sets of them) and then search for a high-scoring graph by combining these scores. From one of the first graph-based DAG parsers proposed by McDonald and Pereira (2006) to the current state-of-the-art models (Wang et al., 2019; He and Choi, 2019), different graph-based SDP approaches have been presented, providing accuracies above their main competitors: transition-based DAG algorithms.
A transition-based parser generates a sequence of actions to incrementally build a valid graph (usually from left to right). This is typically done by local, greedy prediction and can efficiently parse a sentence in a linear or quadratic number of actions (transitions); however, the lack of global inference makes them more prone to suffer from error propa
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7035-7046 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
gation: i.e., since transitions are sequentially and locally predicted, an erroneous action can affect future predictions, having a significant impact in long sentences and being, to date, less appealing for SDP. In fact, in recent years only a few contributions, such as the system developed by Wang et al. (2018), present a purely transition-based SDP parser. It is more common to find hybrid systems that combine transition-based approaches with graph-based techniques to alleviate the impact of error propagation in accuracy (Du et al., 2015), but this penalizes the efficiency provided by transition-based algorithms.
Away from the current mainstream, we present a purely transition-based parser that directly generates SDP graphs without the need of any additional techniques. We rely on Pointer Networks (Vinyals et al., 2015) to predict transitions that can attach multiple heads to the same word and incrementally build a labelled DAG. This kind of neural networks provide an encoder-decoder architecture that is capable of capturing information from the whole sentence and previously created arcs, alleviating the impact of error propagation and already showing remarkable results in transition-based dependency parsing (Ma et al., 2018; Fernández-González and Gómez-Rodríguez, 2019). We further enhance our neural network with deep contextualized word embeddings extracted from the pre-trained language model BERT (Devlin et al., 2019).
The proposed SDP parser<sup>1</sup> can process sentences in SDP treebanks (where structures are sparse DAGs with a low in-degree) in  $O(n^{2} \log n)$  time, or  $O(n^{2})$  without cycle detection. This is more efficient than the current fully-supervised state-of-the-art system by Wang et al. (2019)  $O(n^{3})$  without cycle detection), while matching its accuracy on the SemEval 2015 Task 18 datasets (Oepen et al., 2015). In addition, we also prove that our novel transition-based model provides promising accuracies in the semi-supervised scenario, achieving some state-of-the-art results.
An early approach to DAG parsing was implemented as a modification to a graph-based parser by McDonald and Pereira (2006). This produced DAGs using approximate inference by first finding a dependency tree, and then adding extra edges that would increase the graph's overall score. A
few years later, this attempt was outperformed by the first transition-based DAG parser by Sagae and Tsujii (2008). They extended the existing transition system by Nivre (2003) to allow multiple heads per token. The resulting algorithm was not able to produce DAGs with crossing dependencies, requiring the pseudo-projective transformation by Nivre and Nilsson (2005) (plus a cycle removal procedure) as a post-processing stage.
More recently, there has been a predominance of purely graph-based DAG models since the SemEval 2015 Task 18 (Oepen et al., 2015). Almeida and Martins (2015) adapted the pre-deep-learning dependency parser by Martins et al. (2013) to produce SDP graphs. This graph-based parser encodes higher-order information with hand-crafted features and employs the  $AD^3$  algorithm (Martins et al., 2011) to find valid DAGs during decoding. This was extended by Peng et al. (2017) with BiLSTM-based feature extraction and multitask learning: the three formalisms considered in the shared task were jointly learned to improve final accuracy.
After the success of Dozat et al. (2017) in graph-based dependency parsing, Dozat and Manning (2018) proposed minor adaptations to use this bi-affine neural architecture to produce SDP graphs. To that end, they removed the maximum spanning tree algorithm (Chu and Liu, 1965; Edmonds, 1967) necessary for decoding well-formed dependency trees and simply kept those edges with a positive score. In addition, they trained the unlabelled parser with a sigmoid cross-entropy (instead of the original softmax one) in order to accept multiple heads.
The parser by Dozat and Manning (2018) was recently improved by two contributions. Firstly, Wang et al. (2019) manage to add second-order information for score computation and then apply either mean field variational inference or loopy belief propagation information to decode the highest-scoring SDP graph. While significantly boosting parsing accuracy, the original  $O(n^{2})$  runtime complexity is modified to  $O(n^{3})$  in the resulting SDP system. Secondly, He and Choi (2019) significantly improve the original parser's accuracy by not only using contextualized word embeddings extracted from BERT (Devlin et al., 2019), but also introducing contextual string embeddings (called Flair) (Akbik et al., 2018), which consist in a novel type of word vector representations based on character
<sup>1</sup>Source code available at https://github.com/danifg/SemanticPointer.
7036
level language modeling. Both extensions, (Wang et al., 2019) and (He and Choi, 2019), are currently the state of the art on the SemEval 2015 Task 18 in the fully-supervised and semi-supervised scenarios, respectively.
Kurita and Søgaard (2019) have also recently proposed a complex approach that iteratively applies the syntactic dependency parser by Zhang et al. (2017), sequentially building a DAG structure. At each iteration, the graph-based parser selects the highest-scoring arcs, keeping the single-head constraint. The process ends when no arcs are added in the last iteration. The combination of partial parses results in an SDP graph. Since the graph is built in a sequential process, they use reinforcement learning to guide the model through more optimal paths. Following Peng et al. (2017), multi-task learning is also added to boost final accuracy.
On the other hand, the use of transition-based algorithms in the SDP task had been less explored until very recently. Du et al. (2015) presented a voting-based ensemble of fourteen graph- and transition-based parsers. In their work, they noticed that individual graph-based models outperform transition-based algorithms, assigning, during voting, higher weights to them. Among the transition systems used, we can find the one developed by Titov et al. (2009), which is not able to cover all SDP graphs.
We have to wait until the work by Wang et al. (2018) to see that a purely transition-based SDP parser (enhanced with a simple model ensemble technique) can achieve competitive results. They simply modified the preconditions of the complex transition system by Choi and McCallum (2013) to produce unrestricted DAG structures. In addition, their system was implemented by means of stack-LSTMs (Dyer et al., 2015), enhanced with BiLSTMs and Tree-LSTMs for feature extraction.
We are, to the best of our knowledge, first to explore DAG parsing with Pointer Networks, proposing a purely transition-based algorithm that can be a competitive alternative to graph-based SDP models.
Finally, during the reviewing process of this work, the proceedings of the CoNLL 2019 shared task (Oepen et al., 2019) were released. In that event, SDP parsers were evaluated on updated versions of SemEval 2015 Task 18 datasets, as well as on datasets in other semantic formalisms such as Abstract Meaning Representation (AMR)
(Banarescu et al., 2013) and Universal Cognitive Conceptual Annotation (UCCA) (Abend and Rappoport, 2013). Although graph-based parsers achieved better accuracy in the SDP track, several BERT-enhanced transition-based approaches were proposed. Among them we can find an extension (Che et al., 2019) of the system by Wang et al. (2018), several adaptations for SDP (Hershcovich and Arviv, 2019; Bai and Zhao, 2019) of the transition-based UCCA parser by Hershcovich et al. (2017), as well as an SDP variant (Lai et al., 2019) of the constituent transition system introduced by Fernandez-Gonzalez and Gomez-Rodriguez (2019). Also in parallel to the development of this research, Zhang et al. (2019) proposed a transition-based parser that, while it can be applied for SDP, was specifically designed for AMR and UCCA parsing (where graph nodes do not correspond with words and must be generated during the parsing process). In particular, this approach incrementally builds a graph by predicting at each step a semantic relation composed of the target and source nodes plus the arc label. While this can be seen as an extension of our approach for those tasks where nodes must be generated, its complexity penalizes accuracy in the SDP task.
3 Multi-head Transition System
We design a novel transition system that is able to straightforwardly attach multiple heads to each word in a single pass, incrementally building, from left to right, a valid SDP graph: a labelled DAG.
To implement it, we use Pointer Networks (Vinyals et al., 2015). These neural networks are able to learn the conditional probability of a sequence of discrete numbers that correspond to positions in an input sequence and, at decoding time, perform as a pointer that selects a position from the input. In other words, we can train this neural network to, given a word, point to the position of the sentence where its head (Fernández-González and Gómez-Rodríguez, 2019) or dependent words (Ma et al., 2018) are located, depending on what interpretation we use during training. In particular, (Fernández-González and Gómez-Rodríguez, 2019) proved to be more suitable for dependency parsing than (Ma et al., 2018) since it requires half as many steps to produce the same dependency parse, being not only faster, but also more accurate (as this mitigates the impact of error propagation).
Inspired by Fernández-González and Gómez-
7037
Rodriguez (2019), we train a Pointer Network to point to the head of a given word and propose an algorithm that does not use any kind of data structures (stack or buffer, required in classic transition-based parsers (Nivre, 2008)), but just a focus word pointer  $i$  for marking the word currently being processed. More in detail, given an input sentence of  $n$  words  $w_{1},\ldots ,w_{n}$ , the parsing process starts with  $i$  pointing at the first word  $w_{1}$ . At each time step, the current focus word  $w_{i}$  is used by the Pointer Network to return a position  $p$  from the input sentence (or 0, where the ROOT node is located). This information is used to choose between the two available transitions:
- If  $p \neq i$ , then the pointed word  $w_{p}$  is considered as a semantic head word (predicate) of  $w_{i}$  and an Attach- $p$  transition is applied, creating the directed arc  $w_{p} \rightarrow w_{i}$ . The Attach- $p$  transition is only permissible if the resulting predicate-argument arc neither exists nor generates a cycle in the already-built graph, in order to output a valid DAG.
- On the contrary, if  $p = i$  (i.e., the model points to the current focus word), then  $w_{i}$  is considered to have found all its head words, and a Shift transition is chosen to move  $i$  one position to the right to process the next word  $w_{i+1}$ .
The parsing ends when the last word from the sentence is shifted, meaning that the input is completely processed. As stated by Ma et al. (2018) for attaching dependent words, it is necessary to fix the order in which (in our case, head) words are assigned in order to define a deterministic decoding. As the sentence is parsed in a left-to-right manner, we adopt the same order for head assignments. For instance, the SDP graph in Figure 1(a) is produced by the transition sequence described in Table 1. We just need  $n$  Shift transitions to move the focus word pointer through the whole sentence and  $m$  Attach-  $p$  transitions to create the  $m$  arcs present in the SDP graph.
It is worth mentioning that we manage to significantly reduce the amount of transitions necessary for generating DAGs in comparison to those proposed in the complex transition systems by Choi and McCallum (2013) and Titov et al. (2009), used in the SDP systems by Wang et al. (2018) and Du et al. (2015), respectively. In addition, the described multi-head transition system is able to
p	transition	focus wordi	added arc
The1
1	Shift	results2
1	Attach-1	results2	The1 → results2
4	Attach-4	results2	results2 ← in4
2	Shift	were3
3	Shift	in4
0	Attach-0	in4	ROOT0 → in4
6	Attach-6	in4	in4 ← with6
4	Shift	line5
4	Attach-4	line5	in4 → line5
5	Shift	with6
6	Shift	analysts7
7	Shift	'8
8	Shift	expectations9
6	Attach-6	expectations9	with6 → expectations9
7	Attach-7	expectations9	analysts7 → expectations9
9	Shift	.10
10	Shift
Table 1: Transition sequence for generating the SDP graph in Figure 1(a).
directly produce any DAG structure without exception, while some transition systems, such as the mentioned (Sagae and Tsujii, 2008; Titov et al., 2009), are limited to a subset of DAGs.
Finally, while the outcome of the proposed transition system is a SDP graph without cycles, in other research, such as (Kurita and Søgaard, 2019) and state-of-the-art models by Dozat and Manning (2018) and Wang et al. (2019), the parser is not forced to produce well-formed DAGs, allowing the presence of cycles.
4 Neural Network Architecture
4.1 Basic Approach
Vinyals et al. (2015) introduced an encoder-decoder architecture, called Pointer Network, that uses a mechanism of neural attention (Bahdanau et al., 2014) to select positions from the input sequence, without requiring a fixed size of the output dictionary. This allows Pointer Networks to easily address those problems where the target classes considered at each step are variable and depend on the length of the input sequence. We prove that implementing the transition system previously defined on this neural network results in an accurate SDP system.
We follow previous work in dependency parsing (Ma et al., 2018; Fernandez-Gonzalez and Gomez-Rodriguez, 2019) to design our neural architecture:
Encoder A BiLSTM-CNN architecture (Ma and Hovy, 2016) is used to encode the input sentence
7038
$w_{1},\ldots ,w_{n}$  ,word by word, into a sequence of encoder hidden states  $\mathbf{h}_1,\dots ,\mathbf{h}_n$  . CNNs with max pooling are used for extracting character-level representations of words and, then, each word  $w_{i}$  is represented by the concatenation of character  $(\mathbf{e}_i^c)$  word  $(\mathbf{e}_i^w)$  , lemma  $(\mathbf{e}_i^l)$  and POS tag  $(\mathbf{e}_i^p)$  embeddings:
$$
\mathbf {x} _ {i} = \mathbf {e} _ {i} ^ {c} \oplus \mathbf {e} _ {i} ^ {w} \oplus \mathbf {e} _ {i} ^ {l} \oplus \mathbf {e} _ {i} ^ {p}
$$
After that, the  $\mathbf{x}_i$  of each word  $w_i$  is fed one-by-one into a BiLSTM that captures context information in both directions and generates a vector representation  $\mathbf{h}_i$ :
$$
\mathbf {h} _ {i} = \mathbf {h} _ {l i} \oplus \mathbf {h} _ {r i} = \mathbf {B i L S T M} (\mathbf {x} _ {i})
$$
In addition, a special vector representation  $\mathbf{h}_0$ , denoting the ROOT node, is prepended at the beginning of the sequence of encoder hidden states.
Decoder An LSTM is used to output, at each time step  $t$ , a decoder hidden state  $\mathbf{s}_t$ . As input of the decoder, we use the encoder hidden state  $\mathbf{h}_i$  of the current focus word  $w_i$  plus extra high-order features. In particular, we take into account the hidden state of the last head word  $(\mathbf{h}_h)$  attached to  $w_i$ , which will be a co-parent of a future predicate assigned to  $w_i$ . Following Ma et al. (2018), we use element-wise sum to add this information without increasing the dimensionality of the input:
$$
\mathbf {r} _ {i} = \mathbf {h} _ {i} + \mathbf {h} _ {h}; \mathbf {s} _ {t} = \mathbf {L S T M} (\mathbf {r} _ {i})
$$
Note that feature information like this can be easily added in transition-based models without increasing the parser's runtime complexity, something that does not happen in graph-based models, where, for instance, the second-order features added by Wang et al. (2019) penalize runtime complexity.
We experimented with other high-order features such as grandparent or sibling information of the current focus word  $w_{i}$ , but no significant improvements were obtained from their addition, so they were discarded for simplicity. Further feature exploration might improve parser performance, but we leave this for future work.
Once  $\mathbf{s}_t$  is generated, the attention vector  $\mathbf{a}^t$ , which will work as a pointer over the input, must be computed in the pointer layer. First, following the previously cited work, the scores between  $\mathbf{s}_t$  and each encoder hidden representation  $\mathbf{h}_j$  from the input sentence are computed using this biaffine attention scoring function (Dozat and Manning,
2017):
$$
\begin{array}{l} \mathbf {v} _ {j} ^ {t} = \mathbf {s c o r e} (\mathbf {s} _ {t}, \mathbf {h} _ {j}) = f _ {1} (\mathbf {s} _ {t}) ^ {T} W f _ {2} (\mathbf {h} _ {j}) \\ + \mathbf {U} ^ {T} f _ {1} (\mathbf {s} _ {t}) + \mathbf {V} ^ {T} f _ {2} (\mathbf {h} _ {j}) + \mathbf {b} \\ \end{array}
$$
where parameter  $W$  is the weight matrix of the bilinear term,  $\mathbf{U}$  and  $\mathbf{V}$  are the weight tensors of the linear terms and  $\mathbf{b}$  is the bias vector. In addition,  $f_{1}(\cdot)$  and  $f_{2}(\cdot)$  are two single-layer multilayer perceptrons (MLP) with ELU activation, proposed by (Dozat and Manning, 2017) for reducing dimensionality and minimizing overfitting.
Then, a softmax is applied on the resulting score vector  $\mathbf{v}^t$  to compute a probability distribution over the input words:
$$
\mathbf {a} ^ {t} = \operatorname {s o f t m a x} (\mathbf {v} ^ {t})
$$
The resulting attention vector  $\mathbf{a}^t$  can now be used as a pointer to select the highest-scoring position  $p$  from the input. This information will be employed by the transition system to choose between the two available actions and create a predicate-argument relation between  $w_{p}$  and  $w_{i}$  (Attach- $p$ ) or move the focus word pointer to  $w_{i+1}$  (Shift). In case the chosen Attach- $p$  is forbidden due to the acyclicity constraint, the next highest-scoring position in  $\mathbf{a}^t$  is considered as output instead. Figure 2 depicts the neural architecture and the decoding procedure for the SDP structure in Figure 1(a).
Label prediction We jointly train a multi-class classifier that scores every label for each pair of words. This shares the same encoder and uses the same biaffine attention function as the pointer:
$$
\begin{array}{l} \mathbf {s} _ {t p} ^ {l} = \mathbf {s c o r e} (\mathbf {s} _ {t}, \mathbf {h} _ {p}, l) = g _ {1} (\mathbf {s} _ {t}) ^ {T} W _ {l} g _ {2} (\mathbf {h} _ {p}) \\ + \mathbf {U} _ {l} ^ {T} g _ {1} (\mathbf {s} _ {t}) + \mathbf {V} _ {l} ^ {T} g _ {2} (\mathbf {h} _ {p}) + \mathbf {b} _ {l} \\ \end{array}
$$
where a distinct weight matrix  $W_{l}$ , weight tensors  $\mathbf{U}_l$  and  $\mathbf{V}_l$  and bias  $\mathbf{b}_l$  are used for each label  $l$ , where  $l \in \{1,2,\dots,L\}$  and  $L$  is the number of labels. In addition,  $g_{1}(\cdot)$  and  $g_{2}(\cdot)$  are two single-layer MLPs with ELU activation.
The scoring function is applied over each predicted arc between the dependent word  $w_{i}$  (represented by  $\mathbf{s}_t$ ) and the pointed head word  $w_{p}$  in position  $p$  (represented by  $\mathbf{h}_p$ ) to compute the score of each possible label and assign the highest-scoring one.
Training Objectives The Pointer Network is trained to minimize the negative log likelihood
7039
Figure 2: Neural network architecture and decoding steps to partially parse the SDP graph in Figure 1.
(implemented as cross-entropy loss) of producing the correct SDP graph  $y$  for a given sentence  $x$ :  $P_{\theta}(y|x)$ . Let  $y$  be a DAG for an input sentence  $x$  that is decomposed into a set of  $m$  directed arcs  $a_1, \ldots, a_m$  following a left-to-right order. This probability can be factorized as follows:
$$
P _ {\theta} (y | x) = \prod_ {k = 1} ^ {m} P _ {\theta} \left(a _ {k} \mid a _ {<   k}, x\right)
$$
where  $a_{< k}$  denotes previous predicted arcs.
On the other hand, the labeler is trained with softmax cross-entropy to minimize the negative log likelihood of assigning the correct label  $l$ , given a dependency arc with head word  $w_{h}$  and dependent word  $w_{i}$ .
The whole neural model is jointly trained by summing the parser and labeler losses prior to computing the gradients. In that way, model parameters are learned to minimize the sum of the cross-entropy loss objectives over the whole corpus.
4.2 Deep Contextualized Word Embeddings Augmentation
In order to further improve the accuracy of our approach, we augment our model with deep contextualized word embeddings provided by the widely-used pre-trained language model BERT (Devlin et al., 2019).
Instead of including and training the whole BERT model as encoder of our system, we follow the common, greener and more cost-effective approach of leveraging the potential of BERT by extracting the weights of one or several layers as word-level embeddings. To that end, the pretrained uncased  $\mathrm{BERT}_{\mathrm{BASE}}$  model is used.