SlowGuess/ACL_2020_OCR huggingface dataset:
Tianze Shi Cornell University tianze@cs.cornell.edu
Lillian Lee Cornell University llee@cs.cornell.edu
An interesting and frequent type of multiword expression (MWE) is the headless MWE, for which there are no true internal syntactic dominance relations; examples include many named entities ("Wells Fargo") and dates ("July 5, 2020") as well as certain productive constructions ("blow for blow", "day after day"). Despite their special status and prevalence, current dependency-annotation schemes require treating such flat structures as if they had internal syntactic heads, and most current parsers handle them in the same fashion as headed constructions. Meanwhile, outside the context of parsing, taggers are typically used for identifying MWEs, but taggers might benefit from structural information. We empirically compare these two common strategies—Parsing and tagging—for predicting flat MWEs. Additionally, we propose an efficient joint decoding algorithm that combines scores from both strategies. Experimental results on the MWE-Aware English Dependency Corpus and on six non-English dependency treebanks with frequent flat structures show that: (1) tagging is more accurate than parsing for identifying flat-structure MWEs, (2) our joint decoder reconciles the two different views and, for non-BERT features, leads to higher accuracies, and (3) most of the gains result from feature sharing between the parsers and taggers.
Headless multi-word expressions (MWEs), including many named entities and certain productive constructions, are frequent in natural language and are important to NLP applications. In the context of dependency-based syntactic parsing, however, they pose an interesting representational challenge. Dependency-graph formalisms for syntactic structure represent lexical items as nodes and head-dominates-modifier/argument relations between
Figure 1: Dependency tree from the MWE-Aware English Dependency Corpus, imposing a "head" relationship between the words in the actually headless MWE Mellon Capital. Also shown are MWE BIO labels.
lexical items as directed arcs on the corresponding pair of nodes. Most words can be assigned clear linguistically-motivated syntactic heads, but several frequently occurring phenomena do not easily fit into this framework, including punctuation, coordinating conjunctions, and "flat", or headless MWEs. While the proper treatment of headless constructions in dependency formalisms remains debated (Kahane et al., 2017; Gerdes et al., 2018), many well-known dependency treebanks handle MWEs by giving their component words a "default head", which is not indicative of a true dominance relation, but rather as "a tree encoding of a flat structure without a syntactic head" (de Marneffe and Nivre, 2019, pg. 213). Fig. 1 shows an example: the headless MWE Mellon Capital has its first word, Mellon, marked as the "head" of Capital.
Despite the special status of flat structures in dependency tree annotations, most state-of-the-art dependency parsers treat all annotated relations equally, and thus do not distinguish between headed and headless constructions. When headless-span identification (e.g., as part of namedentity recognition (NER)) is the specific task at hand, begin-chunk/inside-chunk/outside-chunk (BIO) tagging (Ramshaw and Marcus, 1995) is generally adopted. It is therefore natural to ask whether parsers are as accurate as taggers in identifying these "flat branches" in dependency trees. Additionally, since parsing and tagging represent
8780
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8780-8794 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
two different views of the same underlying structures, can joint decoding that combines scores from the two modules and/or joint training under a multitask learning (MTL) framework derive more accurate models than parsing or tagging alone?
To facilitate answering these questions, we introduce a joint decoder that finds the maximum sum of scores from both BIO tagging and parsing decisions. The joint decoder incorporates a special deduction item representing continuous headless spans, while retaining the cubic-time efficiency of projective dependency parsing. The outputs are consistent structures across the tagging view and the parsing view.
We perform evaluation of the different strategies on the MWE-Aware English Dependency Corpus and treebanks for five additional languages from the Universal Dependencies 2.2 corpus that have frequent multi-word headless constructions. On average, we find taggers to be more accurate than parsers at this task, providing  $0.59\%$  (1.42%) absolute higher F1 scores with(out) pretrained contextualized word representations. Our joint decoder combining jointly-trained taggers and parsers further improves the tagging strategy by  $0.69\%$  (1.64%) absolute. This corroborates early evidence (Finkel and Manning, 2009) that joint modeling with parsing improves over NER. We also show that neural representation sharing through MTL is an effective strategy, as it accounts for a large portion of our observed improvements. Our code is publicly available at https://github.com/tzshi/flat-mweParsing.
2 Background on Headless Structures
A (multi-word) headless construction, or flat structure, is a span of lexical items that together reference a single concept and where no component is a syntactically more plausible candidate for the span's head than any other component. Examples are boldfaced in the following English sentences.
(1) Within the scope of this paper:
a. ACL starts on July 5, 2020.
b. My bank is Wells Fargo.
c. The candidates matched each other insult for insult. (Jackendoff, 2008)
(1)a and (1)b show that dates and many named entities can be headless constructions, suggesting that they are frequent. Indeed, in the MWE-Aware English Dependency Corpus (Kato et al., 2017),
nearly half of the sentences contain headless constructions,  $75\%$  of which are named entities. For comparison, (2) shows examples of non-flat MWEs, which are also interesting and important, but they are beyond the focus of our paper.
(2) Outside the scope of this paper:
a. congressman at large (Sag et al., 2002) [head = "congressman"]
b. I have moved on. [verb-particle construction, head = "moved"]
c. I take your argument into account. (Constant et al., 2017) [light-verb construction, head = "take"]
Returning to headless MWEs, the choice of representation for headless spans depends on the task. In named-entity recognition, such spans are often treated as BIO tag sequences: for example, in Fig. 1, "Mellon" is tagged as "B" and "Capital" is tagged as "T". In dependency parsing, where labeled dependency arcs are the only way to express a syntactic analysis (short of treating MWEs as atomic lexical items, which would result in a chicken-and-egg problem) is to impose arcs within the MWE's span. Different corpora adopt different annotation conventions. The MWE-Aware English Dependency Corpus uses the arc label mwe_NNP, as shown in Fig. 1. The Universal Dependencies (UD; Nivre et al., 2018) annotation guidelines have all following tokens in such constructions attached to the first one via arcs labeled flat, a choice that is admittedly "in principle arbitrary".<sup>2</sup>
The frequency of flat structures across different treebanks varies according to language, genre, and even tokenization guidelines, among other factors. Table 1 lists the UD 2.2 treebanks with the highest and lowest percentage of flat relations. While the Korean treebank ko_gsd (with the highest percentage) splits up most names into multiple tokens and connects them through flat, the Japanese treebank ja_gsd (no flats at all) treats all names as compound nouns, and thus represents them as having internal structure without any indication that a special case has occurred.3 Fig. 2 shows examples from the UD parallel treebanks, illustrating
<sup>1</sup>In this paper, we adopt the original BIO tagset, which cannot properly represent discontinuous MWEs. See Schneider et al. (2014) for modified tagsets providing such support.
${}^{2}$  universaldependencies.org/u/dep/flat.html
<sup>3</sup>Some flat structures can end up using other dependency labels such as compound, as a result of the fact that many UD treebanks, including ja_gsd, are automatically converted from non-UD style annotations. The UD annotations depend
8781
Figure 2: An illustration of flat-structure annotation variation across treebanks: a set of parallel sentences, all containing the conceptually headless MWE "Martin Luther King, Jr." (underlined), from UD 2.2 (treebank code _pud) in English, German, Chinese, Japanese, Turkish, and Portuguese (top to bottom). The intent of this figure is not to critique particular annotation decisions, but to demonstrate the notation, concepts, and data extraction methods used in our paper. To wit: Highlights/black-background indicate well-formed flat-MWE tree fragments according to the principles listed in §4. BIO sequences are induced by the longest-spanning flat arcs. When there is a mismatch between the highlighted tree fragments and the BI spans—here, in the German, Chinese and Turkish examples—it is because the dependency trees do not fully conform to the UD annotation guidelines on headless structures.
8782
Treebank (Language)	% of flat
graphs ↓	arcs
19 treebanks with highest percentages:
ko_gsd (Korean)	67.84	15.35
id_gsd (Indonesian)	61.63	9.39
ca_ancora (Catalan)	41.11	3.32
nl_lassysmall (Dutch)	38.90	5.87
ar_nyuad (Arabic)	37.63	2.19
es_ancora (Spanish), sr_set (Serbian), it_postwita (Italian), pt_bosque (Portuguese), pt_gsd (Portuguese), fa_seraji (Persian), de_gsd (German), hu_szeged (Hungarian), fr_gsd (French), es_gsd (Spanish), he_htb (Hebrew), kk_ktb (Kazakh), be_hse (Belarusian), nl_alpino (Dutch)	>20.00
...	...
12 treebanks without flat arcs:
cs_cltt (Czech), grc_perseus (Ancient Greek), hi_hdtb (Hindi), ja_gsd (Japanese), ja_bccwj (Japanese), la_ittb (Latin), la_perseus (Latin), no_nynorsklia (Norwegian), swl_sslc (Swedish Sign Language), ta_ttb (Tamil), ur.udtb (Urdu), vi_vtb (Vietnamese)	0.00	0.00
Table 1: The UD 2.2 training treebanks with highest and lowest percentage of flat arcs, out of 90 treebanks.
the diversity of annotation for the same sentence rendered in different languages.
Overall, more than  $20\%$  of the treebanks in the UD 2.2 collection have flat structures in more than  $20\%$  of their training-set sentences. $^{4}$  Therefore, a parsing approach taking into account the special status of headless structural representations can potentially benefit models for a large number of languages and treebanks.
Formally, given an  $n$ -word sentence  $w = w_{1}, w_{2}, \ldots, w_{n}$ , we define its dependency structure to be a graph  $G = (V, E)$ . Each node in  $V$  corresponds to a word in the sentence. Each (labeled) edge  $(h, m, r) \in E$  denotes a syntactic relation labeled  $r$  between the head word  $w_{h}$  and modifier word  $w_{m}$ , where  $h, m \in \{0, 1, \ldots, n\}$  and  $0$  denotes the dummy root of the sentence. Since we work with dependency treebanks, we require that the edges in  $E$  form a tree. To represent a multi-word headless span  $w_{i}, \ldots, w_{j}$ , all subsequent words in the span are attached to the beginning word  $w_{i}$ , i.e.,  $\forall k \in \{i + 1, \ldots, j\}$ ,  $(i, k, f) \in E$ , where  $f$  is the special syntactic relation label de
noting headless structures (flat in UD annotation). Alternatively, one can also use a BIO tag sequence  $T = (t_{1}, t_{2}, \ldots, t_{n}) \in \{\mathrm{B}, \mathrm{I}, \mathrm{O}\}^{n}$  to indicate the location of any headless spans within  $w$ . The headless MWE span  $w_{i}, \ldots, w_{j}$  has the corresponding tags  $t_{i} = \mathrm{B}$  and  $\forall k \in \{i + 1, \ldots, j\}$ ,  $t_{k} = \mathrm{I}$ ; tokens outside any spans are assigned the tag O. We call  $G$  and  $T$  consistent if they indicate the same set of headless spans for  $w$ .
3 Three Approaches
We first present the standard approaches of edge-factored parsing (§3.2) and tagging (§3.3) for extracting headless spans in dependency trees, and then introduce a joint decoder (§3.4) that finds the global maximum among consistent (tree structure, tag sequence) pairs.
Given a length- $n$  sentence  $w$ —which we henceforth denote with the variable  $x$  for consistency with machine-learning conventions—we first extract contextualized representations from the input to associate each word with a vector  $\mathbf{x}_0$  (for the dummy word "root"),  $\mathbf{x}_1$ , ...,  $\mathbf{x}_n$ . We consider two common choices of feature extractors: (1) bidirectional long short-term memory networks (biLSTMs; Graves and Schmidhuber, 2005) which
on how detailed the original syntactic analyses are and the accuracies of the conversion algorithms.
Measured on the 90 treebanks with training splits.
8783
have been widely adopted in dependency parsing (Kiperwasser and Goldberg, 2016; Dozat and Manning, 2017) and sequence tagging (Ma and Hovy, 2016); and (2) the Transformer-based (Vaswani et al., 2017) BERT feature extractor (Devlin et al., 2019), pre-trained on large corpora and known to provide superior accuracies on both tasks (Kitaev et al., 2019; Kondratyuk and Straka, 2019). For BERT models, we fine-tune the representations from the final layer for our parsing and tagging tasks. When the BERT tokenizer renders multiple tokens from a single pre-tokenized word, we follow Kitaev et al. (2019) and use the BERT features from the last token as its representation.
3.2 (Edge-Factored) Parsing
Since we consider headless structures that are embedded inside parse trees, it is natural to identify them through a rule-based post-processing step after full parsing. Our parsing component replicates that of the state-of-the-art Che et al. (2018) parser, which has the same parsing model as Dozat and Manning (2017). We treat unlabelled parsing as a head selection problem (Zhang et al., 2017) with deep biaffine attention scoring:
$$
\mathbf {h} _ {i} ^ {\mathrm {a t t a c h}} = \mathbf {M L P} ^ {\mathrm {a t t a c h - h e a d}} (\mathbf {x} _ {i})
$$
$$
\mathbf {m} _ {j} ^ {\text {a t t a c h}} = \operatorname {M L P} ^ {\text {a t t a c h - m o d}} (\mathbf {x} _ {j})
$$
$$
\mathbf {s} _ {i, j} = \left[ \mathbf {h} _ {i} ^ {\text {a t t a c h}}; 1 \right] ^ {\top} U ^ {\text {a t t a c h}} \left[ \mathbf {m} _ {j} ^ {\text {a t t a c h}}; 1 \right]
$$
$$
P (h _ {j} = i \mid x) = \operatorname {s o f t m a x} _ {i} (\mathbf {s} _ {:, j}),
$$
where  $\mathbf{MLP}^{\mathrm{attach - head}}$  and  $\mathbf{MLP}^{\mathrm{attach - mod}}$  are multilayer perceptrons (MLPs) that project contextualized representations into a  $d$ -dimensional space;  $[\cdot ;1]$  indicates appending an extra entry of 1 to the vector;  $U^{\mathrm{att}}\in \mathbb{R}^{(d + 1)\times (d + 1)}$  generates a score  $s_{i,j}$  for  $w_{j}$  attaching to  $w_{i}$  (which we can then refer to as the head of  $w_{j},h_{j}$ ); a softmax function defines a probability distribution over all syntactic head candidates in the argument vector (we use the range operator “:” to evoke a vector); and, recall, we represent potential heads as integers, so that we may write  $h_j = i\in \{0,\ldots ,n\}$ .
The model for arc labeling employs an analogous deep biaffine scoring function:
$$
\mathbf {h} _ {i} ^ {\text {r e l}} = \operatorname {M L P} ^ {\text {r e l - h e a d}} \left(\mathbf {x} _ {i}\right)
$$
$$
\mathbf {m} _ {j} ^ {\mathrm {r e l}} = \operatorname {M L P} ^ {\mathrm {r e l - m o d}} \left(\mathbf {x} _ {j}\right)
$$
$$
\mathbf {v} _ {i, j, r} = \left[ \mathbf {h} _ {i} ^ {\mathrm {r e l}}; 1 \right] ^ {\top} U _ {r} ^ {\mathrm {r e l}} \left[ \mathbf {m} _ {j} ^ {\mathrm {r e l}}; 1 \right]
$$
$$
P (r _ {j} = r \mid x, h _ {j} = i) = \operatorname {s o f t m a x} _ {r} (\mathbf {v} _ {i, j,:}),
$$
where  $r_j$  is the arc label between  $w_{h_j}$  and  $w_j$ .
The objective for training the parser is to minimize the cumulative negative log-likelihood
$$
\begin{array}{l} L ^ {\text {p a r s e}} = \sum_ {(i ^ {*}, j ^ {*}, r ^ {*}) \in E} \left[ - \log P \left(h _ {j ^ {*}} = i ^ {*} \mid x\right) \right. \\ - \log P (r _ {i} = r ^ {*} \mid x, h _ {j ^ {*}} = i ^ {*}) ]. \\ \end{array}
$$
After the model predicts a full parse, we extract headless structures as the tokens "covered" by the longest-spanning  $f$ -arcs ( $f =$  flat in UD).
3.3 Tagging
For extracting spans in texts, if one chooses to ignore the existence of parse trees, BIO tagging is a natural choice. We treat the decision for the label of each token as an individual multi-class classification problem. We let
$$
P \left(t _ {i} = t \mid x\right) = \operatorname {s o f t m a x} _ {t} \left(\mathbf {M L P} ^ {\text {t a g}} \left(\mathbf {x} _ {i}\right)\right),
$$
where  $\mathbf{MLP}^{\mathrm{tag}}$  has 3 output units corresponding to the scores for tags B, I and O respectively.
We train the tagger to minimize
$$
L ^ {\text {t a g}} = \sum_ {i} - \log P \left(t _ {i} = t _ {i} ^ {*} \mid x\right),
$$
where  $t^*$  corresponds to the gold BIO sequence. During inference, we predict the BIO tags independently at each token position and interpret the tag sequence as a set of MWE spans. As a post-processing step, we discard all single-token spans, since the task is to predict multi-word spans.
3.4 A Joint Decoder
A parser and a tagger take two different views of the same underlying data. It is thus reasonable to hypothesize that a joint decoding process that combines the scores from the two models might yield more accurate predictions. In this section, we propose such a joint decoder to find the parser+tagger-consistent structure with the highest product of probabilities. Formally, if  $\mathcal{V}$  is the output space for all consistent parse tree structures and BIO tag sequences, for  $y\in \mathcal{V}$  with components consisting
Sequence tagging is traditionally handled by conditional random fields (Lafferty et al., 2001, CRFs). However, in recent experiments using contextualized representations on tagging (Clark et al., 2018; Devlin et al., 2019), CRF-style loss functions provide little, if any, performance gains compared with simple multi-class classification solutions, at slower training speeds, to boot. Our preliminary experiments with both bilSTM and BERT-based encoders corroborate these findings, and thus we report results trained without CRFs.
8784
Axioms:
Deduction Rules:
Figure 3: Eisner's (1996) algorithm adapted to parsing headless structures (unlabeled case), our modifications highlighted in blue. All deduction items are annotated with their scores. R-MWE combines BIO tagging scores and head selection parsing scores. We need no L-MWE because of the rightward headless-structure-arc convention.
of tags  $t_i$ , head assignments  $h_i$ , and relation labels  $r_i$ , our decoder aims to find  $\hat{y}$  satisfying
$$
\hat {y} = \operatorname * {a r g   m a x} _ {y \in \mathcal {Y}} P (y \mid x),
$$
$$
P (y \mid x) = \prod_ {i} P (t _ {i} \mid x) P (h _ {i} \mid x) P (r _ {i} \mid x, h _ {i}).
$$
Fig. 3 illustrates our joint decoder in the unlabeled case. $^{6}$  It builds on Eisner's (1996) decoder for projective dependency parsing. In addition to having single-word spans as axioms in the deduction system, we further allow multi-word spans to enter the decoding procedures through the axiom R-MWE. Any initial single-word spans receive an O-tag score for that word, while the newly introduced MWE spans receive B-tag, I-tag, attachment and relation scores that correspond to the two consistent views of the same structure. The time complexity for this decoding algorithm remains the same  $O(n^{3})$  as the original Eisner algorithm.
During training, we let the parser and the tagger share the same contextualized representation  $\mathbf{x}$  and optimize a linearly interpolated joint objective
$$
L ^ {\text {j o i n t}} = \lambda L ^ {\text {p a r s e}} + (1 - \lambda) L ^ {\text {t a g}},
$$
where  $\lambda$  is a hyper-parameter adjusting the relative weight of each module. $^{7}$  This is an instance of multi-task learning (MTL; Caruana, 1993, 1997). MTL has proven to be a successful technique (Colobert and Weston, 2008) on its own; thus, in our experiments, we compare the joint decoder and using the MTL strategy alone.
Data We perform experiments on the MWE-Aware English Dependency Corpus (Kato et al., 2017) and treebanks selected from Universal Dependencies 2.2 (UD; Nivre et al., 2018) for having frequent occurrences of headless MWE structures. The MWE-Aware English Dependency Corpus provides automatically unified named-entity annotations based on OntoNotes 5.0 (Weischedel et al., 2013) and Stanford-style dependency trees (de Marneffe and Manning, 2008). We extract MWE spans according to mwe_NNP dependency relations. We choose the UD treebanks based on two basic properties that hold for flat structures
In the labeled case, the parser further adds the arc-labeling scores to the R-MWE and LINK rules.
<sup>7</sup>The joint decoder combines tagging and parsing scores regardless of whether the two modules are jointly trained. However, since feature extraction is the most time-consuming step in our neural models, especially with BERT-based feature extractors, it is most practical to save memory and time by sharing common feature representations across modules.
8785
Treebank	# tokens	# headless arcs	%	# headless spans	Average span length	Compliance ratio
English	731,677	32,065	4.38%	16,997	2.89	100.00%
de_gsd	263,804	6,786	2.57%	5,663	2.59	93.00%
it_postwita	99,441	2,733	2.75%	2,277	2.26	94.89%
nl_alpino	186,046	4,734	2.54%	3,269	2.45	100.00%
nl_lassysmall	75,134	4,408	5.87%	3,018	2.46	99.82%
no_nynorsk	245,330	5,578	2.27%	3,670	2.54	99.78%
pt_bosque	206,739	5,375	2.60%	4,310	2.25	97.38%
Table 2: Dataset statistics. Language codes: de=German; it=Italian; nl=Dutch; no=Norwegian; pt=Portuguese.
conforming to the UD annotation guidelines: (1) all words that are attached via flat relations must be leaf nodes and (2) all words within a flat span should be attached to a common "head" word, and each arc label should be either flat or punct. For each treebank, we compute its compliance ratio, defined as the percentage of its trees containing flat arc labels that satisfy both properties above; and we filter out those with compliance ratios below  $90\%$ . We rank the remaining treebanks by their ratios of flat relations among all dependency arcs, and pick those with ratios higher than  $2\%$ . Six treebanks representing 5 languages, German (McDonald et al., 2013), Italian (Sanguinetti et al., 2018), Dutch (Bouma and van Noord, 2017), Norwegian (Solberg et al., 2014) and Portuguese (Rademaker et al., 2017), are selected for our experiments. Data statistics are given in Table 2. To construct gold-standard BIO labels, we extract MWE spans according to the longest-spanning arcs that correspond to headless structures.
Implementation Details We use 3-layer biLSTMs where each layer has 400 dimensions
${}^{8}$  punct inside a headless span is often used for hyphens and other internal punctuation in named entities. See the English sentence in Fig. 2 for an example.
The two properties defined in the UD guidelines for headless structures provide us with a common basis for uniform treatment across languages and treebanks. Unfortunately, the two properties can be violated quite often, due to issues in annotation and automatic treebank conversion into UD style. In 6 out of the top 10 treebanks containing the most flat relations, (at least one of) these properties are violated in more than  $35\%$  of the sentences with flat relations and have to be excluded from our experiments. We hope that ongoing community effort in data curation will facilitate evaluation on more diverse languages.
<sup>10</sup>It is a coincidence that all the selected languages are Indo-European (IE). Although there are some non-IE treebanks with high flat ratio, such as Korean (see Table 1), the annotated structures frequently break one or both of the basic properties. See Fig. 2 for violation examples.