Venue: NAACL 2018

# Sentences with Gapping: Parsing and Reconstructing Elided Predicates

Sebastian Schuster  Joakim Nivre\({}^{\ddagger}\) Christopher D. Manning

###### Abstract

Sentences with gapping, such as _Paul likes coffee and Mary tea_, lack an overt predicate to indicate the relation between two or more arguments. Surface syntax representations of such sentences are often produced poorly by parsers, and even if correct, not well suited to downstream natural language understanding tasks such as relation extraction that are typically designed to extract information from sentences with canonical clause structure. In this paper, we present two methods for parsing to a Universal Dependencies graph representation that explicitly encodes the elided material with additional nodes and edges. We find that both methods can reconstruct elided material from dependency trees with high accuracy when the parser correctly predicts the existence of a gap. We further demonstrate that one of our methods can be applied to other languages based on a case study on Swedish.

+
Footnote †: New Orleans, Louisiana, June 1 - 6, 2018. ©2018 Association for Computational Linguistics

## 1 Introduction

Sentences with gapping Ross (1970) such as _Paul likes coffee and Mary tea_ are characterized by having one or more conjuncts that contain multiple arguments or modifiers of an elided predicate. In this example, the predicate _likes_ is elided for the relation _Mary likes tea_. While these sentences appear relatively infrequently in most written texts, they are often used to convey a lot of factual information that is highly relevant for language understanding (NLU) tasks such as open information extraction and semantic parsing. For example, consider the following sentence from the WSJ portion of the Penn Treebank Marcus et al. (1993).

1. Unemployment has reached 27.6% in Azerbaijan, 25.7% in Tadzhikistan, 22.8% in Uzbekistan, 18.8% in Turkmenia, 18% in Armenia and 16.3% in Kirgizia, [...]

To extract the information about unemployment rates in the various countries, an NLU system has to identify that the percentages indicate unemployment rates and the locational modifiers indicate the corresponding country. Given only this sentence, or this sentence and a strict surface syntax representation that does not indicate elided predicates, this is a challenging task. However, given a dependency graph that reconstructs the elided predicate for each conjunct, the problem becomes much easier and methods developed to extract information from dependency trees of clauses with canonical structures are much more likely to extract the correct information from a gapped clause.

While gapping constructions receive a lot of attention in the theoretical syntax literature e.g., Ross (1970); Jackendoff (1971); Steedman (1990); Coppock (2001); Osborne (2006); Johnson (2014); Toosarvandani (2016); Kubota and Levine (2016), they have been almost entirely neglected by the NLP community so far. The Penn Treebank explicitly annotates gapping constructions, by co-indexing arguments in the clause with a predicate and the clause with the gap, but these co-indices are not included in the standard parsing metrics

Figure 1: Overview of our two approaches. Both methods first parse a sentence with gapping to one of two different dependency tree representations and then reconstruct the elided predicate from this tree.



and almost all parsers ignore them.1 Despite the sophisticated analysis of gapping within CCG (Steedman, 1990), sentences with gapping were deemed too difficult to represent within the CCGBank (Hockenmaier and Steedman, 2007). Similarly the treebanks for the Semantic Dependencies Shared Task (Oepen et al., 2015) exclude all sentences from the Wall Street Journal that contain gapping. Finally, while the tectogrammatical layer of the Prague Dependency Treebank (Bejcek et al., 2013) as well as the enhanced Universal Dependencies (UD) representation (Nivre et al., 2016) provide an analysis with reconstructed nodes for gapping constructions, there exist no methods to automatically parse to these representations.

Footnote 1: To the best of our knowledge, the parser by Kummerfeld and Klein (2017) is the only parser that tries to output the co-indexing of constituents in clauses with gapping but they lack an explicit evaluation of their co-indexing prediction accuracy.

Here, we provide the first careful analysis of parsing of gapping constructions, and we present two methods for reconstructing elided predicates in sentences with gapping within the UD framework. As illustrated in Figure 1, we first parse to a dependency tree and then reconstruct the elided material. The methods differ in how much information is encoded in the dependency tree. The first method adapts an existing procedure for parsing sentences with elided function words (Seeker et al., 2012), which uses composite labels that can be deterministically turned into dependency graphs in most cases. The second method is a novel procedure that relies on the parser only to identify a gap, and then employs an unsupervised method to reconstruct the elided predicates and reattach the arguments to the reconstructed predicate. We find that both methods can reconstruct elided predicates with very high accuracy from gold standard dependency trees. When applied to the output of a parser, which often fails to identify gapping, our methods achieve a sentence-level accuracy of 32% and 34%, significantly outperforming the recently proposed constituent parser by Kummerfeld and Klein (2017).

## 2 Background

### Gapping constructions

Gapping constructions in English come in many forms that can be broadly classified as follows.

_Single predicate gaps_: John **bought** books, and Mary flowers.

_Contiguous predicate-argument gap_

_(including ACCs)_: Eve **gave flowers** to Al and Sue to Paul. Eve **gave** a CD to Al and __ roses to Sue.

_Non-contiguous predicate-argument gap_: Arizona **elected** Goldwater **Senator**, and Pennsylvania ____ Schwelker ________.

_Verb cluster gap_: I **want to try to begin to write a novel** and ... Mary ____ a play. ... Mary ____ to write a play. ... Mary ____ to begin to write a play. ... Mary ____ to try to begin to write a play. (Ross, 1970)

The defining characteristic of gapping constructions is that there is a clause that lacks a predicate (the _gap_) but still contains two or more arguments or modifiers of the elided predicate (the _remnants_ or _orphans_). In most cases, the remnants have a corresponding argument or modifier (the _correspondent_) in the clause with the overt predicate.

These types of gapping also make up the majority of attested constructions in other languages. However, Wyngaard (2007) notes that Dutch permits gaps in relative clauses, and Farudi (2013) notes that Farsi permits gaps in finite embedded clauses even if the overt predicate is not embedded.2

Footnote 2: See Johnson (2014) or Schuster et al. (2017) for a more comprehensive overview of cross-linguistically attested gapping constructions.

### Target representation

We work within the UD framework, which aims to provide cross-linguistically consistent dependency annotations that are useful for NLP tasks. UD defines two types of representation: the _basic_ UD representation which is a strict surface syntax dependency tree and the _enhanced_ UD representation (Schuster and Manning, 2016) which may be a graph instead of a tree and may contain additional nodes. The analysis of gapping in the enhanced representation makes use of copy nodes for elided predicates and additional edges for elided arguments, which we both try to automatically reconstruct in this paper. In the simple case in which only one predicate was elided, there is exactly one 

copy node for the elided predicate, which leads to a structure that is identical to the structure of the same sentence without a gap.3

Footnote 3: To enhance the readability of our examples, we place the copy node in the sentence where the elided predicate would have been pronounced. However, as linear order typically does not matter for extracting information with dependency patterns, our procedures only try to recover the structure of canonical sentences but not their linear order.

John bought books and Mary **bought\({}^{\prime}\)** flowers

If a clause contains a more complex gap, the enhanced representation contains copies for all content words that are required to attach the remnants.

The motivation behind this analysis is that the semantically empty markers _to_ are not needed for interpreting the sentence and minimizing the number of copy nodes leads to less complex graphs.

Finally, if a core argument was elided along with the predicate, we introduce additional dependencies between the copy nodes and the shared arguments, as for example, the open clausal complement (xcomp) dependency between the copy node and _Senator_ in the following example.

The rationale for not copying all arguments is again to keep the graph simple, while still encoding all relations between content words. Arguments can be arbitrarily complex and it seems misguided to copy entire subtrees of arguments which, e.g., could contain multiple adverbial clauses. Note that linking to existing nodes would not work in the case of verb clusters because they do not satisfy the subtree constraint.

## 3 Methods

### Composite relations

Our first method adapts one of the procedures by Seeker et al. (2012), which represents gaps in dependency trees by attaching dependents of an elided predicate with composite relations. These relations represent the dependency path that would have existed if nothing had been elided. For example, in the following sentence, the verb _bought_, which would have been attached to the head of the first conjunct with a conj relation, was elided from the second conjunct and hence all nodes that would have depended on the elided verb, are attached to the first conjunct using a composite relation consisting of conj and the type of argument.

John bought books and Mary flowers

The major advantage of this approach is that the dependency tree contains information about the types of arguments and so it should be straightforward to turn dependency trees of this form into enhanced UD graphs. For most dependency trees, one can obtain the enhanced UD graph by splitting the composite relations into its atomic parts and inserting copy nodes at the splitting points.4

Footnote 4: Note that this representation does not indicate conjunct boundaries, and for sentences with multiple gapped conjuncts, it is thus unclear how many copy nodes are required.

At the same time, this approach comes with the drawback of drastically increasing the label space. For sentences with more complex gaps as in (5), one has to use composite relations that consist of more than two atomic relations and theoretically, the number of composite relations is unbounded:

...and Mary a play

### Orphan procedure

Our second method also uses a two-step approach to resolve gaps, but compared to the previous method, it puts less work on the parser. We first parse sentences to the basic UD v2 representation, which analyzes gapping constructions as follows. One remnant is promoted to be the head of the clause and all other remnants are attached to the promoted phrase. For example, in this sentence, the subject of the second clause, _Mary_, is the head of the clause and the other remnant, _flowers_, is attached to _Mary_ with the special orphan relation: 

This analysis can also be used for more complex gaps, as in the example with a gap that consists of a chain of non-finite embedded verbs in (5).

When parsing to this representation, the parser only has to identify that there is a gap but does not have to recover the elided material or determine the type of remnants. As a second step, we use an unsupervised procedure to determine which nodes to copy and how and where to attach the remnants. In developing this procedure, we made use of the fact that in the vast majority of cases, all arguments and modifiers that are expressed in gapped conjunct are also expressed in the full conjunct. The problem of determining which nodes to copy and which relations to use can thus be reduced to the problem of aligning arguments in the gapped conjunct to arguments in the full conjunct. We apply the following procedure to all sentences that contain at least one orphan relation.

1. Create a list \(F\) of arguments of the head of the full conjunct by considering all core argument dependents of the conjunct's head as well as clausal and nominal non-core dependents, and adverbial modifiers.
2. Create a list \(G\) of arguments in the gapped conjunct that contains the head of the gapped conjunct and all its orphan dependents.
3. Find the highest-scoring monotonic alignment of arguments in \(G\) to arguments in \(F\).
4. Copy the head of the full conjunct and attach the copy node \(c\) to the head of the full conjunct with the original relation of the head of the gapped conjunct (usually conj).
5. For each argument \(g\in G\) that has been aligned to \(f\in F\), attach \(g\) to \(c\) with the same relation as the parent relation of \(f\), e.g., if \(f\) is attached to the head of the full conjunct with an nsubj relation, also attach \(g\) to \(c\) with an nsubj relation. Attach arguments \(g^{\prime}\in G\) that were not aligned to any token in \(F\) to \(c\) using the general dep relation.
6. For each copy node \(c\), add dependencies to all core arguments of the original node which do not have a corresponding remnant in the gapped clause. For example, if the full conjunct contains a subject, an object, and an oblique modifier but the clause with the gap, only a subject and an oblique modifier, add an object dependency between the copy node and the object in the full conjunct.

A crucial step is the third step, determining the highest-scoring alignment. This can be done straightforwardly with the sequence alignment algorithm by Needleman and Wunsch (1970) if one defines a similarity function \(sim(g,f)\) that returns a similarity score between the arguments \(g\) and \(f\). We defined \(sim\) based on the intuitions that often, parallel arguments are of the same syntactic category, that they are introduced by the same function words (e.g., the same preposition), and that they are closely related in meaning. The first intuition can be captured by penalizing mismatching POS tags, and the other two by computing the distance between argument embeddings. We compute these embeddings by averaging over the 100-dim. pretrained GloVe Pennington et al. (2014) embeddings for each token in the argument. Given the POS tags \(t_{g}\) and \(t_{f}\) and the argument embeddings \(v_{g}\) and \(v_{f}\), \(sim\) is defined as follows.5

Footnote 5: As suggested by one of the reviewers, we also ran a post-hoc experiment with a simpler similarity score function without the embedding distance term, which only takes into account whether the POS tags match. We found that quantitatively, the embeddings do not lead to significant better scores on the test set according to our metrics but qualitatively, they lead to better results for the examples with verb cluster gaps.

\[sim(g,f)=-\|v_{g}-v_{f}\|_{2}+\mathbbm{1}\left[t_{g}=t_{f}\right]\\ \times pos\_mismatch\_penalty\]

We set \(pos\_mismatch\_penalty\), a parameter that penalizes mismatching POS tags, to \(-2\).6

Footnote 6: We optimized this parameter on the training set by trying integer values from \(-1\) to \(-15\).

This procedure can be used for almost all sentences with gapping constructions. However, if parts of an argument were elided along with the main predicate, it can become necessary to copy multiple nodes. We therefore consider the alignment not only between complete arguments in the full clause and the gapped clause but also between partial arguments in the full clause and the complete arguments in the gapped clause. For example, for the sentence "_Mary wants to write a play and Sue a book_" the complete arguments of the full clause are {_Mary_, _to write a play_} and the arguments of the gapped clause are {_Sue_, _a book_}. In this case, we also consider the partial arguments {_Mary_, _a play_} and if the 

conjunct align better to the partial arguments, we use this alignment. However, now that the token _write_ is part of the dependency path between _want_ and _play_, we also have to make a copy of _write_ to reconstruct the UD graph of the gapped clause.

## 4 Experiments

Both methods rely on a dependency parser followed by a post-processing step. We evaluated the individual steps and the end-to-end performance.

### Data

We used the UD English Web Treebank v2.1 (henceforth ewt; Silveira et al., 2014; Nivre et al., 2017) for training and evaluating parsers. As the treebank is relatively small and therefore only contains very few sentences with gapping, we also extracted gapping constructions from the WSJ and Brown portions of the PTB Marcus et al. (1993) and the GENIA corpus Ohta et al. (2002). Further, we copied sentences from the Wikipedia page on gapping7 and from published papers on gapping. The sentences in the ewt already contain annotations with the orphan relation and copy nodes for the enhanced representation, and we manually added both of these annotations for the remaining examples. The composite relations can be automatically obtained from the enhanced representation by removing the copy nodes and concatenating the dependency labels, which we did to build the training and test corpus for the composite relation procedure. Table 1 shows properties of the data splits of the original treebank, the additional sentences with gapping, and their combination; Table 2 shows the number of sentences in our corpus for each of the gap types.

Footnote 7: [https://en.wikipedia.org/wiki/Gapping](https://en.wikipedia.org/wiki/Gapping), accessed on Aug 24, 2017.

### Parsing experiments

ParserWe used the parser by Dozat and Manning (2017) for parsing to the two different intermediate dependency representations. This parser is a graph-based parser McDonald et al. (2005) that uses a biLSTM to compute token representations and then uses a multi-layer perceptron with biaffine attention to compute arc and label scores.

SetupWe trained the parser on the combined training corpus with gold tokenization, and predicted fine-grained and universal part-of-speech tags, for which we used the tagger by Dozat et al. (2017). We trained the tagger on the combined training corpus. As pre-trained embeddings, we used the word2vec Mikolov et al. (2013) embeddings that were provided for the CoNLL 2017 Shared Task Zeman et al. (2017), and we used the same hyperparameters as Dozat et al. (2017).

EvaluationWe evaluated the parseability of the two dependency representations using labeled and unlabeled attachment scores (LAS and UAS). Further, to specifically evaluate how well parsers are able to parse gapping constructions according to the two annotation schemes, we also computed the LAS and UAS just for the head tokens of remnants (\({\text{LAS}}_{g}\) and \({\text{UAS}}_{g}\)). For all our metrics, we excluded punctuation tokens. To determine sta

\begin{table}
\begin{tabular}{l c c c c c c c c c}  & \multicolumn{2}{c}{ewt} & \multicolumn{6}{c}{gapping} & \multicolumn{3}{c}{ewt + gapping (combined)} \\  & **Train** & **Dev** & **Test** & **Train** & **Dev** & **Test** & **Train** & **Dev** & **Test** \\ \hline sentences & 12,543 & 2,002 & 2,077 & 164 & 79 & 79 & 12,707 & 2,081 & 2,156 \\ \hline tokens & 204,585 & 25,148 & 25,096 & 4,698 & 2,383 & 2,175 & 209,283 & 27,531 & 27,271 \\ \hline sentences with & 21 & 1 & 1 & 164 & 79 & 79 & 185 & 80 & 80 \\ gapping & 0.15\% & 0.05\% & 0.05\% & 100\% & 100\% & 100\% & 1.46\% & 3.84\% & 3.71\% \\ \hline copy nodes & 22 & 2 & 1 & 201 & 96 & 102 & 223 & 98 & 103 \\ \hline unique composite relations & 16 & 6 & 2 & 41 & 29 & 31 & 46 & 29 & 31 \\ \end{tabular}
\end{table}
Table 1: Treebank statistics. The _copy nodes_ row lists the number of copy nodes and the _unique compositions relations_ row lists the number of unique composite relations in the treebanks annotated according to the composite analysis. The percentages are relative to the total number of sentences.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}} \multicolumn{2}{p{113.8pt}}{**Gap type**} & \multicolumn{1}{c}{**Frequency**} \\ \hline Single predicate & 172 \\ Contiguous predicate-argument & 140 \\ Non-contiguous predicate-argument & 9 \\ Verb cluster & 24 \\ \end{tabular}
\end{table}
Table 2: Distribution of gap types in our corpus. The classification is according to the four types of gaps that we discussed in Section 2.1.

 

tistical significance of pairwise comparisons, we performed two-tailed approximate randomization tests (Noreen, 1989; Yeh, 2000) with an adapted version of the sigf package (Pado, 2006).

ResultsTable 3 shows the overall parsing results on the development and test sets of the two treebanks. There was no significant difference between the parser that was trained on the UD representation (orphan) and the parser trained on the composite representation (composite) when tested on the EWT data sets, which is not surprising considering that there is just one sentence with gapping each in the development and the test split. When evaluated on the gapping datasets, the orphan parser performs significantly better (\(p<0.01\)) in terms of labeled attachment score, which suggests that the parser trained on the composite representation is indeed struggling with the greatly increased label space. This is further confirmed by the attachment scores of the head tokens of remnants (Table 4). The labeled attachment score of remnants is significantly higher for the orphan parser than for the composite parser. Further, the unlabeled attachment score on the test set is also higher for the orphan parser, which suggests that the composite parser is sometimes struggling with finding the right attachment for the multiple long-distance composite dependencies.

### Recovery experiments

Our second set of experiments concerns the recovery of the elided material and the reattachment of the orphans. We conducted two experiments: an oracle experiment that used gold standard dependency trees and an end-to-end experiment that used the output of the parser as input. For all experiments, we used the combined treebank.

EvaluationHere, we evaluated dependency graphs and therefore used the labeled and unlabeled precision and recall metrics. However, as our two procedures are only changing the attachment of orphans, we only computed these metrics for copy nodes and their dependents. Further, we excluded punctuation and coordinating conjunctions as their attachment is usually trivial and including them would inflate scores. Lastly, we computed the sentence-level accuracy for all sentences with gapping. For this metric, we considered a sentence to be correct if all copy nodes and their dependents of a sentence were attached to the correct head with the correct label.

Oracle resultsThe top part of Table 5 shows the results for the oracle experiment. Both methods are able to reconstruct the elided material and the canonical clause structure from gold dependency trees with high accuracy. This was expected for the composite procedure, which can make use of the composite relations in the dependency trees, but less so for the orphan procedure which has to recover the structure and the types of relations. The two methods work equally well in terms of all metrics except for the sentence-level accuracy, which is significantly higher for the composite procedure. This difference is caused by a difference in the types of mistakes. All errors of the composite procedure are of a structural nature and stem from copying the wrong number of nodes while the dependency labels are always correct because they are part of the dependency tree. The majority of errors of the orphan procedure stem from incorrect dependency labels, and these mistakes are scattered across more examples, which leads to the lower sentence-level accuracy.

End-to-end resultsThe middle part of Table 5 shows the results for the end-to-end experiment. The performance of both methods is considerably lower than in the oracle experiment, which is pri

\begin{table}
\begin{tabular}{l c c c c}  & \multicolumn{2}{c}{**Development**} & \multicolumn{2}{c}{**Test**} \\  & UAS\({}_{g}\) & LAS\({}_{g}\) & UAS\({}_{g}\) & LAS\({}_{g}\) \\ \hline orphan & **72.36** & **64.73*** & **72.56*** & **65.79*** \\ composite & 68.36 & 49.45 & 62.41 & 46.24 \\ \end{tabular}
\end{table}
Table 4: Labeled (LAS\({}_{g}\)) and unlabeled attachment score (UAS\({}_{g}\)) of head tokens of remnants for parsers trained and evaluated on the UD representation (orphan) and the composite relations representation (composite) on the development and test sets of the combined treebank. Results that differ significantly are marked with * (\(p<0.05\)) or *** (\(p<0.001\)).

\begin{table}
\begin{tabular}{l l r r r r}  & & \multicolumn{2}{c}{EWT} & \multicolumn{2}{c}{GAPPING} \\  & & UAS & LAS & UAS & LAS \\ \hline \multirow{2}{*}{**Dev**} & orphan & **90.57** & 87.32 & **89.34** & **85.69**** \\  & composite & 90.46 & **87.37** & 88.86 & 84.21 \\ \hline \multirow{2}{*}{**Test**} & orphan & 90.42 & 87.06 & **87.44** & **83.97**** \\  & composite & **90.54** & **87.33** & 86.51 & 81.69 \\ \end{tabular}
\end{table}
Table 3: Labeled (LAS) and unlabeled attachment score (UAS) of parsers trained and evaluated on the UD representation (orphan) and the composite relations representation (composite) on the development and test sets of the ewt and the gapping treebank. ** indicates that results differ significantly at \(p<0.01\).

 

marily driven by the much lower recall. Both methods assume that the parser detects the existence of a gap and if the parser fails to do so, neither method attempts to reconstruct the elided material. In general, precision tends to be a bit higher for the orphan procedure whereas recall tends to be a bit higher for the composite method but overall and in terms of sentence-level accuracy both methods seem to perform equally well.

Error analysisFor both methods, the primary issue is low recall, which is a result of parsing errors. When the parser correctly predicts the orphan relation, the main sources of error for the orphan procedure are missing correspondents for remnants (e.g., _[for good]_ has no correspondent in _They had left the company, many for good_) or that the types of argument of the remnant and its correspondent differ (e.g., in _She was convicted of selling unregistered securities in Florida and of unlawful phone calls in Ohio_, _[of selling unregistered securities]_ is an adverbial clause whereas _[of unlawful phone calls]_ is an oblique modifier).

Apart from the cases where the composite procedure leads to an incorrect structure, the remaining errors are all caused by the parser predicting the wrong composite relation.

### Comparison to Kummerfeld and Klein

Kummerfeld and Klein (henceforth K&K; 2017) recently proposed a one-endpoint-crossing graph parser that is able to directly parse to PTB-style trees with traces. They also briefly discuss gapping constructions and their parser tries to output the co-indexing that is used for gapping constructions in the PTB. The EWT and all the sentences that we took from the WSJ, Brown, and GENIA treebanks already come with constituency tree annotations, and we manually annotated the remaining sentences according to the PTB guidelines (Bies et al., 1995). This allowed us to train the K&K parser with exactly the same set of sentences that we used in our previous experiments. As this parser outputs constituency trees, we could not compute dependency graph metrics for this method. For the sentence-level accuracy, we considered an example to be correct if a) each argument in the gapped conjunct was the child of a single constituent node, which in return was the sibling of the full clause/verb phrase, and b) the co-indexing of each argument in the gapped conjunct was correct. For example, the following bracketing would be considered correct despite the incorrect internal structure of the first conjunct:

\begin{table}
\begin{tabular}{l l c c c c c c c c c c}  & \multicolumn{6}{c}{**Development**} & \multicolumn{4}{c}{**Test**} \\ \hline  & & UP & UR & LP & LR & SAcc. & UP & UR & LP & LR & SAcc. \\ \hline \multirow{3}{*}{oracle} & composite & 91.32 & 88.20 & **91.32** & **88.20** & **91.14*** & 90.71 & 86.81 & **90.71** & 86.81 & **86.08*** \\  & orphan & **94.08** & **93.79** & 87.54 & 87.27 & 72.15 & **92.02** & **92.02** & 87.12 & **87.12** & 72.15 \\ \hline \hline end-to-end & composite & 70.48 & **49.69*** & **65.64** & **46.27*** & **31.65** & 67.39 & **47.55** & 61.74 & **43.56** & 31.65 \\  & orphan & **71.73** & 42.55 & 65.45 & 38.82 & 30.38 & **78.92**** & 44.78 & **68.11** & 38.65 & **34.18** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Labeled and unlabeled precision and recall as well as sentence-level accuracy of the two gapping reconstructions methods and the K&K parser on the development and test set of the combined treebank. Results that differ significantly from the other result within the same section are marked with * (\(p<0.05\)) or ** (\(p<0.01\)).

 

our method does not make use of lexical information, and on the other hand, this is because we developed our method on top of the UD annotation scheme, which has already been applied to many languages and for which many treebanks exist.

Footnote 10: [https://www.cds.org/en/lat/](https://www.cds.org/en/lat/)

Currently, all treebanks but the English one lack copy nodes for gapping constructions and many of them incorrectly use the orphan relation Droganova and Zeman (2017) and therefore we could not evaluate our method on a large variety of languages. In order to demonstrate that our method can be applied to other languages, we therefore did a case study on the Swedish UD treebank. The Swedish UD treebank is an automatic conversion from a section of the Talbanken Einarsson (1976) with extensive manual corrections. While the treebank is overall of high quality, we noticed conversion errors that led to incorrect uses of the orphan relation in 11 of the 29 sentences with orphan relations, which we excluded from our evaluation. We applied our gapping resolution procedure without any modifications to the remaining 18 sentences. We used the Swedish word2vec embeddings that were prepared for the CoNLL 2017 Shared Task. Our method correctly predicts the insertion of 29 copy nodes and is able to predict the correct structure of the enhanced representation in all cases, including complex ones with elided verb clusters such as the example in Figure 2. It also predicts the correct dependency label for 108/110 relations, leading to a labeled precision and labeled recall of 98.18%, which are both higher than the English numbers despite the fact that we optimized our procedure for English. The main reason for the higher performance seems to be that many of the Swedish examples come from informational texts from public organizations, which are more likely to be written to be clear and unambiguous. Further, the Swedish data does not contain challenging examples from the linguistic literature.

As Swedish is a Germanic language like English and thus shares many structural properties, we cannot conclude that our method is applicable to any language based on just this experiment. However, given that our method does not rely on language-specific structural patterns, we expect it to work well for a wide range of languages.

## 6 Related work

Gapping constructions have been little studied in NLP, but several approaches (e.g., Dukes and Habash 2011; Simko and Vincze 2017) parse to dependency trees with empty nodes. Seeker et al. (2012) compared three ways of parsing with empty heads: adding a transition that inserts empty nodes, using composite relation labels for nodes that depend on an elided node, and pre-inserting empties before parsing. These papers all focus on recovering nodes for elided function words such as auxiliaries; none of them attempt to recover and resolve the content word elisions of gapping. Ficler and Goldberg (2016) modified PTB annotations of argument-cluster coordinations (ACCs), i.e., gapping constructions with two post-verbal orphan phrases, which make up a subset of the gapping constructions in the PTB. While the modified annotation style leads to higher parsing accuracy of ACCs, it is specific to ACCs and does not generalize to other gapping constructions. Moreover, they did not reconstruct gapped ACC clauses. Traditional grammar-based chart parsers Kay (1980); Klein and Manning (2001) did handle empty nodes and so could in principle provide a parse of gapping sentences though additional mechanisms would be needed for reconstruction. In practice, though, dealing with gapping in a grammar-based framework is not straightforward and can lead to a combinatorial explosion that slows down parsing in general, as has been noted for the English Resource Grammar Flickinger (2017), p.c.) and for an HPSG implementation for Norwegian Haugereid (2017). The grammar-based parser built with augmented transition networks Woods (1970) provided an extension in the form of the SYSCONJ operation Woods (1973) to parse some gapping constructions, but also this approach lacked explicit reconstruction mechanisms and provided only limited coverage.

There also exists a long line of work on post-processing surface-syntax constituency trees to recover traces in the PTB Johnson (2002); Levy and Manning (2004); Campbell (2004); Gabbard et al. (2006), pre-processing sentences such that they contain tokens for traces before parsing Dienes and Dubey (2003), or directly parsing sentences to either PTB-style trees with empty elements or pre-processed trees that can be deterministically converted to PTB-style trees Collins ( 

1997; Dienes and Dubey, 2003a; Schmid, 2006; Cai et al., 2011; Hayashi and Nagata, 2016; Kato and Matsubara, 2016; Kummerfeld and Klein, 2017). However, all of these works are primarily concerned with recovering traces for phenomena such as Wh-movement or control and raising constructions and, with the exception of Kummerfeld and Klein (2017), none of these works attempt to output the co-indexing that is used for analyzing gapping constructions. And again, none of these works try to reconstruct elided material.

Lastly, several methods have been proposed for resolving other forms of ellipsis, including VP ellipsis Hardt (1997); Nielsen (2004); Lappin (2005); McShane and Babkin (2016) and sluicing Anand and Hardt (2016) but none of these methods consider gapping constructions.

## 7 Conclusion

We presented two methods to recover elided predicates in sentences with gapping. Our experiments suggest that both methods work equally well in a realistic end-to-end setting. While in general, recall is still low, the oracle experiments suggest that both methods can recover elided predicates from correct dependency trees, which suggests that as parsers become more and more accurate, the gap recovery accuracy should also increase.

We also demonstrated that our method can be used to automatically add the enhanced UD representation to UD treebanks in other languages than English. Apart from being useful in a parsing pipeline, we therefore also expect our method to be useful for building enhanced UD treebanks.

## Reproducibility

All data, pre-trained models, system outputs as well as a package for running the enhancement procedure are available from [https://github.com/sebschu/naacl-gapping](https://github.com/sebschu/naacl-gapping).

## Acknowledgments

We thank the anonymous reviewers for their thoughtful feedback. Also thanks to Vera Gribanova and Boris Harizanov for continuous feedback throughout this project, and to Matthew Lamm for help with annotating the data. This work was supported in part by gifts from Google, Inc. and IPSoft, Inc. The first author is also supported by a Goodman Family Graduate Fellowship.

## References

* Anand and Hardt (2016) Pranav Anand and Daniel Hardt. 2016. Antecedent selection for sluicing: Structure and content. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016)_. pages 1234-1243. [https://doi.org/10.18653/v1/D16-1131](https://doi.org/10.18653/v1/D16-1131).
* Bejcek et al. (2013) Eduard Bejcek, Eva Haji\({\rm\breve{c}}\)ova, Jan Hajic, Pavlina JinovA, Vaclava Kettnerova, Veronika Kolafrova, Marie Mikulova, Jiri Mirovsky, Anna Nedoluzhko, Jarmila Panevova, Lucie Polakova, Magda Sevcikova, Jan Stepanek, and Sarkka Zikanova. 2013. Prague Dependency Treebank 3.0. LIMIT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University. [http://hdl.handle.net/11858/00-097C-0000-0023-1AAF-3](http://hdl.handle.net/11858/00-097C-0000-0023-1AAF-3).
* Bies et al. (1995) Ann Bies, Mark Ferguson, Karen Katz, Robert MacIntyre, Victoria Tredinnick, Grace Kim, Mary Ann Marcinkiewicz, and Britta Schasberger. 1995. Bracketing guidelines for Treebank II style Penn Treebank project. Technical report, University of Pennsylvania.
* Cai et al. (2011) Shu Cai, David Chiang, and Yoay Goldberg. 2011. Language-independent parsing with empty elements. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2011)_. pages 212-216. [http://www.aclweb.org/anthology/P11-2037](http://www.aclweb.org/anthology/P11-2037).
* Campbell (2004) Richard Campbell. 2004. Using linguistic principles to recover empty categories. In _Proceedings of the

Figure 2: Dependency graph for part of the sentence sv-ud-train-1102 as output by the orphan procedure. The system correctly predicts the copy nodes for the matrix and the embedded verb, and correctly attaches the arguments to the copy nodes.

 

42nd Annual Meeting on Association for Computational Linguistics (ACL 2004)_. [https://doi.org/10.3115/1218955.1219037](https://doi.org/10.3115/1218955.1219037).
* Collins [1997] Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In _Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL 1997)_. pages 16-23. [https://doi.org/10.3115/976909.979620](https://doi.org/10.3115/976909.979620).
* Coppock [2001] Elizabeth Coppock. 2001. Gapping: In defense of deletion. In Mary Andronis, Christopher Ball, Heidi Elston, and Sylvain Neuvel, editors, _Papers from the 37th Meeting of the Chicago Linguistic Society_. Chicago Linguistic Society, Chicago, pages 133-148.
* Dienes and Dubey [2003a] Peter Dienes and Amit Dubey. 2003a. Antecedent recovery: Experiments with a trace tagger. In _Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing (EMNLP 2003)_. pages 33-40. [http://www.aclweb.org/anthology/W03-1005](http://www.aclweb.org/anthology/W03-1005).
* Dienes and Dubey [2003b] Peter Dienes and Amit Dubey. 2003b. Deep syntactic processing by combining shallow methods. In _Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (ACL 2003)_. pages 431-438. [https://doi.org/10.3115/1075096.1075151](https://doi.org/10.3115/1075096.1075151).
* Dozat and Manning [2017] Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In _Proceedings of the 5th International Conference on Learning Representations (ICLR 2017)_. pages 1-8. [https://openreview.net/pdf?id=Hk95PK91e](https://openreview.net/pdf?id=Hk95PK91e).
* Dozat et al. [2017] Timothy Dozat, Peng Qi, and Christopher D. Manning. 2017. Stanford's graph-based neural dependency parser at the CoNLL 2017 Shared Task. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. pages 20-30. [https://doi.org/10.18653/vlK17-3002](https://doi.org/10.18653/vlK17-3002).
* Droganova and Zeman [2017] Kira Droganova and Daniel Zeman. 2017. Elliptic constructions: Spotting patterns in UD treebanks. In _Proceedings of the NoDaLiDA 2017 Workshop on Universal Dependencies (UDW 2017)_. pages 48-57. [http://www.aclweb.org/anthology/W17-0406](http://www.aclweb.org/anthology/W17-0406).
* Dukes and Habash [2011] Kais Dukes and Nizar Habash. 2011. One-step statistical parsing of hybrid dependency-constituency syntactic representations. In _Proceedings of the 12th International Conference on Parsing Technologies_. pages 92-103. [http://www.aclweb.org/anthology/Wl1-2912](http://www.aclweb.org/anthology/Wl1-2912).
* Einarsson [1976] Jan Einarsson. 1976. Talbankens skriftsprakskonordans. Institutionen for nordsika sprak, Lunds universitet.
* Farudi [2013] Annahita Farudi. 2013. _Gapping in Farsi: A Crosslinguistic Investigation_. Ph.D. thesis, University of Massachusetts Amherst. [http://scholarworks.umass.edu/dissertations/AAI3556244](http://scholarworks.umass.edu/dissertations/AAI3556244).
* Ficler and Goldberg [2016] Jessica Ficher and Yoav Goldberg. 2016. Improved parsing for argument-clusters coordination. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)_. pages 72-76. [https://doi.org/10.18653/vlP16-2012](https://doi.org/10.18653/vlP16-2012).
* Gabbard et al. [2006] Ryan Gabbard, Mitchell Marcus, and Seth Kulick. 2006. Fully parsing the Penn Treebank. In _Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL (NAACL 2006)_. pages 184-191. [https://doi.org/10.3115/1220835.1220859](https://doi.org/10.3115/1220835.1220859).
* Hardt [1997] Daniel Hardt. 1997. An empirical approach to VP ellipsis. _Computational Linguistics_ 23(4):525-541. [http://www.aclweb.org/anthology/J97-4002](http://www.aclweb.org/anthology/J97-4002).
* Haugereid [2017] Petter Haugereid. 2017. An incremental approach to gapping and conjunction reduction. In _Proceedings of the 24th International Conference on Head-Driven Phrase Structure Grammar_. pages 179-198. [http://cslipublications.stanford.edu/HPSG/2017/hpsg2017-haugereid.pdf](http://cslipublications.stanford.edu/HPSG/2017/hpsg2017-haugereid.pdf).
* Hayashi and Nagata [2016] Katsuhiko Hayashi and Masaaki Nagata. 2016. Empty element recovery by spinal parser operations. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)_. pages 95-100. [https://doi.org/10.18653/vlV1/P16-2016](https://doi.org/10.18653/vlV1/P16-2016).
* Hokeenmaier and Steedman [2007] Julia Hokeenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank. _Computational Linguistics_ 33(3):355-396. [http://www.aclweb.org/anthology/J07-3004](http://www.aclweb.org/anthology/J07-3004).
* Jackendoff [1971] Ray S. Jackendoff. 1971. Gapping and related rules. _Linguistic Inquiry_ 2(1):21-35.
* Johnson [2014] Kyle Johnson. 2014. Gapping. Unpublished manuscript, University of Massachusetts at Amherst. [http://people.umass.edu/kbj/homepage/Content/gapping.pdf](http://people.umass.edu/kbj/homepage/Content/gapping.pdf).
* Johnson [2002] Mark Johnson. 2002. A simple pattern-matching algorithm for recovering empty nodes and their antecedents. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002)_. pages 136-143. [https://doi.org/10.3115/1073083.1073107](https://doi.org/10.3115/1073083.1073107).
* Kato and Matsubara [2016] Yoshihide Kato and Shigeki Matsubara. 2016. Transition-based left-corner parsing for identifying PTB-style nonlocal dependencies. In _Proceedings of the 54th Annual Meeting of 

the Association for Computational Linguistics (ACL 2016)_. pages 930-940. [https://doi.org/10.18653/v1/P16-1088](https://doi.org/10.18653/v1/P16-1088).
* Kay [1980] Martin Kay. 1980. Algorithm schemata and data structures in syntactic processing. Technical report, Xerox PARC.
* Klein and Manning [2001] Dan Klein and Christopher Manning. 2001. Parsing with treebank grammars: empirical bounds, theoretical models, and the structure of the Penn Treebank. In _Proceedings of 39th Annual Meeting of the Association for Computational Linguistics (ACL 2001)_. pages 338-345. [https://doi.org/10.3115/1073012.1073056](https://doi.org/10.3115/1073012.1073056).
* Kubota and Levine [2016] Yusuke Kubota and Robert Levine. 2016. Gapping as hypothetical reasoning. _Natural Language and Linguistic Theory_ 34(1):107-156. [https://doi.org/10.1007/s11049-015-9298-4](https://doi.org/10.1007/s11049-015-9298-4).
* Kummerfeld and Klein [2017] Jonathan K. Kummerfeld and Dan Klein. 2017. Parsing with traces: An \(O(n^{4})\) algorithm and a structural representation. _Transactions of the Association for Computational Linguistics_ 5:441-454. [https://www.transacl.org/ojs/index.php/tacl/article/view/1170](https://www.transacl.org/ojs/index.php/tacl/article/view/1170).
* Lappin [2005] Shalom Lappin. 2005. A sequenced model of anaphora and ellipsis resolution. In Antonio Branco, Tony McEnery, and Ruslan Mitkov, editors, _Anaphora Processing: Linguistic, Cognitive and Computational Modelling_, John Benjamins Publishing, Amsterdam, pages 3-16. [https://doi.org/10.1075/cilt.263.031ap](https://doi.org/10.1075/cilt.263.031ap).
* Levy and Manning [2004] Roger Levy and Christopher D. Manning. 2004. Deep dependencies from context-free statistical parsers. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics (ACL 2004)_. pages 327-334. [https://doi.org/10.3115/1218955.1218997](https://doi.org/10.3115/1218955.1218997).
* Marcus et al. [1993] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. _Computational Linguistics_ 19(2):313-330. [http://www.aclweb.org/anthology/J93-2004](http://www.aclweb.org/anthology/J93-2004).
* McDonald et al. [2005] Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-projective dependency parsing using spanning tree algorithms. In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT-EMNLP 2005)_. pages 523-530. [https://doi.org/10.3115/1220575.1220641](https://doi.org/10.3115/1220575.1220641).
* McShane and Babkin [2016] Marjorie McShane and Petr Babkin. 2016. Detection and resolution of verb phrase ellipsis. _Linguistic Issues in Language Technology (LiLT)_ 13(1):1-34.
* Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems 26 (NIPS 2013)_. pages 3111-3119.
* Needleman and Wunsch [1970] Saul B. Needleman and Christian D. Wunsch. 1970. A general method applicable to the search for similarities in the amino acid sequence of two proteins. _Journal of Molecular Biology_ 48(3):443-453. [https://doi.org/10.1016/0022-2836](https://doi.org/10.1016/0022-2836)(70)90057-4.
* Nielsen [2004] Leif Arda Nielsen. 2004. Verb phrase ellipsis detection using automatically parsed text. In _Proceedings of the 20th International Conference on Computational Linguistics (COLING 2004)_. 1, pages 1093-1099. [https://doi.org/10.3115/1220355.1220512](https://doi.org/10.3115/1220355.1220512).
* Nivre et al. [2016] Joakim Nivre, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Attuxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, John Bauer, Kepa Bengootexa, Riyaz Ahmad Bhat, Eckhard Bick, Victoria Bobicex, Carl Borstell, Cristina Bosco, Goose Bouma, Sam Bowman, Alposabk Burchardt, Marie Candito, Gauthier Caron, Gulsen Cebiroglu Figgit, Giuseppe G. A. Celano, Savas Cetin, Fabricio Chalub, Jinho Choi, Silvie Cinkova, Cagri Coltekin, Mirian Connor, Elizabeth Davidson, Mariet Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Feter Dirix, Kaja Dobrovolle, Timothy Dorat, Kira Droganova, Puneuel Dwivedi, Marhaba Eli, Ali Elkahky, Tomaz Erjavec, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Claudia Freitas, Katarina Galgodsova, Daniel Galbraith, Marcos Garcia, Moa Gardenfors, Kim Gerdes, Filip Gunter, Lakes Goenaga, Koldo Gojeenda, Memdu Gokirmka, Yoav Goldberg, Xavier Gomez Guinovart, Berta Gonzales Saavedra, Mariias Grioni, Normunds Gurzitis, Bruno Guillaume, Nizar Habash, Jan Hajic, Jan Hajic Jr., Linh Ha My, Kim Harris, Dag Haug, Barbora Hladka, Jaroslava Hlavacova, Florinel Hociung, Peter Hohle, Radu Ion, Elenza Irinia, Tomas Jelinek, Anders Johannsen, Fredrik Jorgensen, Huner Kaskarla, Hiroshi Kanayama, Jenna Kanerva, Tolga Kayadelen, Vaclava Kettnerova, Jesse Kirchner, Natalia Kostsyba, Simon Krek, Veronika Liappala, Lorenzo Lambertino, Tatiana Lando, John Lee, Phutong Le Hong, Alessandro Lenci, Saran Lerprodatt, Herman Leung, Cheuk Ying Li, Josie Li, Keying Li, Nikola Ljubesic, Olga Loginova, Olga Lyshesvkaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl, Christopher D. Manning, Catalina Maranduc, David Marecek, Katrin Marheineecke, Hector Martinez Alonso, Andre Martins, Jan Masek, Yuji Matsumoto, Ryan McDonald, Gustavo Mendonca, Niko Miekka, Anna Missila, Catalina Mititelu, Yusuke Miyao, Simonetta Montemagni, Amir More, Laura Moreno Romero, Shinsuke Mori, Bohdan Mosalevskyki, Kadri Muischnek, Kaili Muurisep, Pinkey Nainwani, Anna Nedoluzhko, Gunta Negore-Berzkalne, Lutng Nguyen Thi, Huyen Nguyan Thi Minh, Vitaly Nikolaev, Hanna Nurmi, Stina Ojala, Petya Osentova, Robert Ostling, Lilja Ovrelid, Elena Pascual, 

Marco Passarotti, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Emily Pitler, Barbara Plank, Martin Popel, Lauma Pretktalnina, Prokopis Prokopidis, Tiana Poukalainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy, Taraka Rama, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Larissa Rinaldi, Laura Rituma, Mykhailo Romanenko, Rudolf Rosa, Davide Revati, Benoit Sagot, Shadi Saleh, Tanja Samardzic, Manuela Sanguinetti, Baiba Saullette, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atsuko Shimada, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, Maria Simkova, Kiril Simov, Aaron Smith, Antonio Stella, Milan Straka, Jana Strandova, Alane Suhr, Umut Sulubacak, Zsolt Szanto, Dima Tai, Takaki Tanaka, Trond Trosterud, Anna Truhkina, Retu Tsarfaty, Francis Tyers, Sumire Uematsu, Zdenka Uresova, Larratitz Uria, Hans Uszkoreit, Sowmya Vajala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Lars Wallim, Jonathan North Washington, Mats Wirren, Tak-sum Wong, Zhuoran Yu, Zdenek Zabokritsky, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. 2017. Universal Dependencies 2.1. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University. [http://hdl.handle.net/11234/1-2515](http://hdl.handle.net/11234/1-2515).
* Nivre et al. [2016] Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampa Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_. pages 1659-1666. [http://www.lrec-conf.org/proceedings/lrec2016/pdf/348_Paper.pdf](http://www.lrec-conf.org/proceedings/lrec2016/pdf/348_Paper.pdf).
* Noreen [1989] Eric W. Noreen. 1989. _Computer-Intensive Methods for Testing Hypotheses_. John Wiley & Sons, New York, NY.
* Oepen et al. [2015] Stephan Oepen, Marco Kuhlmann, Yusuke Miyao, Daniel Zeman, Silvie Cinkova, Dan Flickinger, Jan Hajic, and Zdenka Uresova. 2015. SemEval 2015 Task 18: Broad-coverage semantic dependency parsing. In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_. pages 915-926. [https://doi.org/10.18653/v1/S15-2153](https://doi.org/10.18653/v1/S15-2153).
* Ohta et al. [2002] Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. 2002. The GENIA corpus: An annotated research abstract corpus in molecular biology domain. _Proceedings of the Second International Conference on Human Language Technology Research (HLT 2002)_ pages 82-86. [https://doi.org/10.3115/1289189.1289260](https://doi.org/10.3115/1289189.1289260).
* Osborne [2006] Timothy Osborne. 2006. Gapping vs. non-gapping coordination. _Linguistiche Berichte_ 207:307-337.
* Pado [2006] Sebastian Pado. 2006. _User's guide to sigf: Significance testing by approximate randomisation_. [https://nlpado.de/~sebastian/software/sigf.shtml](https://nlpado.de/~sebastian/software/sigf.shtml).
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014)_. pages 1532-1543. [https://doi.org/10.3115/v1/D14-1162](https://doi.org/10.3115/v1/D14-1162).
* Ross [1970] John Robert Ross. 1970. Gapping and the order of constituents. In Manfred Bierwisch and Karl Erich Heidolph, editors, _Progress in Linguistics_, De Gruyter, The Hague, pages 249-259.
* Schmid [2006] Helmut Schmid. 2006. Trace prediction and recovery with unlexicalized PCFGs and slash features. In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL (ACL 2006)_. pages 177-184. [https://doi.org/10.3115/1220175.1220198](https://doi.org/10.3115/1220175.1220198).
* Schuster et al. [2017] Sebastian Schuster, Matthew Lamm, and Christopher D. Manning. 2017. Mapping constructions in Universal Dependencies v2. In _Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017)_. pages 123-132. [http://www.aclweb.org/anthology/W17-0416](http://www.aclweb.org/anthology/W17-0416).
* Schuster and Manning [2016] Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English Universal Dependencies: An improved representation for natural language understanding tasks. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016)_. pages 2371-2378. [http://www.lrec-conf.org/proceedings/lrec2016/pdf/779_Paper.pdf](http://www.lrec-conf.org/proceedings/lrec2016/pdf/779_Paper.pdf).
* Seeker et al. [2012] Wolfgang Seeker, Richard Farkas, Bernd Bohnet, Helmut Schmid, and Jonas Kuhn. 2012. Data-driven dependency parsing with empty heads. In _Proceedings of COLING 2012_. pages 1081-1090. [http://www.aclweb.org/anthology/Cl2-2105](http://www.aclweb.org/anthology/Cl2-2105).
* Silveira et al. [2014] Natalia Silveira, Timothy Dzatz, Marie-Catherine de Marneffe, Samuel Bowman, Mriam Connor, Joh Bauer, and Christopher D Manning. 2014. A gold standard dependency corpus for English. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2014)_. pages 2897-2904. [http://www.lrec-conf.org/proceedings/lrec2014/pdf/1089_Paper.pdf](http://www.lrec-conf.org/proceedings/lrec2014/pdf/1089_Paper.pdf).
* Simko and Vincze [2017] Katalin Ilona Simko and Veronika Vincze. 2017. Hungarian copula constructions in dependency syntax and parsing. In _Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017)_. pages 240-247. [http://www.aclweb.org/anthology/W17-6527](http://www.aclweb.org/anthology/W17-6527) 

* Steedman (1990) Mark Steedman. 1990. Gapping as constituent coordination. _Linguistics and Philosophy_ 13(2):207-263. [https://doi.org/10.1007/BF00630734](https://doi.org/10.1007/BF00630734).
* Toasrvandani (2016) Maziar Toosrvandani. 2016. Embedding the antecedent in gapping: Low coordination and the role of parallelism. _Linguistic Inquiry_ 47(2):381-390. [https://doi.org/10.1162/LING_a_00216](https://doi.org/10.1162/LING_a_00216).
* Woods (1970) William A. Woods. 1970. Transition network grammars for natural language analysis. _Communications of the ACM_ 13(10):591-606.
* Woods (1973) William A. Woods. 1973. An experimental parsing system for transition network grammars. In Randall Rustin, editor, _Natural Language Processing_, Algorithmics Press, New York, pages 111-154.
* Wyngaard (2007) G. Vanden Wyngaard. 2007. Gapping constituents. Unpublished manuscript, FWO/K.U. Brussel. [https://lirias.kuleuven.be/bitstream/123456789/408979/1/09HRPL&L02.pdf](https://lirias.kuleuven.be/bitstream/123456789/408979/1/09HRPL&L02.pdf).
* Yeh (2000) Alexander Yeh. 2000. More accurate tests for the statistical significance of result differences. In _Proceedings of the The 18th International Conference on Computational Linguistics (COLING 2000)_. pages 947-953. [http://www.aclweb.org/anthology/C00-2137](http://www.aclweb.org/anthology/C00-2137).
* Zeman et al. (2011) Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokrmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacova, Vaclava Kettnerova, Zelenska Uresova, Jenna Kanerva, Stata Ojala, Anna Missila, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Hector Martinez Alonso, Hans Usz Koreti, Vivien Macketanz, Aljoacha Burchardt, Kim Harris, Katrin Marhenecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadova, Esha Banerjee, Ruli Manurung, Antonio Stella, Astusho Shimada, Sookyoung Kwak, Gustavo Mendonca, Tiatama Lando, Ratta Nitisaroj, and Josie Li. 2017. CoNLL 2017 Shared Task: Multilingual parsing from raw text to Universal Dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. pages 1-19. [https://doi.org/10.18653/vl/K17-3001](https://doi.org/10.18653/vl/K17-3001).