Venue: NAACL 2018

# Monte Carlo Syntax Marginals for Exploring and Using

Dependency Parses

Katherine A. Keith, Su Lin Blodgett, and Brendan O'Connor

College of Information and Computer Sciences

University of Massachusetts Amherst

{kkeith,blodgett,brenocon}@cs.umass.edu

###### Abstract

Dependency parsing research, which has made significant gains in recent years, typically focuses on improving the accuracy of single-tree predictions. However, ambiguity is inherent to natural language syntax, and communicating such ambiguity is important for error analysis and better-informed downstream applications. In this work, we propose a _transition sampling_ algorithm to sample from the full joint distribution of parse trees defined by a transition-based parsing model, and demonstrate the use of the samples in probabilistic dependency analysis. First, we define the new task of _dependency path prediction_, inferring syntactic substructures over part of a sentence, and provide the first analysis of performance on this task. Second, we demonstrate the usefulness of our _Monte Carlo syntax marginal_ method for parser error analysis and calibration. Finally, we use this method to propagate parse uncertainty to two downstream information extraction applications: identifying persons killed by police and semantic role assignment.1

Footnote 1: Supporting code available at [https://github.com/slanglab/transition_sampler](https://github.com/slanglab/transition_sampler)

## 1 Introduction

Dependency parsers typically predict a single tree for a sentence to be used in downstream applications, and most work on dependency parsers seeks to improve accuracy of such single-tree predictions. Despite tremendous gains in the last few decades of parsing research, accuracy is far from perfect--but perfect accuracy may be impossible since syntax models by themselves do not incorporate the discourse, pragmatic, or world knowledge necessary to resolve many ambiguities.

In fact, although relatively unexamined, substantial ambiguity already exists within commonly used discriminative probabilistic parsing models, which define a parse forest--a probability distribution \(p(y\mid x)\) over possible dependency trees \(y\in\mathcal{Y}(x)\) for an input sentence \(x\).

For example, the top of Figure 1 shows the predicted parse \(y^{(greedy)}\) from such a parser Chen and Manning (2014), which resolves a prepositional (PP) attachment ambiguity in one manner; this prediction was selected by a standard greedy transition-based algorithm (SS2.1). However, the bottom of Figure 1 shows _marginal_ probabilities of individual (relation, governor, child) edges under this same model. These denote our estimated probabilities, across all possible parse structures, that a pair of words are connected with a particular relation (SS2.4). For example, the two different PP attachment readings both exist within this parse forest with marginal probabilities

\[p(\text{~{}nmod}(\text{saw}_{2},\text{telescope}_{7})\mid x)=0.72 \tag{1}\] \[p(\text{~{}nmod}(\text{man}_{4},\text{telescope}_{7})\mid x)=0.28, \tag{2}\]

where (1) implies she used a telescope to see the man, and (2) implies she saw a man who had a telescope.

These types of irreducible syntactic ambiguities exist and should be taken into consideration when analyzing syntactic information; for instance, one could transmit multiple samples Finkel et al. (2006) or confidence scores Bunescu (2008) over

Figure 1: Example of a sentence with inherent ambiguity. **Top:** output from a greedy parser. **Bottom:** edge marginal probabilities from 100 samples in parentheses.



ambiguous readings to downstream analysis components.

In this work, we introduce a simple _transition sampling_ algorithm for transition-based dependency parsing (SS2.2), which, by yielding exact samples from the full joint distribution over trees, makes it possible to infer probabilities of long-distance or other arbitrary structures over the parse distribution (SS2.4). We implement transition sampling--a very simple change to pre-existing parsing software--and use it to demonstrate several applications of probabilistic dependency analysis:

* Motivated by how dependency parses are typically used in feature-based machine learning, we introduce a new parsing-related task--_dependency path prediction_. This task involves inference over variable length _dependency paths_, syntactic substructures over only parts of a sentence.
* To accomplish this task, we define a _Monte Carlo syntax marginal_ inference method which exploits information across samples of the entire parse forest. It achieves higher accuracy predictions than a traditional greedy parsing algorithm, and allows tradeoffs between precision and recall (SS4).
* We provide a quantitative measure of the model's inherent uncertainty in the parse, _whole-tree entropy_, and show how it can be used for error analysis (SS3).
* We demonstrate the method's (surprisingly) reasonable calibration (SS5).
* Finally, we demonstrate the utility of our method to propagate uncertainty to downstream applications. Our method improves performance for giving probabilistic semantics to a rule-based event extractor to identify civilians killed by police (SS6), as well as semantic role assignment (SS7).

## 2 Monte Carlo dependency analysis

### Overview of transition-based dependency parsing

We examine the _basic_ form of the Universal Dependencies formalism Nivre et al. (2016), where, for a sentence \(x\) of length \(N\), a possible dependency parse \(y\) is a set of (relation, governorToken, childToken) edges, with a tree constraint that every token in the parse has exactly one governor--that is, for every token \(w\in\{1..N\}\), there is exactly one triple \((r,g,w)\in y\) where it participates as a child. The governor is either one of the observed tokens, or a special ROOT vertex.

There exist a wide variety of approaches to machine learned, discriminative dependency parsing, which often define a probability distribution \(p(y~{}|~{}x)\) over a domain of formally legal dependency parse trees \(y\in\mathcal{Y}(x)\). We focus on _transition-based_ dependency parsers Nivre (2003); Kubler et al. (2009), which (typically) use a stack-based automaton to process a sentence, incrementally building a set of edges. Transition-based parsers are very fast, have runtimes linear in sentence length, feature high performance (either state-of-the-art, or nearly so), and are easier to implement than other modeling paradigms (SS2.5).

A probabilistic transition-based parser assumes the following stochastic process to generate a parse tree:

Initialize state \(S_{0}\)For \(n=1,2,\ldots\):

An interpretable graph \(G_{n}\mid S_{n-1}\)\(S_{n}\mathrel{\mathop{:}}=\text{Update}(S_{n-1},a_{n})\)\(C\)Break if InEndState(\(S_{n}\))

Most state transition systems Bohnet et al. (2016) use shift and reduce actions to sweep through tokens from left to right, pushing and popping them from a stack to create the edges that populate a new parse tree \(y\). The action decision probability, \(p(a_{next}~{}|~{}S_{current})\), is a softmax distribution over possible next actions. It can be parameterized by any probabilistic model, such as log-linear features of the sentence and current state Zhang and Nivre (2011), multilayer perceptrons Chen and Manning (2014), or recurrent neural networks Dyer et al. (2015); Kiperwasser and Goldberg (2016).

To predict a single parse tree on new data, a common inference method is _greedy decoding_, which runs a close variant of the above transition model as a deterministic automaton, replacing stochastic step (A) with a best-action decision, \(a_{n}\mathrel{\mathop{:}}=\operatorname*{arg\,max}_{a_{n}}p(a_{n}~{}|~{}S_{ n-1})\).2 An inferred ac 

tion sequence \(a_{1:n}\) determines the resulting parse tree (edge set) \(y\); the relationship can be denoted as \(y(a_{1:n})\).

### Transition sampling

In this work, we propose to analyze the full joint posterior \(p(y~{}|~{}x)\), and use _transition sampling_, a very simple forward/ancestral sampling algorithm,3 to draw parse tree samples from that distribution. To parse a sentence, we run the automaton stochastically, sampling the action probability in step (A). This yields one action sequence \(a_{1:n}\) from the full joint distribution of action sequences, and therefore a parse \(y(a_{1:n})\) from the distribution of parses. We can obtain as many parse samples as desired by running the transition sampler \(S\) times, yielding a collection (multiset) of parse structures \(\{y^{(s)}~{}|~{}s\in\{1..S\}\}\), where each \(y^{(s)}\sim p(y~{}|~{}x)\) is a full dependency parse tree.4 Runtime to draw one parse sample is very similar to the greedy algorithm's runtime. We denote the set of unique parses in the sample \(\tilde{\mathcal{Y}}(x)\).

Footnote 3: “Ancrestral” refers to a directed Bayes net (e.g. Barber (2012)) of action decisions, each conditioned on the full history of previous actions—not ancestors in a parse tree.

We implement a transition sampler by modifying an implementation of Chen and Manning's multilayer perceptron transition-based parser5 and use it for all subsequent experiments.

Footnote 4: Dyer et al. (2016) use the same algorithm to draw samples from a transition-based constituency parsing model, as an importance sampling proposal to support parameter learning and single-tree inference.

### MC-MAP single parse prediction

One minor use of transition sampling is a method for predicting a single parse, by selecting the most probable (common) parse tree in the sample,

\[\hat{y}^{\text{MC-MAP}} =\arg\max_{y\in\tilde{\mathcal{Y}}}~{}\tilde{p}(y~{}|~{}x) \tag{3}\] \[=\arg\max_{y\in\tilde{\mathcal{Y}}}~{}\frac{c(y)}{S} \tag{4}\]

where \(\tilde{p}(y~{}|~{}x)\) denotes the Monte Carlo estimate of a parse's probability, which is proportional to how many times it appears in the sample: \(c(y)\equiv\sum_{s}^{S}1\{y=y^{(s)}\}\). Note that \(\tilde{p}(y~{}|~{}x)\) correctly accounts for the case of an ambiguous transition system where multiple different action sequences can yield the same tree--i.e., \(y(a_{1:n})\) is not one-to-one--since the transition sampler can sample the multiple different paths.

This "MC-MAP" method is asymptotically guaranteed to find the model's most probable parse (\(\operatorname*{arg\,max}_{y}p(y~{}|~{}x)\)) given enough samples.6 By contrast, greedy decoding and beam search have no theoretical guarantees. MC-MAP's disadvantage is that it may require a large number of samples, depending on the difference between the top parse's probability compared to other parses in the domain.

Footnote 6: This holds since the Monte Carlo estimated probability of any tree converges to its true probability, according to, e.g., Hoeffding’s inequality or the central limit theorem. Thus, with enough samples, the tree with the highest true probability will have estimated probability higher than any other tree’s.

### Monte Carlo Syntax Marginal (MCSM) inference for structure queries

Beyond entire tree structures, parse posteriors also define marginal probabilities of particular events in them. Let \(f(y)\rightarrow\{0,1\}\) be a boolean-valued _structure query_ function of a parse tree--for example, whether the tree contains a particular edge:

\[f(y)=1\{\mbox{dobj(kill,Smith)}\in y\}\]

or more complicated structures, such as a length-2 dependency path:

\[f(y)=1\{\mbox{nsubj(kill,cop)$}\wedge\mbox{dobj(kill,Smith)}\in y\}.\]

More precisely, these queries are typically formulated to check for edges between specific tokens, and may check tokens' string forms.

Although \(f(y)\) is a deterministic function, since the parsing model is uncertain of the correct parse, we find the _marginal probability_, or expectation, of a structure query by integrating out the posterior parse distribution--that is, the predicted probability that the parse has the property in question:

\[p(f(y)~{}|~{}x) =\sum_{y\in\mathcal{Y}(x)}f(y)~{}p(y~{}|~{}x) \tag{5}\] \[\approx\tilde{p}(f(y)~{}|~{}x) =\sum_{y\in\tilde{\mathcal{Y}}(x)}f(y)\frac{c(y)}{S}. \tag{6}\]

Eq. 5 is the expectation with regard to the model's true probability distribution (\(p\)) over parses from the domain of all possible parse trees \(\mathcal{Y}(x)\) for a sentence, while Eq. 6 is a Monte Carlo estimate of the query's marginal probability--the fraction of parse tree samples where the structure query is true. We use this simple method for all inference 

in this work, though importance sampling (Dyer et al., 2016), particle filters (Buys and Blunsom, 2015), or diverse k-best lists (Zhang and McDonald, 2014) could support more efficient inference in future work.

### Probabilistic inference for dependencies: related work

Our transition sampling method aims to be an easy-to-implement algorithm for a highly performant class of dependency models, that conducts exact probabilistic inference for arbitrary structure queries in a reasonable amount of time. A wide range of alternative methods have been proposed for dependency inference that cover some, but perhaps not all, of these goals.

For transition-based parsing, beam search is a commonly used inference method that tries to look beyond a single structure. Beam search can be used to yield an approximate \(K\)-best list by taking resulting structures on the beam, though there are no theoretical guarantees about the result, and runtime is no better than the transition sampler.7 Finkel et al. (2006) further discuss trade-offs between beam search and sampling, and find they give similar performance when propagating named entity recognition and PCFG parse information to downstream tasks.

Footnote 7: Loosely, if it takes \(N\) transitions to complete a parse, and \(B\) possible actions at each transition must be evaluated, our method evaluates \(KNB\) actions to obtain \(K\) trees. Beam search evaluates a similar number of actions when using a \(K\)-sized beam, but also requires non-parallelizable management of the beam’s priority queue.

Graph-based parsers are the major alternative modeling paradigm for dependency parsing; instead of a sequence of locally normalized decisions, they directly parameterize an entire tree's globally normalized probability. Parse samples could be drawn from a graph-based model via Markov chain Monte Carlo (Zhang et al., 2014), which is asymptotically correct, but may require a large amount of time to obtain non-autocorrelated parses. A range of methods address inference for specific queries in graph-based models--for example, edge marginals for edge-factored models via the matrix-tree theorem (Koo et al., 2007), or approximate marginals with loopy belief propagation (Smith and Eisner, 2008).8 By contrast, our method is guaranteed to give correct marginal inferences for arbitrary, potentially long-distance, queries.

Footnote 8: These papers infer marginals to support parameter learning, but we are not aware of previous work that directly analyzes or uses dependency parse marginals.

Given the strong performance of graph-based parsers in the single-structure prediction setting (e.g. Zeman et al. (2017); Dozat et al. (2017)), it may be worthwhile to further explore probabilistic inference for these models. For example, Nicolulae et al. (2018) present an inference algorithm for a graph-based parsing model that infers a weighted, sparse set of highly-probable parse trees, and they illustrate that it can infer syntactic ambiguities similar to Figure 1.

Dynamic programming for dependency parsing, as far as we are aware, has only been pursued for single-structure prediction (e.g. Huang and Sagae (2010)), but in principle could be generalized to calculate local structure query marginals via an inside-outside algorithm, or to sample entire structures through an inside-outside sampler (Eisner, 2016), which Finkel et al. (2006) use to propagate parse uncertainty for downstream analysis.

## 3 Exploratory error analysis via whole-tree entropy calculations

In this section we directly explore the model's intrinsic uncertainty, while SS5 conducts a quantitative analysis of model uncertainty compared to gold standard structures. Parse samples are able to both pass on parse uncertainty and yield useful insights that typical error analysis approaches cannot. For a sentence \(x\), we can calculate the _whole-tree entropy_, the model's uncertainty of whole-tree parse frequencies in the samples:

\[H(p)=-\sum_{y\in\mathcal{Y}(x)}p(y\mid x)\log p(y\mid x)\] \[\approx H(\tilde{p})=-\sum_{y\in\mathcal{Y}(x)}\frac{c(y)}{S} \log\frac{c(y)}{S}. \tag{7}\]

Since this entropy estimate is only based on an \(S\)-sample approximation of \(p\), it is upper bounded at \(\log(S)\) in the case of a uniform MC distribution. Another intuitive measure of uncertainty is simply the number of unique parses, that is, the cardinality of the MC distribution's domain \((|\vec{\mathcal{Y}}|)\); this quantity is not informative for the true distribution \(p\), but in the MC distribution it is intuitively upper bounded by \(S\).9 

We run our dependency sampler on the 2002 sentences in the Universal Dependencies 1.3 English Treebank development set, generating 100 samples per sentence; Table 1 shows example sentences along with \(|{\tilde{\mathcal{Y}}}|\) and entropy statistics for each sentence. We find that in general, as sentence length increases, so does the entropy of the parse distribution (Fig. 2). Moreover, we find that entropy is a useful diagnostic tool. For example, 7% of sentences in the UD development corpus with fewer than 15 tokens and \(H(\tilde{p})\geq 2\) exhibit uncertainty around the role of '-' (compare _Sciences - principally biology_ and _thought-provoking_), and another 7% of such sentences exhibit uncertainty around 's' (potentially representing a plural or a possessive).

## 4 Monte Carlo Syntax Marginals for partial dependency parsing

Here we examine the utility of marginal inference for predicting parts of dependency parses, using the UD 1.3 Treebank's English development set to evaluate.10

Footnote 10: UD 1.3 is the UD version that this parsing model is most similar to: [https://mailman.stanford.edu/pipermail/parser-user/2017-November/003460.html](https://mailman.stanford.edu/pipermail/parser-user/2017-November/003460.html)

### Greedy decoding

Using its off-the-shelf pretrained model with greedy decoding, the CoreNLP parser achieves 80.8% labeled attachment score (LAS). LAS is equivalent to both the precision and recall of predicting (rel,gov,child) triples in the parse tree.11

Footnote 11: LAS is typically defined as proportion of tokens whose governor (and relation on that governor-child edge) are correctly predicted; this is equivalent to precision and recall of edges if all observed tokens are evaluated. If, say, punctuation is excluded from evaluation, this equivalence does not hold; in this work we always use all tokens for simplicity.

### Minimum Bayes risk (MBR) decoding

A simple way to use marginal probabilities for parse prediction is to select, for each token, the governor and relation that has the highest marginal probability. This method gives a minimum Bayes risk (MBR) prediction of the parse, minimizing the model's expected LAS with regards to local uncertainty; similar MBR methods have been shown to improve accuracy in tagging and constituent parsing (e.g. Goodman (1996); Petrov and Klein (2007)). This method yields 81.4% LAS, outperforming greedy parsing, though it may yield a graph that is not a tree.

### Syntax marginal inference for dependency paths

An alternative view on dependency parsing is to consider what structures are needed for downstream applications. One commonly used parse substructure is the _dependency path_ between two words, which is widely used in unsupervised lexical semantics Lin and Pantel (2001), distantly supervised lexical semantics Snow et al. (2005), relation learning Riedel et al. (2013), and supervised semantic role labeling Hacioglu (2004); Das et al. (2014), as well as applications in economics Ghose et al. (2007), political science O'Connor et al. (2013), biology Fundel et al. (2006), and the humanities Bamman et al. (2013, 2014).

\begin{table}
\begin{tabular}{l c c l} Sentence & Domain Size & Top 3 Freq. & Entropy \\ \hline \hline In Ramadi , there was a big demonstration . & 3 & [ 98, 1, 1 ] & 0.112 \\ \hline US troops there dashed with guerrillas in a fight that left one Iraqi dead . & 40 & [ 33, 11, 6 ] & 2.865 \\ \hline The sheikh in wheel - chair has been attacked with a F - 16 - launched bomb . & 98 & [ 2, 2, 1 ] & 4.577 \\ \end{tabular}
\end{table}
Table 1: Example sentences from the UD development set and summaries of their Monte Carlo parse distributions. _Domain Size_ gives \(|{\tilde{\mathcal{Y}}}_{100}|\), the number of unique parse structures in 100 samples. _Top 3 Freq._ gives the frequencies of the 3 most probable structures in \({\tilde{\mathcal{Y}}}_{100}\). _Entropy_ is calculated according to Eq. 7; its upper bound is \(\log(100)=4.605\).

Figure 2: Length of sentences (in number of tokens in UD development set) against entropy (100 samples per sentence).

 

In this work, we consider a dependency path to be a set of edges from the dependency parse; for example, a length-2 path \(p=\{\text{nsubj}(3,1),\text{dobj}(3,4)\}\) connects tokens 1 and 4. Let \(\mathcal{P}_{d}(y)\) be the set of all length-\(d\) paths from a parse tree \(y\).12 Figure 3's "Greedy" table column displays the F-scores for the precision and recall of retrieving \(\mathcal{P}_{d}(y^{(gold)})\) from the prediction \(\mathcal{P}_{d}(y^{(greedy)})\) for a series of different path lengths. \(\mathcal{P}_{1}\) gives individual edges, and thus is the same as LAS (80.8%). Longer length paths see a rapid decrease in performance; even length-2 paths are retrieved with only \(\approx 66\%\) precision and recall.13 We are not aware of prior work that evaluates dependency parsing beyond single edge or whole sentence accuracy.

Footnote 12: Path construction may traverse both up and down directed edges; we represent a path as an edge set to evaluate its existence in a parse. A path may not include the same vertex twice. The set of all paths for a parse includes all paths from all pairs of vertexes (observed tokens and ROOT).

Footnote 13: For length 1 paths, precision and recall are identical; this does not hold for longer paths, though precision and recall from a single parse prediction are similar.

We define _dependency path prediction_ as the task of predicting a set of dependency paths for a sentence; the paths do not necessarily have to come from the same tree, nor even be consistent with a single syntactic analysis. We approach this task with our Monte Carlo syntax marginal method, by predicting paths from the transition sampling parser. Here we treat each possible path as a structure query (SS2.4) and return all paths whose marginal probabilities are at least threshold \(t\). Varying \(t\) trades off precision and recall.

We apply this method to 100 samples per sentence in the UD treebank. When we take all length-1 paths that appear in every single sample (i.e., estimated marginal probability \(1.0\)), precision greatly increases to \(0.969\), while recall drops to \(0.317\) (the top-left point on Figure 3's teal length-1 curve.) We can also accommodate applications which may prefer to have a higher recall: predicting all paths with at least \(0.01\) probability results in \(0.936\) recall (the bottom-right point on the curve in Figure 3).14

Footnote 14: The 6.4% of gold-standard edges with predicted 0 probability often correspond to inconsistencies in the formalism standards between the model and UD; for example, 0.7% of the gold edges are ‘name’ relations among words in a name, which the model instead analyzes as ‘compound’. Inspecting gold edges’ marginal probabilities helps error analysis, since when one views a single predicted parse, it is not always clear whether observed errors are systematic, or a fluke for that one instance.

This marginal path prediction method dominates the greedy parser: for length-1 paths, there are points on the marginal decoder's PR curve that achieve both higher precision and recall than the greedy decoder, giving F1 of 82.4% when accepting all edges with marginal probability at least \(0.45\). Furthermore, these advantages are more prominent for longer dependency paths. For example, for length-3 paths, the greedy parser only achieves 50.6% F1, whil

Figure 3: Precision-recall analysis against gold UD 1.3 English dev (§4.1) set for all dependency paths of lengths 1 to 7. **Left:** Each path length is a different color (1 in top-right, 7 in bottom-left), with greedy performance (\(\bigcirc\)) as well as the marginal path predictions’ PR curve, with points at confidence thresholds 0.9 (\(\blacktriangle\)), 0.1 (\(\blacktriangledown\)), and where F1 is highest (\(\blacksquare\)). **Right:** F1 for each method, as well as the confidence threshold for the marginal PR curve’s max-F1 point. For path length 1, Greedy and MC-MAP F1 are the same as labeled attachment score (LAS).

 

proves a bit to 55.0% F1; strikingly, it is possible to select high-confidence paths to get much higher 90.1% precision (at recall 11.6%, with confidence threshold \(t=0.95\)). Figure 3 also shows the precision/recall points on each curve for thresholds \(t=0.9\) and \(t=0.1\).

We also evaluated the MC-MAP single-parse prediction method (SS2.3), which slightly, but consistently, underperforms the greedy decoder at all dependency lengths. More work is required to understand whether this is is an inference or modeling problem: for example, we may not have enough samples to reliably predict a high-probability parse; or, as some previous work finds in the context of beam search, the label bias phenomenon in this type of locally-normalized transition-based parser may cause it to assign higher probability to non-greedy analyses that in fact have lower linguistic quality [48, 7].

## 5 Calibration

The precision-recall analysis shows that the predicted marginal probabilities are meaningful in a ranking sense, but we can also ask whether they are meaningful in a sense of _calibration_: predictions are calibrated if, among all structures with predicted probability \(q\pm\epsilon\), they exist in the gold parses with probability \(q\). That is, predictions with confidence \(q\) have precision \(q\).15 If probabilities are calibrated, that implies expectations with regard to their distribution are unbiased, and may also justify intuitive interpretations of probabilities in exploratory analysis (SS3). Calibration may also have implications for joint inference, EM, and active learning methods that use confidence scores and confidence-based expectations.

Footnote 15: This is a _local_ precision, as opposed to the more usual tail probability of measuring precision of all predictions _higher_ than some \(l\)—the integral of local precision. For example, Figure 3’s length-\(1=0.9\) precision of \(0.942~{}(\triangle)\) is the average \(y\) value of several rightmost bins in Figure 4. This contrast corresponds to Efron (2010)’s dichotomy of local versus global false discovery rates.

We apply Nguyen and O'Connor (2015)'s adaptive binning method to analyze the calibration of structure queries from an NLP system, by taking the domain of all seen length-\(d\) paths from the 100 samples' parse distribution for the treebank, grouping by ranges of predicted probabilities to have at least 5000 paths per bin, to ensure stability of the local precision estimate.16

Footnote 16: This does not include gold-standard paths with zero predicted probability. As Nguyen and O’Connor found for sequence tagging and coreference, we find the prediction distribution is heavily skewed to near 0 and 1, necessitating adaptive bins, instead of fixed-width bins, for calibration analysis [108, 7].

We find that probabilities are reasonably well calibrated, if slightly overconfident--Figure 4 shows the average predicted probability per bin, compared to how often these paths appear in the gold standard (local precision). For example, for edges (length-1 paths), predictions near 60% confidence (the average among predictions in range \([0.42,0.78]\)) correspond to edges that are actually in the gold standard tree only 52.8% of the time. The middle confidence range has worse calibration error, and longer paths perform worse. Still, this level of calibration seems remarkably good, considering there was no attempt to re-calibrate predictions [107] or to use a model that specifically parameterizes the energy of dependency paths [109, 7]--these predictions are simply a side effect of the overall joint model for incremental dependency parsing.

## 6 Probabilistic rule-based IE: classifying police fatalities

Supervised learning typically gives the most accurate information extraction or semantic parsing systems, but for many applications where train

Figure 4: Calibration curves for marginal predictions for several path lengths. Predictions below the \(y=x\) line indicate overconfidence. Points denote the average predicted probability versus empirical (gold) probability among predicted paths within each dynamically allocated bin.

 

ing data is scarce, Chiticariu et al. (2013) argue that rule-based systems are useful and widespread in practice, despite their neglect in contemporary NLP research. Syntactic dependencies are a useful abstraction with which to write rule-based extractors, but they can be brittle due to errors in the parser. We propose to integrate over parse samples to infer a _marginal_ probability of a rule match, increasing robustness and allowing for precision-recall tradeoffs.

### Police killings victim extraction

We examine the task of extracting the list of names of persons killed by police from a test set of web news articles in Sept-Dec 2016. We use the dataset released by Keith et al. (2017), consisting of 24,550 named entities \(e\in{\cal E}\) and sentences from noisy web news text extractions (that can be difficult to parse), each of which contains at least one \(e\) (on average, 2.8 sentences/name) as well as keywords for both police and killing/shooting. The task is to classify whether a given name is a person who was killed by police, given 258 gold-standard names that have been verified by journalists.

### Dependency rule extractor

Keith et al. present a baseline rule-based method that uses Li and Ji (2014)'s off-the-shelf RPIJIE ACE event parser to extract (event type, agent, patient) tuples from sentences, and assigns \(f_{\text{JE}}(x_{i},e)=1\) iff the event type was a killing, the agent's span included a police keyword, and the patient was the candidate entity \(e\). An entity is classified as a victim if at least one sentence is classified as true, resulting in a 0.17 F1 score (as reported in previous work).17

Footnote 17: This measures recall of the entire gold-standard victim database, though the corpus only includes 57% of the victims.

We define a similar syntactic dependency rule system using a dependency parse as input: our extractor \(f(x,e,y)\) returns 1 iff the sentence has a killing keyword \(k\),18 which both

Footnote 18: Police and killing/shooting keywords are from Keith et al.’s publicly released software.

1. has an agent token \(a\) (defined as, governed by _nsubj_ or _nmod_) which is a police keyword, or \(a\) has a (_amod_ or _compound_) modifier that is a police keyword; and,
2. has a patient token \(p\) (defined as, governed by _nsubjpass_ or _dobj_) contained in the candidate name \(e\)'s span.

Applying this \(f(x,e,y)\) classifier to greedy parser output, it performs better than the RPI-JIE-based rules (Figure 5, right), perhaps because it is better customized for the particular task.

Treating \(f\) as a structure query, we then use our Monte Carlo marginal inference (SS2) method to calculate the probability of a rule match for each sentence--that is, the fraction of parse samples where \(f(x,e,y^{(s)})\) is true--and infer the entity's probability with the _noisy-or_ formula Craven and Kumlien (1999); Keith et al. (2017). This gives soft classifications for entities.

### Results

The Monte Carlo method achieves slightly higher F1 scores once there are at least 10 samples (Fig. 5, right). More interestingly, the soft entity-level classifications also allow for precision-recall tradeoffs (Fig. 5, left), which could be used to prioritize the time of human reviewers updating the victim database (filter to higher precision), or help ensure victims are not missed (with higher recall). We found the sampling method retrieved several true-positive entities where only a single sentence had a non-zero rule prediction at probability 0.01--that is, the rule was only matched in one of 100 sampled parses. Since current practitioners are already manually reviewing millions of news articles to create police fatality victim databases, the ability to filter to high recall--even with low precision--may be useful to help ensure victims are not missed.

### Supervised learning

Sampling also slightly improves supervised learning for this problem. We modify Keith et al.'s logistic regression model based on a dependency

Figure 5: **Left: Rule-based entity precision and recall for police fatality victims, with greedy parsing and Monte Carlo inference. Right: F1 scores for RPI-JIE, Greedy, and 1-sample methods, and maximum F1 on PR curve for probabilistic (multiple sample) inference.**



path feature vector \(f(x_{i},y)\), instead creating feature vectors that average over multiple parse samples (\(E_{\tilde{p}(y)}[f(x_{i},y)]\)) at both train and test time. With the greedy parser, the model results in 0.229 F1; using 100 samples slightly improves performance to 0.234 F1.

## 7 Semantic role assignment

Semantic role labeling (SRL), the task to predict argument structures Gildea and Jurafsky (2002), is tightly tied to syntax, and previous work has found it beneficial to conduct it with joint inference with constituency parsing, such as with top-\(k\) parse trees Toutanova et al. (2008) or parse tree samples Finkel et al. (2006). Since SS4 shows that Monte Carlo marginalization improves dependency edge prediction, we hypothesize dependency sampling could improve SRL as well.

SRL includes both identifying argument spans, and assigning spans to specific semantic role labels (argument types). We focus on just the second task of semantic role assignment: assuming argument spans are given, to predict the labels. We experiment with English OntoNotes v5.0 annotations Weischedel et al. (2013) according to the CoNLL 2012 test split Pradhan et al. (2013). We focus only on predicting among the five core arguments (Arg0 through Arg4) and ignore spans with gold-standard adjunct or reference labels. We fit a separate model for each predicate19 among the 2,160 predicates that occur at least once in both the training and test sets (115,811 and 12,216 sentences respectively).

Footnote 19: That is, for each unique (lemma, framesetID) pair, such as (view, view-02).

Our semantic model of label \(z_{t}\in\{A0..A4\}\) for argument head token \(t\) and predicate token \(p\), \(p_{\text{sem}}(z_{t}~{}|~{}p,y)\), is simply the conditional probability of the label, conditioned on \(y\)'s edge between \(t\) and \(p\) if one exists.20 (If they are not directly connected, the model instead conditions on a 'no edge' feature.) Probabilities are maximum likelihood estimates from the training data's (predicate, argument label, path) counts, from either greedy parses, or averaged among parse samples. To predict at test time, the greedy parsing model simply uses \(p(z_{t}~{}|~{}p,y^{(greedy)})\). The Monte Carlo model, by contrast, treats it as a directed joint model and marginalizes over syntactic analyses:

Footnote 20: The dataset’s argument spans must be reconciled with predicted parse structures to define the argument head \(t\); 90% of spans are consistent with the greedy parser in that all the span’s tokens have the same highest ancestor contained with the span, which we define as the argument head. For inconsistent cases, we select the largest subtree (that is, highest within-span ancestor common to the largest number of the span’s tokens). It would be interesting to modify the sampler to restrict to parses that are consistent with the span, as a form of rejection sampling.

\[p_{MC}(z_{t}~{}|~{}p,x)=\sum_{y\in\mathcal{Y}(x)}p_{\text{sem}}( z_{t}~{}|~{}p,y)~{}\tilde{p}_{\text{syn}}(y~{}|~{}x).\]

The baseline accuracy of predicting the predicate's most common training-time argument label yields 0.393 accuracy, and the greedy parser performs at 0.496. The Monte Carlo method (with 100 samples) improves accuracy to 0.529 (Table 2). Dependency samples' usefulness in this limited case suggests they may help systems that use dependency parses more broadly for SRL Hacioglu (2004); Das et al. (2014).

## 8 Conclusion

In this work, we introduce a straightforward algorithm for sampling from the full joint distribution of a transition-based dependency parser. We explore using these parse samples to discover both parsing error and structural ambiguities. Moreover, we find that our Monte Carlo syntax marginal method not only dominates the greedy method for dependency path prediction (especially for longer paths), but also allows for control of precision-recall tradeoffs. Propagating dependency uncertainty can potentially help a wide variety of semantic analysis and information extraction tasks.

## Acknowledgments

The authors would like to thank Rajarshi Das, Daniel Cohen, Abe Handler, Graham Neubig, Emma Strubell, and the anonymous reviewers for their helpful comments.

## References

* Andor et al. (2014) Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav

\begin{table}
\begin{tabular}{l l} Method & Accuracy \\ \hline Baseline (most common) & 0.393 \\ Greedy & 0.496 \\ MCSM method, 100-samples & **0.529** \\ \end{tabular}
\end{table}
Table 2: Semantic role assignment accuracy on English OntoNotes v5.0. The baseline is for each unique predicate, predict the argument that was seen the most at training time.

 

Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In _Proceedings of ACL_.
* Bamman et al. (2013) David Bamman, Brendan O'Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In _Proceedings of ACL_.
* Bamman et al. (2014) David Bamman, Ted Underwood, and Noah A. Smith. 2014. A Bayesian mixed effects model of literary character. In _Proceedings of ACL_.
* Barber (2012) David Barber. 2012. _Bayesian Reasoning and Machine Learning_. Cambridge University Press.
* Bennett (2000) Paul N. Bennett. 2000. Assessing the calibration of naive Bayes' posterior estimates. Technical report, Carnegie Mellon University.
* Bohnet et al. (2016) Bernd Bohnet, Ryan McDonald, Emily Pitler, and Ji Ma. 2016. Generalized transition-based dependency parsing via control parameters. In _Proceedings of ACL_. [http://www.aclweb.org/anthology/P16-1015](http://www.aclweb.org/anthology/P16-1015).
* Bunescu (2008) Razvan Bunescu. 2008. Learning with probabilistic features for improved pipeline models. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D08-1070](http://www.aclweb.org/anthology/D08-1070).
* Buys and Blunsom (2015) Jan Buys and Phil Blunsom. 2015. A bayesian model for generative transition-based dependency parsing. In _Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015)_. Uppsala, Sweden. [http://www.aclweb.org/anthology/W15-2108](http://www.aclweb.org/anthology/W15-2108).
* Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D14-1082](http://www.aclweb.org/anthology/D14-1082).
* Chiticariu et al. (2013) Laura Chiticariu, Yunyao Li, and Frederick R. Reiss. 2013. Rule-based information extraction is dead! long live rule-based information extraction systems! In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D13-1079](http://www.aclweb.org/anthology/D13-1079).
* Craven and Kumlien (1999) Mark Craven and Johan Kumlien. 1999. Constructing biological knowledge bases by extracting information from text sources. In _ISMB_. pages 77-86.
* Das et al. (2014) Dipanjan Das, Desai Chen, Andre F. T. Martins, Nathan Schneider, and Noah A. Smith. 2014. Frame-semantic parsing. _Computational Linguistics_ .
* Dozat et al. (2017) Timothy Dozat, Peng Qi, and Christopher D Manning. 2017. Stanford's graph-based neural dependency parser at the conll 2017 shared task. _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ pages 20-30.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of ACL_. [http://www.aclweb.org/anthology/P15-1033](http://www.aclweb.org/anthology/P15-1033).
* Dyer et al. (2016) Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In _Proceedings of NAACL_. [http://www.aclweb.org/anthology/N16-1024](http://www.aclweb.org/anthology/N16-1024).
* Efron (2010) Bradley Efron. 2010. _Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction_. Cambridge University Press, 1st edition.
* Eisner (2016) Jason Eisner. 2016. Inside-outside and forward-backward algorithms are just backprop (tutorial paper). In _Proceedings of the Workshop on Structured Prediction for NLP_. Association for Computational Linguistics, Austin, TX, pages 1-17. [http://aclweb.org/anthology/W16-5901](http://aclweb.org/anthology/W16-5901).
* Finkel et al. (2006) Jenny Rose Finkel, Christopher D. Manning, and Andrew Y. Ng. 2006. Solving the problem of cascading errors: Approximate Bayesian inference for linguistic annotation pipelines. In _Proceedings of EMNLP_.
* Fundel et al. (2006) Katrin Fundel, Robert Kuffner, and Ralf Zimmer. 2006. Relax--relation extraction using dependency parse trees. _Bioinformatics_ 23(3):365-371.
* Ghose et al. (2007) Anindya Ghose, Panagiotis Ipeirotis, and Arun Sundararajan. 2007. Opinion mining using econometrics: A case study on reputation systems. In _Proceedings of ACL_. Prague, Czech Republic. [http://www.aclweb.org/anthology/P07-1053](http://www.aclweb.org/anthology/P07-1053).
* Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. _Computational linguistics_ 28(3):245-288.
* Goodman (1996) Joshua Goodman. 1996. Parsing algorithms and metrics. In _Proceedings of ACL_. [https://doi.org/10.3115/981863.981887](https://doi.org/10.3115/981863.981887).
* Hacioglu (2004) Kadri Hacioglu. 2004. Semantic role labeling using dependency trees. In _Proceedings of the 20th international conference on Computational Linguistics_. Association for Computational Linguistics.
* Huang and Sagae (2010) Liang Huang and Kenji Sagae. 2010. Dynamic programming for linear-time incremental parsing. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Uppsala, Sweden, pages 1077-1086. [http://www.aclweb.org/anthology/P10-1110](http://www.aclweb.org/anthology/P10-1110).
* Keith et al. (2017) Katherine Keith, Abram Handler, Michael Pinkham, Cara Magliozzi, Joshua McDuffie, and Brendan O'Connor. 2017. Identifying civilians killed by police with distantly supervised entity-event extraction. In _Proceedings of EMNLP_ 

* Kiperwasser and Goldberg (2016) Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. _Transactions of ACL_ .
* Koo et al. (2007) Terry Koo, Amir Globerson, Xavier Carreras, and Michael Collins. 2007. Structured prediction models via the matrix-tree theorem. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D/D07/D07-1015](http://www.aclweb.org/anthology/D/D07/D07-1015).
* Kubler et al. (2009) Sandra Kubler, Ryan McDonald, and Joakim Nivre. 2009. Dependency parsing. _Synthesis Lectures on Human Language Technologies_ .
* Kuleshov and Liang (2015) Volodymyr Kuleshov and Percy S. Liang. 2015. Calibrated structured prediction. In _Advances in Neural Information Processing Systems_.
* Li and Ji (2014) Qi Li and Heng Ji. 2014. Incremental joint extraction of entity mentions and relations. In _Proceedings of ACL_. pages 402-412.
* discovery of inference rules from text. In _Proceedings of KDD_.
* Martins et al. (2010) Andre Martins, Noah Smith, Eric Xing, Pedro Aguiar, and Mario Figueiredo. 2010. Turbo parsers: Dependency parsing by approximate variational inference. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D10-1004](http://www.aclweb.org/anthology/D10-1004).
* Nguyen and O'Connor (2015) Khanh Nguyen and Brendan O'Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. In _Proceedings of EMNLP_. [http://aclweb.org/anthology/D15-1182](http://aclweb.org/anthology/D15-1182).
* Niculae et al. (2018) Vlad Niculae, Andre FT Martins, Mathieu Blondel, and Claire Cardie. 2018. SparseMAP: Differentiable sparse structured inference. _arXiv preprint arXiv:1802.04223_ .
* Niculescu-Mizil and Caruana (2005) Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In _Proceedings of ICML_.
* Nivre (2003) Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In _Proceedings of the 8th International Workshop on Parsing Technologies (IWPT)_.
* Nivre et al. (2016) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsafraty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In _Proceedings of LREC_.
* O'Connor et al. (2013) Brendan O'Connor, Brandon Stewart, and Noah A. Smith. 2013. Learning to extract international relations from political context. In _Proceedings of ACL_.
* Petrov and Klein (2007) Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In _Proceedings of NAACL HLT 2007_.
* Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Bjorkelandt, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In _CoNLL_. pages 143-152.
* Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In _Proceedings of NAACL_. Atlanta, Georgia. [http://www.aclweb.org/anthology/N13-1008](http://www.aclweb.org/anthology/N13-1008).
* Smith and Eisner (2008) David Smith and Jason Eisner. 2008. Dependency parsing by belief propagation. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D08-1016](http://www.aclweb.org/anthology/D08-1016).
* Smith and Eisener (2007) David A. Smith and Jason Eisner. 2007. Bootstrapping feature-rich dependency parsers with entropic priors. In _Proceedings of EMNLP_. [http://www.aclweb.org/anthology/D/D07/D07-1070](http://www.aclweb.org/anthology/D/D07/D07-1070).
* Snow et al. (2005) Rion Snow, Dan. Jurafsky, and Andrew Y. Ng. 2005. Learning syntactic patterns for automatic hypernym discovery. In _Advances in Neural Information Processing Systems_.
* Toutanova et al. (2008) Kristina Toutanova, Arita Haghighi, and Christopher D. Manning. 2008. A global joint model for semantic role labeling. _Computational Linguistics_ 34(2):161-191.
* Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 LDCE2013T19. _Linguistic Data Consortium, Philadelphia, PA_ .
* Zeman et al. (2013) Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedduzhko, Silvie Cinkova, Jan Hajic jr, Jaroslavacva Hakvacova, Vaclava Kettenrova, Zdenka Ursesova, Jenna Kanerva, Stina Ojala, Anna Missila, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taiji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria dePaixa, Kira Droganova, Hector Martinez Alonso, Capf Coltekin, Umur Suluback, Hans Uszkoetit, Vivien Macketaanz, Aljoscha Burchardt, Kim Harris, Katrin Marinheecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpratid, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadova, Esha Banerjee, Ruli Manvrup, Antonio Stella, Ausuko Shimada, Sookiyoung Kwak, Gustavo Mendonca, Tatianna Lando, Rattima Nitisaroj, and Josie Li. 2017. Conll 2017 shared task: Multilingual parsing from raw text to universal dependencies. In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_. Association for Computational 

Linguistics, Vancouver, Canada, pp. 1-19. External Links: Link Cited by: SS2.
* H. Zhang and R. McDonald (2014)Enforcing structural diversity in cube-pruned dependency parsing. In Proceedings of ACL, pp. 378-386. External Links: Link Cited by: SS2.
* Y. Zhang, T. Lei, R. Barzilay, and T. Jaakkola (2014)Greedy is good if randomized: new inference for dependency parsing. In Proceedings of EMNLP, Qatar, pp. 115-120. External Links: Link Cited by: SS2.
* Y. Zhang and J. Nivre (2011)Transition-based dependency parsing with rich non-local features. In Proceedings of ACL, pp. 111-1203. External Links: Link Cited by: SS2.
* Y. Zhang and J. Nivre (2012)Analyzing the effect of global learning and beam-search on transition-based dependency parsing. In Proceedings of COLING, pp. 111-116. External Links: Link Cited by: SS2.