SlowGuess/NAACL_2022_OCR huggingface dataset:
Tianyu Liu Yuchen Eleanor Jiang Ryan Cotterell Minmaya Sachan
{tianyu.liu, yucjiang, ryan.cotterell, mrinmaya.sachan}@inf.ethz.ch
ETH zürich
Many natural language processing tasks, e.g., coreference resolution and semantic role labeling, require selecting text spans and making decisions about them. A typical approach to such tasks is to score all possible spans and greedily select spans for task-specific downstream processing. This approach, however, does not incorporate any inductive bias about what sort of spans ought to be selected, e.g., that selected spans tend to be syntactic constituents. In this paper, we propose a novel grammar-based structured span selection model which learns to make use of the partial span-level annotation provided for such problems. Compared to previous approaches, our approach gets rid of the heuristic greedy span selection scheme, allowing us to model the downstream task on an optimal set of spans. We evaluate our model on two popular span prediction tasks: coreference resolution and semantic role labeling. We show empirical improvements on both.
https://github.com/lyutyuh/structured-span-selector
The problem of selecting a continuous segment of the input text, termed a span, is a common design pattern in NLP. In this work, we call this design pattern span selection. Common tasks that have a span prediction component include coreference resolution (Stede, 2011), where the selected spans are mentions, semantic role labeling (Palmer et al., 2010), where the selected spans are arguments, question answering (Jurafsky and Martin, 2009), where the selected spans are answers, and named entity recognition (Smith, 2011), where the selected spans are entities.
In most of the tasks mentioned above, span selection is the first step. After a set of candidate spans
Coref: [The President]1 has said  $\left[\left[\mathrm{he}\right]_{1}\right.$  and  $\left[\mathrm{his}\right]_{1}$  wife, now a [New York]3 senator]2 will spend weekends at [their]2 house in Chappaqua.
SRL: [The most important thing about Disney]1-ARG1  $\boxed{\mathrm{is}_1}$  [that [it]2-ARG1  $\boxed{\mathrm{is}_2}$  [a global brand]2-ARG2]1-ARG2.
Figure 1: Examples of two span prediction tasks: Coreference and SRL. In Coref,  $[s]_i$  denotes a span of text  $s$  referring to the  $i^{\text{th}}$  entity. In SRL,  $\boxed{\mathrm{i}}$  denotes a set of predicates and  $[s]_{i - *}$  denotes a set of arguments for the  $i^{\text{th}}$  predicate.
is determined, a classifier (often a neural network) is typically used to make predictions about the candidate spans. For instance, in coreference resolution, the selected spans (mentions) are clustered according to which entity they refer to; whereas in SRL, the spans (arguments) are classified into a set of roles. Two examples are shown in Fig. 1.
As the number of spans to consider in the input text can be quadratic in the length of the input, candidate spans are greedily selected as potential antecedents, roles, or answers. While greedy span selection has become the de-facto approach in span prediction problems, it has several issues. First, such approaches typically ignore the inherent structure of the problem. For example, spans of interest in problems such as coreference and SRL are typically syntactic constituents, an assumption supported by quantitative results. The lack of syntactic constraints on the spans of interest leads to a waste of computational resources, as all  $\mathcal{O}(n^2)$  possible spans are enumerated by the model.
In this paper, we propose a structured span selector for span selection. Our span selector is a syntactically aware, weighted context-free grammar that learns to score partial, possibly nested span annotations. In the case of partial an
spans and make downstream decisions simultaneously (Lee et al., 2017; He et al., 2018).
For the OntoNotes dataset, He et al. (2018) reports that  $98.7\%$  of the arguments in SRL are constituents. For coreference, we find that  $99.1\%$  of the mentions are constituents.
<sup>1</sup>A software engineering metaphor, which refers to a reusable solution to a commonly occurring problem.
2Note that we also have joint models where we find optimal
Human Language Technologies, pages 2629 - 2641
notation, we marginalize out the missing structure and maximize the marginal log-likelihood. Fig. 2 illustrates an example partial parse of our WCFG and the difference between the traditional greedy approach and our approach.
We apply our span selector to both coreference and SRL. In both cases, we optimize the log-likelihood of the joint distribution defined by the span selector and the conditional for the downstream task as defined in Lee et al. (2017) and He et al. (2018). In contrast to previous approaches, which heavily rely on heuristics<sup>4</sup> to prune the set of spans to be considered, our span selector directly admits a tractable inference algorithm to determine the highest-scoring set of spans. We observe that the number of spans our model selects is significantly lower than the number of spans considered in previous works, resulting in a reduction in the memory required to train these models. Our approach leads to significant gains in both downstream tasks considered in this work: coreference and SRL. We find that our approach improves the performance of the end-to-end coreference model on the OntoNotes English dataset (0.4 F1) and the LitBank dataset (0.7 F1). On SRL, our model also achieves consistent gains over previous work.
2.1 Span Selection as a Design Pattern
Many NLP tasks involve reasoning over textual spans, e.g., coreference resolution, semantic role labeling, named entity recognition, and question answering. Models for these span prediction tasks often follow a common design pattern. They decompose into two components: (i) a span selection component where the model first selects a set of spans of interest, and (ii) a span prediction component where a prediction (e.g., entity or role assignment) is made for the chosen set of spans.
As shown in previous papers (Zhang et al., 2018; Wu et al., 2020), the quality of the span selector can have a large impact on the overall model performance. The span selector typically selects spans by selecting the start token and the end token in the span. Thus, there are inherently  $\binom{n}{2}$  textual spans, which is  $\mathcal{O}(n^2)$ , within a document of  $n$  tokens to choose from. Previous span selection models (Lee et al., 2017; He et al., 2018) enumerate all possible
Figure 2: An example partial parse in our WCFG for coreference resolution and the set of spans it corresponds to. Production rules that do not involve the "span of interest" non-terminal  $(\mathrm{X}_{\sigma})$  are skipped as they do not affect the parsing result (see §3.2). The traditional greedy approach considers  $\mathcal{O}(n^2)$  spans denoted by the grid cells unless some pruning heuristics are applied. While the number of nonterminals in a CNF parse is  $\mathcal{O}(n)$ .
spans in a brute-force manner and feed the greedily selected top- $k$  span candidates to the downstream prediction step.
However, several span selection tasks require spans that are syntactic constituents, which is a useful, but often neglected inductive bias in such tasks. For instance, in coreference resolution, mentions are typically noun phrases, pronouns, and sometimes, verbs. Similarly, in semantic role labeling, semantic arguments of a predicate are also typically syntactic constituents such as noun phrases, prepositional phrases, adverbs, etc. Our work uses a context-free grammar to enumerate spans. The number of valid syntactic constituents in a constituency tree of a sequence of length  $n$  is bounded by  $\mathcal{O}(n)$ , as the constituents can be viewed as the internal nodes of a binary parse tree in Chomsky normal form. Compared with brute-force enumeration and greedy pruning, this inductive bias provides us with a natural pruning strategy to reduce the number of candidate spans from  $\mathcal{O}(n^2)$  to  $\mathcal{O}(n)$  and provides us a more natural and linguistically informed way to model span-based tasks in NLP, through which we can employ parsing techniques and retrieve the optimal span selection for downstream tasks.
To further motivate our approach, we provide background on two popular span prediction tasks considered in our paper: coreference resolution and semantic role labeling (SRL). We also overview some previous papers on these tasks and contrast their methodology with ours. Finally, we describe how our model can work with partial span-level
$^{4}$ Lee et al. (2017) consider spans such that the maximum span width is 10, the number of spans per word is 0.4, etc. Then, the spans are greedily pruned.
annotations provided by datasets for these tasks.
2.2 Coreference Resolution
Most coreference resolution models involve two stages: mention detection and mention clustering. Traditional pipeline systems rely on parse trees and hand-engineered rules to extract mentions (Raghunathan et al., 2010). However, Lee et al. (2017) show that we can directly detect mentions as well as assign antecedents to them in an end-to-end manner.
In addition to this paper, other works have also explored better mention proposers for coreference. Zhang et al. (2018) use multi-task loss to directly optimize the mention detector. Swayamdipta et al. (2018) also leverage syntactic span classification as an auxiliary task to assist coreference. Thirukovaluru et al. (2021), Kirstain et al. (2021), and Dobrovolskii (2021) explore token-level representations to both reduce memory consumption and increase performance on longer documents. Miculicich and Henderson (2020) and Yu et al. (2020) both improve the mention detector with better neural network structures. Yet they still need to manually set a threshold to control the number of selected candidate mentions and none of them could produce an optimal span selection for the downstream task. Finkel and Manning (2009) situate NER in a parsing framework by explicitly incorporating named entity types into parse tree labels. In contrast, our work requires neither syntactic annotations nor hyperparameter tuning for mention selection.
2.3 Semantic Role labeling
Semantic role labeling (SRL) extracts relations between predicates and their arguments. Two major lines of work in SRL are sequence-tagging models (He et al., 2017; Marcheggiani et al., 2017) and span-based models (He et al., 2018; Ouchi et al., 2018; Li et al., 2019). Sequence tagging models for SRL convert semantic role annotations to BIO sequences. The tagger generates a label sequence for one single predicate at a time. However, span-based models generate the set of all candidate arguments in one forward pass and classify their semantic roles with regard to each predicate. As discussed in He et al. (2018), span-based models empirically perform better than sequence tagging models as they incorporate span-level features. Span-based models also do better at long-range dependencies as well as agreements with syntactic boundaries. Thus, we focus on span-based models in this work.
2.4 Nested and Partial Span Annotations
Nested mentions and partially annotated mentions are two major concerns in this paper. Most datasets for span prediction problems contain partial annotations of mentions. For example, singletons are not annotated in OntoNotes (Pradhan et al., 2012). In the coreference resolution example given in Fig. 1, the bracketed nested spans are annotated, while the underlined spans are valid mentions that are unannotated, since they do not co-refer with any other mention in the same document. The same is also true in SRL. In the SRL example in Fig. 1, there are two predicates (boxed words in the example) in one sentence. Their arguments are nested (i.e., ARG1 and ARG2 of the second predicate are located within ARG2 of the first predicate).
3 A Structured Span Selection Model
In this section, we develop the primary contribution of our paper: A new model for span selection. Specifically, we assert that almost all spans that a span selector should select are syntactic constituents; see Fig. 1 for two examples. Under this hypothesis, a context-free grammar (CFG) is a natural model for span selection as spans selected by a CFG cannot overlap, i.e., every pair of spans selected by a CFG would either be nested or disjoint.
We first start by introducing some basic terminology. We define a document  $D$  as a sequence of sentences  $\mathbf{w}_1, \ldots, \mathbf{w}_{|D|}$ . Each sentence  $\mathbf{w}$  in the document is a sequence of words  $[w_1, \ldots, w_{|\mathbf{w}|}]$ . A span is a contiguous subsequence of words in a sentence. For instance, we denote the span from position  $i$  to position  $k$ , i.e.,  $w_i \cdots w_k$ , as  $[i, k]$ .
3.2 Weighted Context-Free Grammars
Next, we define weighted context-free grammars (WCFG), the formalism that we will use to build our span selector. A WCFG is a five-tuple  $\langle \Sigma, N, S, R, \rho \rangle$ , where  $\Sigma$  is an alphabet of terminal symbols,  $N$  is a finite set of non-terminal symbols,  $S \in N$  is the unique start symbol,  $R$  is a set of production rules where a rule is of the form  $X \rightarrow \alpha$  where  $X \in N$  and  $\alpha \in (N \cup \Sigma)^*$ , and  $\rho: R \rightarrow \mathbb{R}_{\geq 0}$  is a scoring function that maps every production rule to a non-negative real number. We
An alphabet is a finite, non-empty set.
The scoring function can easily be generalized to map any production to a semiring value (Goodman, 1999).
say a WCFG is in Chomksy normal form (CNF) if every production rule has one of three forms:  $\mathrm{X}\rightarrow \mathrm{YZ}$ , where  $\mathrm{X,Y,Z}\in N$ ,  $\mathrm{X}\rightarrow x$ , where  $\mathrm{X}\in N$  and  $x\in \Sigma$ , or  $\mathrm{S}\rightarrow \varepsilon$ . For an input sentence  $\mathbf{w}$ , a WCFG defines a weighted set of parse trees, which we will denote  $\mathcal{T}(\mathbf{w})$ ; we drop the argument from  $\mathcal{T}$  when the sentence is clear from the context.
We overload the scoring function  $\rho$  to assign a weight to each parse tree  $t\in \mathcal{T}(\mathbf{w})$ . We define  $\rho$  applied to a tree as follows
$$
\rho (t) = \prod_ {r \in t} \rho (r) \geq 0 \tag {1}
$$
Given that  $\rho$  returns a non-negative weight, our WCFG can be used to define a distribution over the set of all parses of a sentence
$$
p (t \mid \mathbf {w}) = \frac {\rho (t)}{Z} \tag {2}
$$
where  $Z = \sum_{t\in \mathcal{T}(\mathbf{w})}\rho (t)$  is the sum of the scores of all parses. Using the familiar inside-outside algorithm (Baker, 1979), we can exactly compute  $Z$  in  $\mathcal{O}(|\mathbf{w}|^3)$  time.
3.3 A WCFG Span Selector
To convert from parse trees to spans, our paper exploits a simple fact: If our grammar is in CNF, then every parse tree  $t \in \mathcal{T}(\mathbf{w})$  corresponds to a unique set of labeled spans. Specifically, we write  $[i, X, k]$  iff the contiguous subsequence  $w_i \cdots w_k$  corresponds to a constituent rooted at  $X$  in  $t$ . We will denote the tree-to-span bijection as  $\mathsf{span}(\cdot)$  and write  $M_t = \mathsf{span}(t)$  to denote the set of spans  $t$  implies. We will also denote the set of all sets of spans viable under a CFG in CNF as  $\mathcal{M}(\mathbf{w})$ .
To extract spans useful for downstream tasks, we propose a simple WCFG. The grammar has three non-terminals  $N \stackrel{\mathrm{def}}{=} \{S, X_{\sigma}, X_{\overline{\sigma}}\}$  where  $S$  is the distinguished start symbol. A span rooted at non-terminal  $X_{\sigma}$ , denoted  $[i, X_{\sigma}, k]$ , is termed a span of interest; we will abbreviate  $[i, X_{\sigma}, k]$  as  $\sigma_{ik}$ . Likewise, a span rooted at non-terminal  $X_{\overline{\sigma}}$ , denoted  $[i, X_{\overline{\sigma}}, k]$ , is termed a span of non-interest; we will abbreviate  $[i, X_{\overline{\sigma}}, k]$  as  $\overline{\sigma}_{ik}$ . The full grammar is given in App. A.1. We define our weight function  $\rho: R \to \mathbb{R}_{\geq 0}$  as follows:
$$
\rho \left(_ {i} \mathrm {X} _ {k} \rightarrow_ {i} \mathrm {Y} _ {j j} \mathrm {Z} _ {k}\right) \stackrel {\text {d e f}} {=} \left\{\begin{array}{l l}\exp s _ {p} \left(\sigma_ {i k}\right)&\text {i f X = X}\\1&\text {o t h e r w i s e}\end{array}\right. \tag {3}
$$
where  $s_p$  is a learnable span-scorer that assigns a non-negative weight. Note this definition of  $\rho$  is an anchored scoring mechanism as it also makes use of the span indices  $i$  and  $k$ .
Under the simplified scoring function in Eq. (3), the score of a tree  $t$  can be re-expressed as a product of the scores of the spans of interest in  $\operatorname{span}(t)$ . Specifically, for any tree  $t$ , we have
$$
\rho (t) = \prod_ {r \in t} \rho (r) \tag {4}
$$
$$
= \exp \left(\sum_ {\sigma_ {i k} \in \operatorname {s p a n s} (t)} s _ {p} \left(\sigma_ {i k}\right)\right) \tag {5}
$$
$$
\stackrel {\text {d e f}} {=} \exp s _ {p} (M _ {t}) \tag {6}
$$
where  $M_{t} = \{\sigma_{ik}\mid \sigma_{ik}\in \mathsf{span}(t)\}$ . Note that the step from Eq. (4) to Eq. (5) follows from the fact that only those spans rooted at  $\mathrm{X}_{\sigma}$  have a weight other than 1 under our choice of  $\rho$  and, furthermore,  $\rho$  ignores the body of the context-free rule. The problem of mention detection is hence converted from subset selection to finding an optimal parse tree that maximizes the score:
$$
M ^ {*} = \underset {M _ {t} \in \mathcal {M} (\mathbf {w})} {\operatorname {a r g m a x}} s _ {p} \left(M _ {t}\right) \tag {7}
$$
$$
= \operatorname {s p a n s} \left(\underset {t \in \mathcal {T} (\mathbf {w})} {\operatorname {a r g m a x}} \rho (t)\right) \tag {8}
$$
where the "Viterbi version" of the CKY algorithm (see App. A.3), yields an exact algorithm for the argmax function in Eq. (8) in  $\mathcal{O}(|\mathbf{w}|^3)$  time.
Finally, in order to train our WCFG with only partial supervision, i.e., in the case when we do not observe the entire tree, we require the marginal probabilities of the spans of interest. First, let  $\mathcal{T}_{ik}$  be the set of parses that contain  $\sigma_{ik}$ . Then, the marginal probability of the span of interest  $\sigma_{ik}$  can be expressed as:
$$
p \left(\sigma_ {i k} \mid \mathbf {w}\right) = \sum_ {t \in \mathcal {T} _ {i k}} p (t \mid \mathbf {w}) = \sum_ {t \in \mathcal {T} _ {i k}} \frac {\rho (t)}{Z} \tag {9}
$$
As described by Eisner (2016), we can compute  $p(\sigma_{ik} \mid \mathbf{w})$  by computing the derivative of the log-normalizer  $\log Z$  with respect to  $s_p(\sigma_{ik})$ , i.e.,
$$
p \left(\sigma_ {i k} \mid \mathbf {w}\right) = \frac {\partial \log Z}{\partial s _ {p} \left(\sigma_ {i k}\right)} \tag {10}
$$
Automatic differentiation ensures that this marginal computation will have the same runtime as the computation of  $\log Z$  itself—to wit in  $\mathcal{O}(|\mathbf{w}|^3)$  time (Griewank and Walther, 2008).
The necessary and sufficient condition for this to hold is that the grammar has no unary or nullary rules, so every parse tree in the grammar can be written as a bracketed string.
4 Adaptations to Downstream Tasks
In this section, we introduce how our structured span selector can be applied in an end-to-end manner to coreference resolution and SRL.
4.1 Coreference Resolution
The goal of coreference resolution is to link a span of interest  $\sigma_{ij}$ , termed a mention in the context of coreference resolution, to its antecedent. Note that the antecedent is either another mention in the same document or the dummy antecedent, $^{8}$ , which we denote as  $\epsilon$ . We write  $\sigma_{ij} \rightsquigarrow \sigma_{k\ell}$  to denote that  $\sigma_{k\ell}$  is  $\sigma_{ij}$ 's antecedent. When formulating coreference in a probabilistic manner, we have the following natural decomposition:
$$
\begin{array}{l} p \left(\sigma_ {i j}, \sigma_ {i j} \sim \rightarrow \sigma_ {k \ell}\right) \tag {11} \\ = \underbrace {p (\sigma_ {i j})} _ {\text {p r .} \sigma_ {i j} \text {i s a m e n t i o n}} \times \underbrace {p \left(\sigma_ {i j} \rightsquigarrow \sigma_ {k \ell} \mid \sigma_ {i j}\right)} _ {\text {p r .} \sigma_ {i j} \text {s a n t e c e d e n t i s} \sigma_ {k \ell}} \\ \end{array}
$$
The support of the above distribution is  $\mathcal{X} \times \mathcal{Y}_{ij}$ , where  $\mathcal{X}$  is the set of all possible textual spans in  $D$ ,  $\mathcal{Y}_{ij} = \{\sigma_{k\ell} \mid j < \ell\} \cup \{\epsilon\}$  is the set of mentions preceding  $\sigma_{ij}$  plus the dummy antecedent  $\epsilon$  for every  $\sigma_{ij} \in \mathcal{X}$ . In words, the above decomposition means that the probability of  $\sigma_{ij}$  co-referring with the span  $\sigma_{k\ell}$  is the probability of first recognizing that  $\sigma_{ij}$  is itself a mention and then determining the link. In practice, this decomposition means that modelers can select  $p(\sigma_{ij})$  and  $p(\sigma_{ij} \rightsquigarrow \sigma_{k\ell} \mid \sigma_{ij})$  according to their taste and, importantly, independently of each other. In this work, we explore treating  $p(\sigma_{ij})$  as the WCFG span selector described in §3.2 and  $p(\sigma_{ij} \rightsquigarrow \sigma_{k\ell} \mid \sigma_{ij})$  as Lee et al. (2017)'s popular span ranking model for coreference.
Lee et al. (2017) as a Mention-Linker. We now describe the mention-linker in Lee et al. (2017). We define the mention-linker distribution as
$$
\begin{array}{l} p \left(\sigma_ {i j} \rightsquigarrow m \mid \sigma_ {i j}\right) \tag {12} \\ = \frac {\exp s (\sigma_ {i j} , m)}{\sum_ {m ^ {\prime} \in \mathcal {Y} _ {i j}} \exp s (\sigma_ {i j} , m ^ {\prime})} \\ \end{array}
$$
The scoring function  $s(\cdot, \cdot)$  is defined in two cases: One case for  $m = \epsilon$ , the dummy antecedent, and one for  $m = \sigma_{k\ell}$ , a preceding span:
$$
\begin{array}{l} s \left(\sigma_ {i j}, \epsilon\right) = 0 \tag {13} \\ s (\sigma_ {i j}, \sigma_ {k \ell}) = s _ {m} (\sigma_ {i j}) + s _ {m} (\sigma_ {k \ell}) + s _ {a} (\sigma_ {i j}, \sigma_ {k \ell}) \\ \end{array}
$$
The first score function,  $s_m(\sigma_{ij})$ , is a score for span  $[i,j]$  being a mention. The second function,  $s_a(\sigma_{ij},\sigma_{k\ell})$ , a score that  $\sigma_{k\ell}$  is an antecedent of  $[i,j]$ . In this work, both  $s_a$  and  $s_m$  are computed by neural networks that take span representations as inputs. However, in principle, they could be computed by any model.
Training. Lee et al.'s model adopts a naive greedy algorithm by taking the top  $\lambda | D|$  spans with the highest mention scores  $s_m$ , where  $\lambda$  is a hyperparameter that has to be manually defined for different datasets. However, finding a proper ratio  $\lambda$  can be very tricky. In contrast, in our setting we can optimize the final objective function which is the log-likelihood of the joint distribution defined at the beginning of §4.1:
$$
L _ {1} = \sum_ {\mathbf {w} \in D} \sum_ {\sigma_ {i j} \in \mathcal {G} _ {\mathbf {w}}} \log \sum_ {m \in \mathcal {G} _ {i j}} p \left(\sigma_ {i j}, \sigma_ {i j} \rightsquigarrow m\right) \tag {14}
$$
where  $\mathcal{G}$  is the (partially) annotated set of mentions,  $\mathcal{G}_{\mathbf{w}}$  is the set of all textual spans of  $\mathbf{w}$  in  $\mathcal{G}$ , and  $\mathcal{G}_{ij}$  is the ground truth cluster that  $\sigma_{ij}$  belongs to.
Handling partial annotation with no singletons. Since in many coreference datasets, e.g., OntoNotes, only the mentions that are referred to more than once are annotated, learning a mention detector from such data requires the ability to handle the lack of singleton annotations. To handle partial span annotations, we marginalize out the unannotated singletons. This results in the following marginal log-likelihood
$$
\begin{array}{l} L _ {2} = \sum_ {\mathbf {w} \in D} \sum_ {\sigma_ {i j} \in \overline {{\mathcal {G}}} _ {\mathbf {w}}} \log \left(p \left(\sigma_ {i j} \rightsquigarrow \epsilon \mid \sigma_ {i j}\right) p \left(\sigma_ {i j}\right)\right. \\ \left. + \left(1 - p \left(\sigma_ {i j}\right)\right)\right) \tag {15} \\ \end{array}
$$
We optimize the loss  $L = L_{1} + L_{2}$  jointly. Here,  $\overline{\mathcal{G}}_{\mathbf{w}}$  denotes the set of all spans of  $\mathbf{w}$  not in  $\mathcal{G}$ .