SlowGuess/NAACL_2021_OCR huggingface dataset:
Tianze Shi
oirsoy@bloomberg.net
llee@cs.cornell.edu
Naturally-occurring bracketings, such as answer fragments to natural language questions and hyperlinks on webpages, can reflect human syntactic intuition regarding phrasal boundaries. Their availability and approximate correspondence to syntax make them appealing as distant information sources to incorporate into unsupervised constituency parsing. But they are noisy and incomplete; to address this challenge, we develop a partial-brackets-aware structured ramp loss in learning. Experiments demonstrate that our distantly-supervised models trained on naturally-occurring bracketing data are more accurate in inducing syntactic structures than competing unsupervised systems. On the English WSJ corpus, our models achieve an unlabeled F1 score of 68.9 for constituency parsing. $^{1}$
Constituency is a foundational building block for phrase-structure grammars. It captures the notion of what tokens can group together and act as a single unit. The motivating insight behind this paper is that constituency may be reflected in markups of bracketings that people provide in doing natural tasks. We term these segments naturally occurring bracketings for their lack of intended syntactic annotation. These include, for example, the segments people pick out from sentences to refer to other Wikipedia pages or to answer semantically-oriented questions; see Figure 1 for an illustration.
Gathering such data requires low annotation expertise and effort. On the other hand, these data are not necessarily suitable for training parsers, as they often contain incomplete, incorrect and sometimes conflicting bracketing information. It is thus an empirical question whether and how much we
Science fiction (sometimes shortened to sci-fi or SF) is a genre of speculative fiction that typically deals with imaginative and futuristic concepts such as advanced science and technology, space exploration, time travel, parallel universes, and extraterrestrial life.
QA-SRL
Republicans have been imploring the White House to compromise on the wage issue.
Q&As:
Q: Who have been imploring something?
A: Republicans
Q: What have someone been imploring?
A: To compromise on the wage issue
Q: What will someone compromise on?
A: The wage issue
Corresponding bracketsings:
[ Republicans ] have been imploring the White House [ to compromise on [ the wage issue ] ].
Figure 1: Two example types of naturally-occurring bracketings. Blue underlined texts in the Wikipedia sentence are hyperlinks. We bracket the QA-SRL sentence in matching colors according to the answers.
could learn syntax from these naturally-occurring bracketing data.
To overcome the challenge of learning from this kind of noisy data, we propose to train discriminative constituency parsers with structured ramp loss (Do et al., 2008), a technique previously adopted in machine translation (Gimpel and Smith, 2012). Specifically, we propose two loss functions to directly penalize predictions in conflict with available partial bracketing data, while allowing the parsers to induce the remaining structures.
We experiment with two types of naturally-occurring bracketing data, as illustrated in Figure 1. First, we consider English question-answer pairs collected for semantic role labeling (QA-SRL; He et al., 2015). The questions are designed for non
1Our code is publicly available at https://github. com/tzshi/nob-naacl21.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2941-2949 June 6-11, 2021. ©2021 Association for Computational Linguistics
experts to specify semantic arguments of predicates in the sentences. We observe that although no syntactic structures are explicitly asked for, humans tend to select constituents in their answers. Second, Wikipedia articles $^2$  are typically richly annotated with internal links to other articles. These links are marked on phrasal units that refer to standalone concepts, and similar to the QA-SRL data, they frequently coincide with syntactic constituents.
Experiment results show that naturally-occurring bracketings across both data sources indeed help our models induce syntactic constituency structures. Training on the QA-SRL bracketing data achieves an unlabeled F1 score of 68.9 on the English WSJ corpus, an accuracy competitive with state-of-the-art unsupervised constituency parsers that do not utilize such distant supervision data. We find that our proposed two loss functions have slightly different interactions with the two data sources, and that the QA-SRL and Wikipedia data have varying coverage of phrasal types, leading to different error profiles.
In sum, through this work, (1) we demonstrate that naturally-occurring bracketings are helpful for inducing syntactic structures, (2) we incorporate two new cost functions into structured ramp loss to train parsers with noisy bracketings, and (3) our distantly-supervised models achieve results competitive with the state of the art of unsupervised constituency parsing despite training with smaller data size (QA-SRL) or out-of-domain data (Wikipedia).
2 Naturally-Occurring Bracketings
Constituents are naturally reflected in various human cognitive processes, including speech production and perception (Garrett et al., 1966; Gee and Grosjean, 1983), reading behaviors (Hale, 2001; Boston et al., 2008), punctuation marks (Spitkovsky et al., 2011), and keystroke dynamics (Plank, 2016). Conversely, these externalized signals help us gain insight into constituency representations. We consider two such data sources:
a) Answer fragments When questions are answered with fragments instead of full sentences, those fragments tend to form constituents. This phenomenon corresponds to a well-established constituency test in the linguistics literature (Carnie, 2012, pg. 98, inter alia).
Dataset	QA-SRL	Wikipedia
Number of sentences	1,241	926,077
Brackets/sentence	6.26	0.89
Single word	22.4%	35.8%
Constituent in reference	55.2%	31.1%
Conflicting w/ reference	11.8%	5.3%
SBAR	2.8%	0.07%
NP	36.8%	4.42%
VP	6.3%	0.07%
PP	13.3%	0.04%
ADJP	8.6%	1.48%
ADVP	30.5%	0.39%
Total	21.8%	1.91%
Table 1: Dataset statistics: number of bracketings per sentence (top), percentage of bracketing types (middle), and the reference phrases per label found in the natural bracketings (bottom). Conflicting means the bracket crosses some reference span. Reference parses for Wikipedia are generated by a parser trained on PTB.
b) Webpage hyperlinks Since a hyperlink is a pointer to another location or action (e.g., mailto: links), anchor text often represents a conceptual unit related to the link destination. Indeed, Spitkovsky et al. (2010) first give empirical evidence that around half of the anchor text instances in their data respects constituent boundaries and Søgaard (2017) demonstrates that hyperlink data can help boost chunking accuracy in a multi-task learning setup.
Both types of data have been considered in previous work on dependency-grammar induction (Spitkovsky et al., 2010; Naseem and Barzilay, 2011), and in this work, we explore their efficacy for learning constituency structures.
For answer fragments, we use He et al.'s (2015) question-answering-driven semantic role labeling (QA-SRL) dataset, where annotators answer wh-questions regarding predicates in sentences drawn from the Wall Street Journal (WSJ) section of the Penn Treebank (PTB; Marcus et al., 1993). For hyperlinks, we used a  $1\%$  sample of 2020-05-01 English Wikipedia, retaining only within-Wikipedia links.
We compare our extracted naturally-occurring bracketings with the reference phrase-structure annotations.4 Table 1 gives relevant statistics. Our results re-affirm Spitkovsky et al.'s (2010) finding that a large proportion of hyperlinks coin
See Appendix A for details.
4For "ground-truth" structures in the Wikipedia data, we apply a state-of-the-art PTB-trained constituency parser (Kitaev et al., 2019).
2We worked with articles in English.
cide with syntactic constituents. We also find that  $22.4\% / 35.8\%$  of the natural bracketings are single-word spans, which cannot facilitate parsing decisions, while  $11.8\% / 5.3\%$  of QA-SRL/Wikipedia spans actually conflict with the reference trees and can thus potentially harm training. The QA-SRL data seems more promising for inducing better-quality syntactic structures, as there are more bracketings available across a diverse set of constituent types.
3 Parsing Model
Preliminaries The inputs to our learning algorithm are tuples  $(w,B)$ , where  $w = w_{1},\ldots ,w_{n}$  is a length- $n$  sentence and  $B = \{(b_k,e_k)\}$  is a set of naturally-occurring bracketings, denoted by the beginning and ending indices  $b_{k}$  and  $e_k$  into the sentence  $w$ . As a first step, we extract BERT-based contextualized word representations (Devlin et al., 2019) to associate each token  $w_{i}$  with a vector  $\mathbf{x}_i$ . See Appendix B for details.
Scoring Spans Based on the  $\mathbf{x}_i$  vectors, we assign a score  $s_{ij}$  to each candidate span  $(i,j)$  in the sentence indicating its appropriateness as a constituent in the output structure. We adopt a biaffine scoring function (Dozat and Manning, 2017):
$$
s _ {i j} = [ \mathbf {l} _ {i}; 1 ] ^ {T} W [ \mathbf {r} _ {j}; 1 ],
$$
where  $[\mathbf{v};1]$  appends 1 to the end of vector  $\mathbf{v}$ , and
$$
\mathbf {l} _ {i} = \mathrm {M L P} ^ {\mathrm {l e f t}} (\mathbf {x} _ {i}) \quad \mathrm {a n d} \quad \mathbf {r} _ {j} = \mathrm {M L P} ^ {\mathrm {r i g h t}} (\mathbf {x} _ {j})
$$
are the outputs of multi-layer perceptrons (MLPs) that take the vectors at span boundaries as inputs.6
Decoding We define the score  $s(y)$  of a binary-branching constituency tree  $y$  to be the sum of scores of its spans. The best scoring tree among all valid trees  $\mathcal{V}$  can be found using the CKY algorithm (Cocke, 1969; Kasami, 1965; Younger, 1967).
Learning Large-margin training (Taskar et al., 2005) is a typical choice for supervised training of constituency parsers. It defines the following
loss function to encourage a large margin of at least  $\Delta (y,y^{*})$  between the gold tree  $y^{*}$  and any predicted tree  $y$ :
$$
l = \max  _ {y \in \mathcal {Y}} [ s (y) + \Delta (y, y ^ {*}) ] - s (y ^ {*}),
$$
where  $\Delta (y,y^{*})$  is a distance measure between  $y$  and  $y^{*}$ . We can reuse the CKY decoder for cost-augmented inference when the distance decomposes into individual spans with some function  $c$ :
$$
\Delta (y, y ^ {*}) = \sum_ {\text {s p a n} (i, j) \text {i n} y} c (i, j, y ^ {*}).
$$
In our setting, we do not have access to the gold-standard  $y^{*}$ , but instead we have a set of bracketings  $\tilde{y}$ . The scoring  $s(\tilde{y})$  is not meaningful since  $\tilde{y}$  is not a complete tree, so we adopt structured ramp loss (Do et al., 2008; Gimpel and Smith, 2012) and define
$$
\begin{array}{l} l = \left(\max  _ {y \in \mathcal {Y}} [ s (y) + \Delta (y, \tilde {y}) ] - s (\tilde {y})\right) \\ + \left(s (\tilde {y}) - \max  _ {y \in \mathcal {Y}} [ s (y) - \Delta (y, \tilde {y}) ]\right) \\ = \max  _ {y \in \mathcal {Y}} [ s (y) + \Delta (y, \tilde {y}) ] \\ - \max  _ {y \in \mathcal {Y}} [ s (y) - \Delta (y, \tilde {y}) ], \\ \end{array}
$$
using a combination of cost-augmented and cost-diminished inference. This loss function can be understood as a sum of a convex and a concave large margin loss (Collobert et al., 2006), canceling out the term for directly scoring the gold-standard tree. We consider two methods for incorporating the partial bracketings into the cost functions:
$$
c _ {\text {l o o s e}} (i, j, \tilde {y}) = \mathbb {1} (\text {s p a n} (i, j) \text {c o n f l i c t s w i t h} \tilde {y})
$$
$$
c _ {\text {s t r i c t}} (i, j, \tilde {y}) = \mathbb {1} (\text {s p a n} (i, j) \text {n o t i n} \tilde {y}),
$$
where  $\mathbb{1}$  is an indicator function.  $c_{\mathrm{loose}}$  is more lenient than  $c_{\mathrm{strict}}$  as it does not penalize spans that do not conflict with  $\tilde{y}$ . Both cost definitions promote structures containing bracketings in  $\tilde{y}$ . In the supervised setting where  $\tilde{y}$  refers to a fully-annotated tree  $y^{*}$  without conflicting span boundaries,  $c_{\mathrm{strict}}$  is equal to  $c_{\mathrm{loose}}$  and the resulting  $\Delta(y, y^{*})$  cost functions both correspond to the Hamming distance between  $y$  and  $y^{*}$ .
Data and Implementation We evaluate on the PTB (Marcus et al., 1993) with the standard splits
5The use of pre-trained language models can mitigate the fact that our distant supervision data are either out-of-domain (Wikipedia) or small in size (QA-SRL).
This is inspired by span-based supervised constituency-parsing methods (Stern et al., 2017), which in turn was based on Wang and Chang (2016). These papers look at the difference vectors between two boundary points, while our scoring function directly uses the vectors at the boundaries (which is more expressive than only using difference vectors).
One may also consider a linear interpolation of  $c_{\mathrm{loose}}$  and  $c_{\mathrm{strict}}$ , but that would introduce an additional hyper-parameter.
Model	PLM	Mean	Max
Random Trees	19.2	19.5
Left Branching	8.7
Right Branching	39.5
Upper bound	84.3
URNNG (Kim et al., 2019b)	—	45.4
PRPN (Shen et al., 2018)	47.3	47.9
ON (Shen et al., 2019)	48.1	50.0
DIORA (Drozdov et al., 2019)	○	—	58.9
CPCFG (Kim et al., 2019a)	55.2	60.1
S-DIORA (Drozdov et al., 2020)	○	57.6	64.0
Constituency Tests (Cao et al., 2020)	●	62.8	65.9
+URNNG (Cao et al., 2020)	●	67.9	71.3
This work:
\( NOB_{QA-SRL}\), \( c_{loose} \)	●	64.5	65.2
\( NOB_{QA-SRL}\), \( c_{strict} \)	●	68.9	70.0
\( NOB_{Wikipedia}\), \( c_{loose} \)	●	58.2	63.0
\( NOB_{Wikipedia}\), \( c_{strict} \)	●	56.1	57.0
Table 2: Sentence-level unlabeled F1 scores  $(\%)$  on the WSJ test set.  $\bullet$  in the PLM column denotes the use of context-sensitive pre-trained language models;  $\circ$  uses context-insensitive embeddings from PLMs. Methods producing only binary-branching structures (including everything in this table) have an upperbound of  $84.3\%$  F1 score, since the gold trees can be non-binary.
Const. Type	Cao et al. (2020)	\( NOB_{QA-SRL} \)	\( NOB_{Wikipedia} \)
\( C_{loose} \)	\( C_{strict} \)	\( C_{loose} \)	\( C_{strict} \)
SBAR	85.3	89.0	87.7	66.7	48.3
NP	84.3	85.5	85.2	71.8	70.3
VP	80.8	52.3	70.9	62.4	49.6
PP	84.4	83.5	86.5	67.7	74.8
ADJP	55.6	58.1	57.3	62.7	60.1
ADVP	54.6	76.9	75.3	66.4	63.9
Table 3: Average recall  $(\%)$  per constituent type.
section 23 as the test set). QA-SRL contains 1,241 sentences drawn from the training split (sections 02-21) of the PTB. For Wikipedia, we use a sample of 332,079 sentences that are within 100 tokens long and contain multi-token internal hyperlinks. We fine-tune the pretrained  $\mathrm{BERT}_{\mathrm{base}}$  features with a fixed number of mini-batch updates and report results based on five random runs for each setting. See Appendix B for detailed hyper-parameter settings and optimization procedures.
Evaluation We follow the evaluation setting of Kim et al. (2019a). More specifically, we discard punctuation and trivial spans (single-word and full-sentence spans) during evaluation and report sentence-level F1 scores as our main metrics.
Results Table 2 shows the evaluation results of our models trained on naturally-occurring bracketings (NOB); Table 3 breaks down the recall
ratios for each constituent type. Our distantly-supervised models trained on QA-SRL are competitive with the state-of-the-art unsupervised results. When comparing our models with Cao et al. (2020), we obtain higher recalls on most constituent types except for VPs. Interestingly, QA-SRL data prefers  $c_{\text{strict}}$ , while  $c_{\text{loose}}$  gives better F1 score on Wikipedia; this correlates with the fact that QA-SRL has more bracketings per sentence (Table 1). Finally, our Wikipedia data has a larger relative percentage of ADJP bracketings, which explains the higher ADJP recall of the models trained on Wikipedia, despite their lower overall recalls.
Unsupervised Parsing Our distantly-supervised setting is similar to unsupervised in the sense that it does not require syntactic annotations. Typically, lack of annotations implies that unsupervised parsers induce grammar from a raw stream of lexical or part-of-speech tokens (Clark, 2001; Klein, 2005) along with carefully designed inductive biases on parameter priors (Liang et al., 2007; Wang and Blunsom, 2013), language universals (Naseem et al., 2010; Martínez Alonso et al., 2017), cross-linguistic (Snyder et al., 2009; Berg-Kirkpatrick and Klein, 2010; Cohen and Smith, 2009; Han et al., 2019) and cross-modal (Shi et al., 2019) signals, structural constraints (Gillenwater et al., 2010; Noji et al., 2016; Jin et al., 2018), etc. The models are usually generative and learn from (re)constructing sentences based on induced structures (Shen et al., 2018, 2019; Drozdov et al., 2019; Kim et al., 2019a,b). Alternatively, one may use reinforcement learning to induce syntactic structures using rewards defined by end tasks (Yogatama et al., 2017; Choi et al., 2018; Havrylov et al., 2019). Our method is related to learning from constituency tests (Cao et al., 2020), but our use of bracketing data permits discriminative parsing models, which focus directly on the syntactic objective.
Learning from Partial Annotations Full syntactic annotations are costly to obtain, so the alternative solution of training parsers from partially-annotated data has attracted considerable research attention, especially within the context of active learning for dependency parsing (Sassano, 2005; Sassano and Kurohashi, 2010; Mirroshandel and Nasr, 2011; Flannery et al., 2011; Flannery and Mori, 2015; Li et al., 2016; Zhang et al., 2017) and grammar induction for constituency parsing
(Pereira and Schabes, 1992; Hwa, 1999; Riezler et al., 2002). These works typically require expert annotators to generate gold-standard, though partial, annotations. In contrast, our work considers the setting and the challenge of learning from noisy bracketing data, which is more comparable to Spreyer and Kuhn (2009) and Spreyer et al. (2010) on transfer learning for dependency parsing.
We argue that naturally-occurring bracketings are a rich resource for inducing syntactic structures. They reflect human judgment of what constitutes a phrase and what does not. More importantly, they require low annotation expertise and effort; for example, webpage hyperlinks can be extracted essentially for free. Empirically, our models trained on QA-SRL and Wikipedia bracketings achieve competitive results with the state of the art on unsupervised constituency parsing.
Structural probes have been successful in extracting syntactic knowledge from frozen-weight pre-trained language models (e.g., Hewitt and Manning, 2019), but they still require direct syntactic supervision. Our work shows that it is also feasible to retrieve constituency trees from BERT-based models using distant supervision data.
Our models are limited to the unlabeled setting, and we leave it to future work to automatically cluster the naturally-occurring bracketings and to induce phrase labels. Our work also points to potential applications in (semi-)supervised settings including active learning and domain adaptation (Joshi et al., 2018). Future work can also consider other naturally-occurring bracketings induced from sources such as speech production, reading behavior, etc.
We thank the anonymous reviewers for their constructive reviews. This work was supported in part by a Bloomberg Data Science Ph.D. Fellowship to Tianze Shi and a gift from Bloomberg to Lillian Lee.
Taylor Berg-Kirkpatrick and Dan Klein. 2010. Phylogenetic grammar induction. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1288-1297, Uppsala, Sweden. Association for Computational Linguistics.
Marisa Ferrara Boston, John Hale, Reinhold Kliegl, Umesh Patil, and Shravan Vasishth. 2008. Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research, 2(1):1-12.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Unsupervised parsing via constituency tests. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4798-4808, Online. Association for Computational Linguistics.
Andrew Carnie. 2012. Syntax: A Generative Introduction, third edition. Number 4 in Introducing Linguistics. Wiley-Blackwell, Hoboken, New Jersey.
Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In Proceedings of The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 5094-5101, New Orleans, Louisiana, USA. Association for the Advancement of Artificial Intelligence.
Alexander S. Clark. 2001. Unsupervised Language Acquisition: Theory and Practice. Ph.D. thesis, University of Sussex.
John Cocke. 1969. Programming languages and their compilers: Preliminary notes. Technical report, Courant Institute of Mathematical Sciences, New York University.
Shay Cohen and Noah A. Smith. 2009. Shared logistic normal distributions for soft parameter tying in unsupervised grammar induction. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 74-82, Boulder, Colorado, USA. Association for Computational Linguistics.
Ronan Collobert, Fabian Sinz, Jason Weston, and Léon Bottou. 2006. Trading convexity for scalability. In Proceedings of the 23rd International Conference on Machine Learning, pages 201-208, Pittsburgh, Pennsylvania, USA. Association for Computing Machinery.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Chuong B. Do, Quoc V. Le, Choon H. Teo, Olivier Chapelle, and Alex J. Smola. 2008. Tighter bounds for structured estimation. In Advances in Neural Information Processing Systems 21, pages 281-288, Vancouver, Canada. Curran Associates, Inc.