Unified Syntactic Annotation of English in the CGEL Framework
Year: 2023

Abstract:
We investigate whether the Cambridge Grammar of the English Language (2002) and its extensive descriptions work well as a corpus annotation scheme. We develop annotation guidelines and in the process outline some interesting linguistic uncertainties that we had to resolve. To test the applicability of CGEL to real-world corpora, we conduct an interannotator study on sentences from the English Web Treebank, showing that consistent annotation of even complex syntactic phenomena like gapping using the CGEL formalism is feasible. Why introduce yet another formalism for English syntax? We argue that CGEL is attractive due to its exhaustive analysis of English syntactic phenomena, its labeling of both constituents and functions, and its accessibility. We look towards expanding CGELBank and augmenting it with automatic conversions from existing treebanks in the future.

heading: 1 Introduction
text: Ask a linguist about a detail of English grammar, and chances are they will reach for the Cambridge Grammar of the English Language (CGEL; Huddleston and Pullum, 2002). The product of the labors of two editors and 13 other chapter authors over more than a decade, CGEL is the most recent comprehensive reference grammar of English, describing nearly every syntactic facet of present-day Standard English in its 1700+ pages (Culicover, 2004). As but one example, a section1 is devoted to the form and function of sentences like the first sentence of this paragraph, where the part before and is grammatically imperative but interpreted as a condition, and the part after and is interpreted as a consequence. CGEL is a gold mine for such idiosyncrasies that a sharp-eyed English student (or linguist, or treebanker) might want to look up, in bottom-up fashion. It is also a systematic top-down
1“Imperatives interpreted as conditionals” (pp. 937–939)
survey of the building blocks of the language—in this respect, aided by a lucid companion textbook (SIEG2; Huddleston et al., 2021).
In a review for Computational Linguistics, Brew (2003) argued that CGEL is a descriptive reference that echos precise formal thinking about grammatical structures; and as such, it holds considerable relevance for computational linguistics, supplementing formal grammars and treebanks like the venerable Penn Treebank (PTB; Marcus et al., 1994). Brew exhorts: “it should become a routine part of the training of future grammar writers and treebank annotators that they absorb as much as is feasible of this grammar”. It has certainly had an impact, for example, on the Universal Dependencies project (UD; Nivre et al., 2016, 2020; de Marneffe et al., 2021), whose annotation guidelines cite CGEL many times in discussing particular phenomena2 (though the UD trees themselves, for reasons of lexicalism and panlingualism, diverge significantly from the representations given in CGEL).
We ask: What would it take to develop an annotation scheme based on CGEL? If CGEL’s attention to terminological precision and rigor is as strong as Brew suggests, it should not be nearly as difficult as mounting an effort of a completely new annotation framework. Most substantive questions of grammatical analysis should be addressed by CGEL, leaving only minor points to flesh out for treebanking. On the other hand, because CGEL was not designed for annotation, and therefore not tested systematically on corpora, perhaps it has substantial holes, regularly missing constructions that occur in real data.
To answer this question, we bootstrap an annotation manual and small set of sentences based on the
2References to CGEL can be found, for example, at https://universaldependencies.org/u/overview/ complex-syntax.html (regarding content clauses and secondary predicates) and https://universaldependencies. org/u/overview/specific-syntax.html (regarding comparative constructions).
220
descriptions from CGEL. We examine what blanks in the CGEL specifications need to be filled in to realize full-sentence trees in our data—both qualitatively (through working sentence-by-sentence) and quantitatively (by conducting an interannotator agreement study).
What practical benefits hinge on the answer to this question? We are cognizant that substantial English treebanks already exist—constituency treebanks following the Penn Treebank standard, dependency treebanks, and others (§2). Thus, we do not anticipate a significant amount of from-scratch annotation in the CGEL framework. Yet we see practical benefits of the CGEL style of description, perhaps induced automatically as a new “view” of gold PTB trees. First, exhaustiveness: CGEL trees systematize both constituent categories and functions in a unified framework, whereas mainstream approaches for English prioritize either constituent structure (like PTB) or dependency structure (like UD). And second, accessibility: the trees would be consistent with human-readable descriptions and linguistic argumentation in the CGEL and SIEG2 texts, allowing users of a treebank (or parser) to look up the constructions in question.3
Through developing guidelines and annotating data, we find that CGEL offers a powerful foundation for treebanking, though there are points where further specification is needed. Our small but growing treebank—which we call CGELBank—and accompanying code for validation and measuring interannotator agreement are available at https: //github.com/nert-nlp/cgel/. We also publish our annotation manual, which stands at about 75 pages (mostly of example trees): Reynolds et al. (2023).

heading: 2 Related Work
text: Even considering just English, there have been many formalisms deployed for syntactic annotation. A sample is given in Table 1. Each formalism makes different theoretical claims (e.g., is deep structure distinct from surface structure?) which bring computational tradeoffs (e.g. complexity vs. parsing efficiency). Many, beginning with
3PTB has an extensive annotation manual (Bies et al., 1995), but that serves a different purpose from a reference grammar: an annotation manual is a set of policies for an expert reader, not a complete presentation of syntactic phenomena or a defense of design decisions. Moreover, the terminology in the PTB manual draws heavily from particular syntactic theories like Government and Binding, whereas CGEL employs more general descriptive terminology.
PTB, have been used to annotate the Wall Street Journal corpus (WSJ; Marcus et al., 1993). CGEL shares ideas with many treebanks, such as constituency structure (PTB, TAG, etc.), labelled dependency relations (SD, UD, etc.), gapping (PTB), among other features.
A corpus that likewise integrates constituent categories and functions in a single tree is the TIGER treebank for German (Brants et al., 2004).

heading: 3 The CGEL Framework
text: An example parse in the CGEL framework appears in Figure 1.4 Its building blocks are constituents, each of which receives a category indicating the type of unit it is and a function (notated with a colon) indicating its grammatical relation within the higher constituent. The constituent structure is a hierarchical bracketing of the sentence, which is projective with respect to the order of words in the sentence. Terminals consist of lexical items (omitting punctuation) as well as gaps used to handle constructions with noncanonical word order. Categories. CGEL posits a distributionallydefined set of lexical categories, on which basis we developed a part-of-speech tagset with 11 tags: N (noun), Npro (pronoun), V (verb), Vaux (auxiliary verb), P (preposition), D (determinative),5 Adj (adjective), Adv (adverb), Sdr (subordinator), Coordinator, and Int (interjection). (See
4See Appendix C for more examples and a comparison with PTB.
5In CGEL, determinative is a lexical category whereas determiner is a function within an NP. A determinative heads
(Reynolds et al., 2022) for further details and comparison to PTB/UD tagsets, especially regarding P and D.) Pronouns and proper nouns are a subset of nouns, though we have created a distinct tag for pronouns; auxiliary verbs are a subset of verbs. All of these categories except subordinator and coordinator project higher-level phrasal constituents, e.g. N← Nom (nominal)← NP (noun phrase). The basic phrasal categories are: Nom, NP, VP, Clause (the various subtypes of which are unmarked here except Clauserel for relative clauses), PP, DP, AdjP, AdvP, and IntP. Phrases are typically binary- or unary-branching, but n-ary branches are also possible. There is also a non-phrasal constituent category: Coordination, which may have ternary branching or higher.
Functions. Each constituent has a function indicating its syntactic role in the higher constituent. A phrasal constituent is headed, i.e. it has exactly one child in Head function along with zero or more dependents. Coordination constituents are the main exception: there is no head, and each element (conjunct) in the coordination receives a function of Coordinate. Figure 2 illustrates the main CGEL functions, organized into a hierarchy. Note that CGEL contrasts adjuncts (Mod, Supplement) with complements (Comp and subtypes, including Subj, Obj, PredComp, and others). Other dependent functions include Determiner (Det) function in an NP, and Marker for grammatical words that mark but do not head a phrase, notably coordinators and subordinators.
a determinative phrase (DP), not to be confused with the notion of a determiner phrase in generative grammar (Abney, 1987).
Gaps. CGEL employs gap constituents and coindexation, as in Figure 1, to handle unbounded dependency constructions (UDCs) and other constructions that deviate from the canonical declarative order, showing where there is a clear structural gap. Nevertheless “the account is quite informal” (R. Huddleston, personal communication). To make it more formal, we have restricted the use of gaps to UDCs, subject-auxiliary inversion (SAI), and pre- and post-posing of complements. We also use gaps for adjunct fronting when it triggers SAI (e.g., Only once had I – seen it –). Subject–dependent inversion (SDI) is a double-gapped construction with a subject gap and a complement gap in the VP (e.g., Here – is – Jim). All subject-relatives have a gap, as do delayed right constituent coordination and end-attachment coordination. Coindexation is used with and only with a gap. Every gap must be coindexed with exactly one overt constituent (and possibly other gaps). There are no gaps for ellipsis. Fusion. Certain constructions are analyzed with fusion of functions, in which a constituent participates in two different higher constituents (Payne et al., 2007; Pullum and Rogers, 2008). This is shown in Figure 1 for the NP which, short for something like “which items”: the DP is taken to fulfill both the Determiner function in the NP and the Head in its Nominal. Other constructions where CGEL employs fusion of functions include compound determinatives (e.g. someone), other determinatives or adjectives as NP heads (the rich, the tallest, those three6), and fused (a.k.a. free or head-
6Elazar and Goldberg (2019) offer an NLP approach to reasoning about numeric fused heads.
Syntactic Functions
Fused
Determiner-Head
Modifier-Head
Marker-Head
Head-Prenucleus
Dependent
Compounding
Flat
Adjunct Supplement Vocative
Modifier
Nonce (e.g. Obj+Mod)
Marker
Determiner
Coordinate
Complement
Internal
Predicative Complement
Extraposed Extraposed Subject
Extraposed Object
Displaced Subject
Object Direct
Indirect Particle
Indirect
External Extranuclear
Prenucleus
Postnucleus Subject
Head
Figure 2: Hierarchy of functions. The ones annotated directly in the data are the leaves plus Complement (Comp), Object (Obj), and Supplement. The distinction between direct and indirect objects is made only in double object constructions.
less) relative constructions (whatever you want). The hyphenated notation such as Determiner-Head indicates its dual function. Thus, technically the parse is a graph rather than a tree. However, the longer of the two incoming edges can be inferred deterministically based on the Determiner-Head label and the rest of the structure. For computational purposes, then, we can omit the longer edge wherever there is fusion of functions, maintaining the tree property, and automatically add it in postprocessing for visualization. We therefore refer to CGEL-style parses as trees.

heading: 4 Towards CGELBank
text: Despite its detail and richness, in 1700+ pages, CGEL includes just 40 trees, and on some points is inexplicit. Annotating naturally occurring sentences (§5) brought many of these ambiguities to the fore. Here we identify questions we faced and the decisions we made.

heading: 4.1 Categorizing individual lexemes
text: Creating part-of-speech (POS) tagsets and defining tag boundaries have been contentious in treebanking (Atwell, 2008). CGEL’s guidance in this area is extensive but dispersed, and lists of closed-
category items are inexhaustive. For CGELBank, we compiled mentions of lexemes and their categories from CGEL and applied CGEL principles to classify numerous unmentioned lexemes.7 Examples include the determinative said (e.g., as in said contract), the coordinator slash (e.g., Dear God slash Allah slash Buddha slash Zeus), and the preposition o’clock (Pullum and Reynolds, 2013).

heading: 4.2 Simplifying and un-simplifying
text: CGEL uses various subtypes of head within clause structure (Nucleus, Predicate, Predicator); we collapse these to Head. CGEL sometimes removes intermediate unary nodes, such as eliminating Head:Nom between Head:N and its projected NP. We consistently include these nodes.

heading: 4.3 Gaps
text: CGEL posits gaps in tree structures for prenucleus position constituents, but is inconsistent in indicating them. We explicitly indicate a gap in most cases and outline our decisions for unclear cases.
Subject gaps. For open interrogatives such as (1a) and (1b), CGEL’s position is unclear. Given
7Conducted since 2006 in consultation with Huddleston and Pullum, recorded in Simple English Wiktionary.
ambiguity, we follow the standard position that a gap exists (e.g. Maling, 2000; Bies et al., 1995) in questioned or relativized subject clauses.
(1) a. What did she tell you? b. Who told you that?
Adjunct gaps. Adjuncts may appear in various locations, with some not appearing clause finally. We decided against including a gap, except in relative and open interrogative clauses where CGEL marks a gap.
Phrasal genitives. In NPs ending in a gap, we attach ’s to the gap, as in a guy I know ’s house.
Coordination and comparatives. CGEL’s Gapped Coordination refers to ellipsis, not gaps. CGELBank does not include gaps in tree structure for coordination and comparatives.

heading: 4.4 Branching & tree structure
text: In CGEL, some rare phenomena are not explicitly depicted in tree form due to the limited number of actual syntax trees in the text. Also, unary nodes (e.g. N → Nom → NP) are inconsistently indicated due to space considerations. In general, we sought to ensure that tree structure was consistent and thus had to make some decisions on how to treat phenomena such as coordination, complementation, etc.
Lexical Projection Principle. Outside of morphologically derived expressions, and excepting coordinators and subordinators, a lexical node almost always projects a phrase of the corresponding category. Thus, every N must serve as head within a Nom; every V must head a VP; every Adj must head an AdjP; and so forth. The one exception is that subject-auxiliary inversion targets auxiliaries specifically (rather than the VP they would project in normal position), so if the constituent in Prenucleus function consists of a single unmodified Vaux, it will not project a VP there.
Coordinates & markers. A coordination is a non-headed construction with coordinates as children (CGEL p. 1278). Therefore, coordinates in coordinations are neither heads nor dependents. Consider, though, the following coordination the guests and indeed his family too (p. 1278), reproduced here as Figure 3.
Unlike coordinations, NPs like and indeed his family too are headed constructions in CGEL. The NP has two modifiers: indeed and too, which, like
all modifiers, are dependents requiring a head sibling. But if the his family is not a head but a coordinate as labeled, then this NP is headless, an internal contradiction in CGEL.
Markers8 are siblings of heads when they are subordinators (see (9) on p. 954 and (51) on p. 1187), so a marker is a dependent. This, however, is incompatible with the analysis in Figure 3 (p. 1277).
To resolve these inconsistencies, the NP his family in Figure 3 must be a head and not a coordinate. We generalize from this to the principle that, contra Figure 3, a coordinate is never the child of a noncoordination, and a marker is always a dependent with a sibling head. Indirect complements. Indirect complements, such as in enough time to complete the work, are licensed by a dependent in the phrase. We construct a superordinate phrase of the head type and branch the indirect complement from that. When the complement is further delayed, we do the same for nearest possible parent phrase. Verbless clauses. CGEL’s treatment of verbless clauses (VlCs) is incompatible with its general treatment of clauses. VlCs have no verb and no VP, so they must not be clauses in the syntactic sense that CGEL implies. We treat certain PPs as having two complements, analogous to complex transitive verbs, and PPs like while happy as taking predicative complements. Supplement VlCs are analyzed as headless nonce constructions. Names. CGEL claims that the syntactic structure of proper names mostly conforms to the rules for ordinary NPs, but it also notes that there is no convincing evidence for treating one element as head in personal names. We treat proper names, along
8Though CGEL uses “marker” both non-technically (e.g., marker of distinctively informal style), and technically as a function term, we discuss only the latter.
with chemical compounds, as single lexical items, analyzing multiple tokens using the Flat relation.

heading: 5 Annotation Process
text: What began as a pet project to make CGEL-style trees for interesting sentences found in the wild eventually became a corpus-building effort, with two linguists interested in the CGEL framework (the first and third authors) serving as annotators. To date, this has resulted in over 200 trees of naturally occurring sentences—some handpicked, others sampled at random from a corpus. Statistics for CGELBank are presented in Tables 2 and 3.
CGELBank trees were drawn from multiple sources, and were annotated in four phases. 1. Twitter: Exploratory annotation of real-world sentences taken from Twitter by Annotator 2 resulted in this set of 65 trees. At this point, there were no agreed-upon guidelines for CGEL annotation and the project was largely informal. 2. EWT: A set of 100 sentences sampled from the English Web Treebank (Bies et al., 2012) was annotated by Annotator 2 and simultaneously the guidelines were composed in discussion with Annotator 1. To maintain consistency with the guidelines, both the EWT and Twitter treebanks were validated and iteratively corrected. 3. EWT-trial, Twitter-trial: Once the guidelines and validation script were mostly complete, and as the browser-based annotation workflow was under development, the two annotators used it to make 37 more trees (27 from additional EWT sentences, 10 from Twitter and other sources). These trees were singly annotated and validated but not adjudicated. 4. IAA and Pilot: For an interannotator study, both annotators independently annotated and then adjudicated a pilot set of 5 trees and then a larger set of 50 trees. These were also drawn from EWT.
# sent_id = which-liz-bought # text = which Liz bought. # sent = which Liz bought -- (Clause
The initial 165 Twitter and EWT sentences were annotated in LATEX using the forest package and converted into the .cgel format using an ad hoc Python script. Later annotation was done with a customized version9 of Active DOP, a browserbased graphical treebanking tool (van Cranenburgh, 2018). Active DOP incorporates disco-dop (van Cranenburgh et al., 2016), an active learning parser, which considerably sped up annotation. We trained disco-dop on the 202 trees created prior to the start of the IAA pilot. As input to the Active DOP tool, EWT sentences were preannotated with POS tags and gaps heuristically derived from gold UD and PTB trees; the tagging was then manually edited in a text editor.10 For the 50 IAA sentences, after trees were exported to the .cgel format, adjudication was performed cooperatively between the two annotators using a text editor with a file comparison mode.
Each split is stored in a separate file in the .cgel data format illustrated in Figure 4. This combines the sentence metadata style from the CONLL-U format11 with trees in a bracketed format adapted
9https://github.com/nschneid/activedop 10At present, Active DOP does not support editing of tokenization or gap positions in the browser interface. This should be added in the future to make the tool more usable.
11Described in the UD docs.
from PENMAN notation (Kasper, 1989). CGELBank includes a Python API for working with this format, including a script to export it to LATEX for visualization (with any reentrancies due to fusion of functions).
During the initial phases of development, it became clear that certain structural properties (like the number of Nom layers in a complex NP) were sources of annotator inconsistency. We therefore developed a validator, a script to check structural properties for obvious errors (e.g., misspelled labels; phrases with no Head) as well as less obvious errors (a category occurring in an unusual position in the tree; an unnecessary level of nesting of a phrase; a Modifier forming a ternary-branching structure; invalid coindexation of a gap; improper structure of Coordinates in coordination). Some of the validation rules are conservative and need to be broadened as new data is encountered; but flagging them is an opportunity for the annotator to check for an error or inconsistency. In our experience, the rules (implemented in 500 lines of Python) often find small problems that might otherwise have gone undetected. We quantify the impact of the validator in the next section.

heading: 6 Interannotator Study
text: To test the consistency of our CGEL annotation guidelines, we conducted an interannotator study. As a pilot, five sentences sampled from the English Web Treebank (Bies et al., 2012) were annotated independently by the first and third authors. After adjudicating annotation disagreements and adapt-
ing to the annotation tool, we sampled 50 new sentences from EWT for the interannotator study. The annotators independently annotated the new set and then jointly adjudicated disagreements.

heading: 6.1 Evaluation Metric
text: A variety of measures for interannotator agreement on constituency syntax annotation exist in the literature. The standard metric is Parseval (Black et al., 1991), which computes precision and recall of the token spans that each constituent corresponds to. One problem with the usual implementation of Parseval is that it ignores hierarchy when comparing unary nodes (i.e. multiple constituents share the same token span).12 Furthermore, there is no obviously correct way to compare trees with non-identical leaves using Parseval—which can be caused by disagreement on tokenization (e.g. on hyphenated terms) or the existence/placement of gaps, both of which we encountered in our study.
To be able to compare trees with unary nodes and potentially nonidentical tokenized strings, we turn to Tree Edit Distance (TED), which has been pointed out as an alternative to Parseval’s reliance on token spans (Emms, 2008). TED defines a correspondence between trees via insertion, deletion (which promotes children), and substitution of nodes—it can be thought of as an extension of Levenshtein distance from strings to trees.13 Like Levenshtein distance, TED is solved with dynamic programming; we adapt Zhang and Shasha’s (1989) algorithm, with details in Appendix A. We compute microaveraged precision and recall scores based on the three types of edit costs, editing the gold tree to produce the predicted tree: deletions contribute to recall error, insertions to precision error, and substitution cost is split equally between the two. We then compute F1 from precision and recall, which is equivalent to the TreeDice metric of Emms (2008) (as explained in Appendix A).
Score types. We report several scores using TED, based on different criteria for scoring candidate node alignments (matches/substitutions). In increasing order of strictness:
12For example, consider one tree with unary nodes {A,B,C} and another with {A,C,B}, all corresponding to the same token span. Parseval will report both precision and recall to be 100%, which is too lenient for our purposes since the order of unary nodes matters in CGEL.
13If the tree is viewed as a bracketed string, structural operations insert or delete a pair of brackets and the associated node label.
• unlab: Unlabelled constituents. This metric examines the tree structure alone. • flex: Labelled with function, category, and (for lexical nodes) token string, with partial credit for a node that differs in some of these respects. For each of these components, a mismatch incurs a cost of 0.25; together these comprise the node substitution cost. An exact match has cost 0. We consider flex to be the main metric as it is most nuanced and should therefore induce the most accurate alignment between the two trees. • strict: Labelled with all components, and no partial credit: the substitution cost is 1 for any two nodes that are not fully identical.
For gaps, the category is GAP and the token string is empty. Gaps are coindexed to an antecedent; this is factored into the scores by checking, after running the TED algorithm, whether two otherwise matched gaps have “the same” (aligned) antecedents. If not, the gap is not considered a full match (the flex penalty is 0.25).
Other metrics are: • gap: F1 score of gaps per the alignment in-
duced by the flex metric. • full-tree: Proportion of trees that match ex-
actly.

heading: 6.2 Results
text: Agreement scores between the two annotators as well as between the unadjudicated and the final adjudicated trees are reported in Table 4.14 For all metrics, agreement F1 exceeds 90%. In particular, the flex metric shows an interannotator agreement F1 of 93.9%. Therefore, we are confident that, with reference to our guidelines, the CGEL formalism
14Full output of the scorer on the 50 IAA sentences is provided at: https://github.com/nert-nlp/cgel/blob/ b95309f6c2ada885728b80a21b6d576bd85a20c9/datasets/ iaa/iaa.out
can be applied to the annotation of real-world text in a consistent manner.
As expected, the strict score of 91.6% is lower than the flex score, while the unlab score (which considers structure only) is higher, at 94.8%.
A breakdown of flex costs by edit type appears in Table 6. Among nodes aligned by TED, function disagreements were more numerous than category disagreements (75 vs. 44 occurrences, costing 0.25 each). But many nodes were inserted/deleted, e.g. due to attachment differences.
Zooming in to just gaps, of which there were 21 in the 50 adjudicated trees, we find good (but lower) agreement F1 of 87.2%. A major source of disagreement was a phrase in sentence 5 involving a shared object between 4 coordinated verbs— annotator 1 indicated this with 4 gaps while annotator 2 used none. Still, overall this demonstrates that even complex phenomena described in CGEL can be analysed consistently by trained annotators.
Finally, only 18.0% of trees (full-tree) are identical between the two annotators. However, many more of the trees between the annotators and the adjudicated set are identical—54.0% (annotator
1) and 32.0% (annotator 2).
Impact of validator. Output from the validation script was shown to each annotator after their initial pass through the 50 trees.15 Table 5 shows the impact of the validator by reporting flex agreement scores before and after validation. (See also Appendix B for validator effects on exact tree accuracy.) Self-agreement before vs. after validation was 99.1% (A1) and 99.5% (A2). Agreement between the two annotators improved after validation, 93.2%→ 93.9%, as the tool helped to identify spurious errors like missing or extra Nom levels in an NP, and categories in implausible functions. Agreement with the final adjudicated data increased measurably as well (A1: 96.8%→ 97.6%; A2: 95.3%→ 95.5%).
Note that all of the trees in the IAA experiment were created by editing trees proposed by the active learning parser, which at least featured locally wellformed structures—reducing the rate of spurious errors compared to annotation from scratch.
Qualitative findings. Many of the uncertainties and disagreements in the IAA experiment concerned structured names and measurements, including street addresses, age expressions, and temperature expressions. The phrase over $300 exposed the problem of treating currency symbols in orthographic order, as CGEL assigns the structure [over 300] dollars, with a complex DP. Consequently, we added a guideline requiring currency expressions to be treebanked in pronunciation order, regardless of orthographic order.
Another recurring difficulty came from compounds that might have been hyphenated, like flight test functioning as a verb: should these be treated as one lexeme or two?
The choice of function for certain types of phrases (especially PPs) seems to lie on a continuum between Complement, Modifier, and Supplement. On substitutions, the scoring script reports 18 Comp vs. Mod disagreements and 11 Mod vs. Supplement disagreements. While it may be possible to further clarify the boundaries, it seems that some subjectivity along this continuum is inevitable.
Finally, one IAA sentence contained a fronted partitive PP (of the form Out of X and Y, which is the best?). We could not find an explicit account of partitive fronting in CGEL, and plan to revisit this in future work.
15A handful of warnings were false positives, prompting changes to the script.

heading: 7 Conclusion
text: Using the analysis developed in CGEL (Huddleston and Pullum, 2002), we introduced a new expressive and linguistically-informed syntactic formalism to corpus annotation of English, which unifies constituent and dependency information in an accessible format. Creating annotation guidelines confirmed that CGEL was a strong foundation for syntactic analysis, but also revealed some minor points of underspecification for which new policies were necessary. Using our guidelines, we have created trees from naturally occurring sentences in multiple genres, and we conducted an interannotator study. We find high annotator agreement overall and even on the complex phenomenon of gapping. Overall, we are confident that the formalism of CGEL is suitable for consistent annotation of real-world text. In the future, we intend to take advantage of existing resources in other frameworks to obtain CGEL-style trees and parsers on a larger scale and in a wider range of genres.

heading: Acknowledgements
text: Author contributions: All authors contributed to the scientific direction, writing, and analysis of this work. ▷ BR: Initiator of making ad hoc CGEL-style trees (shared via https://twitter. com/DailySyntaxTree) that blossomed into this project; annotation and guidelines lead. ▷ AA: Design of evaluation metric, scripting including data format reader/tree API, quantitative analysis.▷ NS: Secondary annotator, editing of guidelines; design of data format, design of evaluation metric, scripting including validator, scoring, annotation tool customizations, quantitative analysis.
We thank John Payne, Geoffrey Pullum, and Pairoj Kunanupatham for useful discussion on many linguistic issues that arose in annotation as well as the CGEL formalism in general. We also thank various Twitter users (including Rui P. Chaves and Russell Lee-Goldman) who commented on syntactic analyses posted by Brett Reynolds, as well as anonymous reviewers. This research was supported in part by NSF award IIS2144881.

heading: A Tree Edit Distance Details
text: For our evaluation metrics, we adapted Zhang and Shasha’s (1989) TED algorithm as described in the pseudocode of Simic (2022). This is a simple recursive algorithm that compares spans of subforests in both trees, and runs in O(n4) time with memoization where n is the greater number of nodes of the two trees.16 More efficient implementations have been proposed since, such as RTED (Pawlik and Augsten, 2011) and AP-TED (Pawlik and Augsten, 2016), but memoized TED was sufficient for our purposes—50 trees could be compared in <10 seconds with a straightforward Python implementation.
An unexpected source of inefficiency we ran into at first was the direction of recursion. If subtrees are recursed into from the rightmost child, the algorithm is an order of magnitude slower than if recursion starts from the leftmost child. Inspection of the memo-table size revealed that leftmost recursion requires much fewer function calls. We think this is because English tends to be a rightbranching language, and so recursing beginning from the right increases the possible number of spans to compare between trees.
TED has been used to evaluate parsers in the past, including parsers with discontinuous constituents (Maier, 2010) and dependency parsers (Tsarfaty et al., 2011). It has also been applied or extended for other uses of comparing parse trees, such as measures of paraphrase, entailment, and answers to questions (e.g., Punyakanok et al., 2004; Wan et al., 2006; Heilman and Smith, 2010). Relation of TED F1 to TreeDice. Emms (2008) presents TreeDice, a TED-based metric for comparing constituency trees. Briefly: TED can be used to obtain the edits required to transform a gold tree into a predicted tree. With G as the size (number of nodes in) the gold tree, T as the size of the predicted tree, D as the number of deletions, I as the number of insertions, and S as the number of substitutions (where a node’s label changes), TreeDice is given by
16Or, more precisely, O(m2n2), where m and n are the sizes of the respective trees, as the recurrence is parameterized by a contiguous span of nodes in each tree under postorder traversal.
TreeDice = 1− D+ I+S G+T (1)
Using the invariant that T =G−D+ I, one can substitute G+ I−T for D and show that this equals
2T −2I−S G+T = 2(T − I− 1 2 S)
G+T (2) While Emms (2008) does not explicitly present precision and recall metrics based on TED (only ones based on evalb a.k.a. Parseval), we observe that the substitution cost can be split between precision and recall. Defining
Prec = T − I− 12 S T
(3)
Rec = G−D− 12 S G = T − I− 12 S G
(4)
it is easily shown that the F1 of these is equal to the TreeDice score (echoing the correspondence between F1-score and the Dice coefficient over sets).
F1 = 2Rec−1+Prec−1 (5)= 2(Rec−1+Prec−1)−1 (6) = 2⎛⎝ GT − I− 12 S + T T − I− 12 S ⎞⎠ −1 (7)
= 2⎛⎝ G+TT − I− 12 S ⎞⎠ −1
(8)
= 2(T − I− 12 S) G+T (9) F1 = TreeDice (10)

heading: B Exact tree accuracy
text: For comparison with the flex metric IAA results in Table 5, we also report exact tree accuracy below.

heading: C EWT Examples in PTB and CGELBank
text: Trees in the two styles appear in Figures 5 and 6 for comparison. Further cross-framework comparisons appear in Reynolds et al. (2022, §4).
S NP-SBJ
NNP
Pat
NNP
Wood
VP
VP
VBD
made
NP
NP
DT
a
NN
phone
NN
call
PP
IN
to
NP
NNP
Tom
NNP
Noel
CC
and
VP
VBD
told
NP-9
PRP
him
S
NP-SBJ-9
-NONE-
*PRO*
VP
TO
to
VP
VB
get
S
NP-SBJ-1
DT
the
NNP
Texas
NN
market
VP
VBN
fixed
NP-1
-NONE-
*
.
.
Clause
Subj: NP
Head: Nom
Head: N
Flat: N
Flat: N
Head: Coordination
Coordinate: VP
Head: V
made
Obj: NP
Det: DP
Head: Nom
Comp: PP
Head: P
Obj: NP
Coordinate: VP
Marker: Coordinator
and
Head: VP
Head: V
Obj: NP
Comp: Clause

heading: Future Work
text: In the future, we intend to take advantage of existing resources in other frameworks to obtain CGEL-style trees and parsers on a larger scale and in a wider range of genres