Dynamic Head Selection for Neural Lexicalized Constituency Parsing
Authors: Yang Hou, Zhenghua Li
Accepted at: acl25

Dynamic Head Selection for Neural Lexicalized Constituency Parsing

Yang Hou and Zhenghua Li*
School of Computer Science and Technology,
Soochow University, China
yhou1@stu.suda.edu.cn    zhli13@suda.edu.cn

Abstract
Lexicalized parsing, which associates constituent nodes with lexical heads, has historically played a crucial role in constituency parsing by bridging constituency and dependency structures. Nevertheless, with the advent of neural networks, lexicalized structures have generally been neglected in favor of unlexicalized, span-based methods. In this paper, we revisit lexicalized parsing and propose a novel latent lexicalization framework that dynamically infers lexical heads during training without relying on predefined head-finding rules. Our method enables the model to learn lexical dependencies directly from data, offering greater adaptability across languages and datasets. Experiments on multiple treebanks demonstrate state-of-the-art or comparable performance. We also analyze the learned dependency structures, headword preferences, and linguistic biases.

1 Introduction
Constituency parsing is a fundamental task in natural language processing (NLP), aiming to represent the syntactic structure of a sentence as a hierarchical tree. Parsing is beneficial for numerous downstream tasks, including machine translation (Wang et al., 2007; Currey and Heafield, 2019), semantic role labeling (Zhou et al., 2020), and named entity recognition (Li et al., 2017).

Historically, early approaches to constituency parsing emphasized lexicalized parsing, where headwords—key words within phrases—played a central role in guiding syntactic analysis (Magerman, 1995; Charniak, 1997; Collins, 1996, 1997; Charniak, 2000, 2001; Gildea, 2001; Collins, 2003). Headwords encapsulate critical syntactic and semantic information, enabling the resolution of complex phenomena such as agreement and long-distance dependencies (Hindle and Rooth, 1993).

However, the emergence of unlexicalized parsing models challenged the necessity of lexical information. Research demonstrated that unlexicalized models could achieve comparable or superior performance, often with reduced complexity and better generalization across datasets (Klein and Manning, 2003; Petrov and Klein, 2007).

Despite their historical significance, lexicalized structures are largely ignored in modern neural approaches. State-of-the-art constituency parsers predominantly adopt unlexicalized, span-based methods, leveraging pre-trained language models (PLMs) such as BERT (Devlin et al., 2019). These PLMs inherently encode rich lexical and syntactic dependencies, prompting a reassessment of the role of lexicalization in parsing. Specifically, two key questions arise:
1. Does modeling bilexical dependencies still offer benefits for neural constituency parsing?
2. Can the structural dependency representations implicitly encoded in PLMs be effectively utilized to enhance parsing?

To address the first question, we reimplement Collins-formalism of lexicalized parsing with neural models. For the second question, we propose a novel latent lexicalization framework for neural constituency parsing. Unlike traditional methods that rely on fixed head-finding rules, our framework introduces dynamic head selection during training. By treating head selection as a latent variable, the implicit syntactic knowledge encoded in PLMs are effectively leveraged, eliminating the need for explicit dependency supervision and enabling greater adaptability across languages and datasets. Our contributions are as follows:
• We revisit lexicalized parsing in the context of neural models.
• We propose a latent lexicalization framework that dynamically determines lexical heads, making the process fully data-driven and flexible.
• We demonstrate through extensive experiments
on the Penn Treebank (PTB), Chinese Treebank (CTB), and SPMRL datasets that our method achieves state-of-the-art or comparable performance.

• We provide detailed analyses of the learned dependency structures, headword preferences, and linguistic biases.
Our code is publicly available at https://github.com/ironsword666/LatentLexConstParsing.

2 Background

In this section, we provide an overview of the foundational concepts in constituency parsing and lexicalization, and introduce common methods of lexicalization.

2.1 Constituency Parsing

Constituency parsing aims to represent the syntactic structure of a sentence as a hierarchical tree. Given a sentence \( w = w_1 w_2 \ldots w_n \), where \( w_i \) is the \( i \)-th word, the task involves generating a tree \( t \) where each node represents a linguistic unit (word or phrase) labeled with its syntactic category. For example, in the sentence “The dog barks loudly at night”, the noun phrase (NP) “The dog” and the verb phrase (VP) “barks loudly at night” are constituents. Each constituent is defined by a span of words \( (i, j) \) and a syntactic category \( l \), represented as the tuple \( (i, j, l) \), where \( i \) and \( j \) denote the start and end word positions. An example constituency tree is shown in Figure 1a.

![Example trees for the sentence “The dog barks loudly at night.” Part-of-speech tags are omitted.](page_184_134_579_312.png)

Figure 1: Example trees for the sentence “The dog barks loudly at night.” Part-of-speech tags are omitted.

2.2 Lexicalization in Parsing

Lexicalization enhances constituency parsing by associating each constituent with a lexical head—a specific word that carries the most syntactic and semantic importance. For instance:

• Unlexicalized rule: \( S \rightarrow NP\ VP \)
• Lexicalized rule: \( S_{barks} \rightarrow NP_{dog}\ VP_{barks} \)

Here, “barks” serves as the head of the sentence (S), while “dog” is the head of the NP node. These lexical heads propagate from the leaves to the root of the tree, forming a lexicalized tree structure. Figure 1 contrasts unlexicalized and lexicalized trees. By enriching constituency parsing with lexical heads, lexicalization bridges the gap between constituency parsing and dependency parsing, providing a unified framework to represent both phrase structure and head-dependent relations. In the lexicalized tree \( t \) (the same symbol is reused), each constituent is represented as a headed span \( (i, j, h, l) \), where \( h \) denotes the position of the headword. Based on these headed spans, head-dependent relations \( h \rightarrow m \) can be extracted, yielding a complete dependency tree as a by-product.

2.3 Lexicalization Methods

Lexicalization in parsing relies on manual annotations or fixed heuristics to identify the lexical head of each constituent.

• Human Annotation: Dependency trees are annotated in parallel with constituency trees. While accurate, this method is labor-intensive, and consequently, such parallel treebanks are rare or virtually non-existent.
• Naive Heuristics: Simple methods such as always assigning the leftmost or rightmost child as the head are efficient but rarely align with linguistic intuition.
• Linguistic Heuristics: Handcrafted head-finding rules, such as those used by Collins (2003), determine lexical heads based on linguistic principles (e.g., the rightmost noun in a noun phrase is the head).

Among these methods, head-finding rules are most commonly used in research, as they obtain a balance between computational efficiency and linguistic precision.
Figure 2: Different binarization and lexicalization methods for a parent node with four children (circles). Squares represent intermediate nodes. Solid shapes indicate determined states, while hollow shapes indicate undetermined states.

3 Latent Lexicalized Constituency Parsing

3.1 Proposed Latent Lexicalization

Linguistic heuristics, while effective, often fail to generalize across diverse datasets and syntactic variations. Treebanks are annotated using different guidelines, and languages exhibit inherent structural differences (Dixon, 2010). Moreover, multiple dependency representations, such as Stanford Dependencies (de Marneffe et al., 2006) and Universal Dependencies (de Marneffe et al., 2021), can exist for the same sentence, highlighting the need for a general and flexible lexicalization framework.

To address these challenges, we propose a latent lexicalization framework that treats the lexical head of a constituent as a latent variable. Key features of this approach include:
• Dynamic Head Selection: In our framework, the lexical head of a constituent is not predefined. For a constituent \( A \rightarrow B_1 B_2 \ldots B_r \), any child \( B_i \) can serve as the head. This flexibility enables the model to dynamically select the most appropriate head, effectively utilizing implicit syntactic knowledge encoded in PLMs.
• Representation as A Forest: Instead of constructing a single lexicalized tree, we represent each sentence as a forest of possible lexicalized trees, where each tree corresponds to a valid head assignment. During training, the model optimizes over this forest, learning head assignments that maximize parsing performance.
This data-driven approach generalizes effectively across languages and datasets, overcoming the rigidity of traditional approaches.

3.2 Binarization of Lexicalized Trees

Efficient parsing algorithms, such as the Eisner-Satta algorithm (Eisner and Satta, 1999) used in this paper, require constituency trees to be binarized into Chomsky normal form (CNF).\footnote{For a discussion on different binarization strategies and their effects on parsing, see Chen et al. (2021).} The binarization process varies depending on how lexical heads are determined, as shown in Figure 2.
• Left Binarization: The leftmost child is selected as the head of the constituent. The tree is recursively binarized by recursively grouping the leftmost child with one sibling at a time, creating new intermediate nodes.\footnote{This process ensures that all children depend on the leftmost child.}
• Right Binarization: Assign the rightmost child as the head. The tree is transformed by grouping the rightmost child with one sibling.
• Head Binarization: Head-finding rules are used to identify the head, and these heads guide how the nodes are binarized. The detailed process is described in Appendix A.1.

Latent binarization In our latent lexicalization framework, where head selection is dynamic, binarization must account for all possible head assignments. This leads to a latent binarization\footnote{This differs from the implicit binarization (Stern et al., 2017), where an empty label \( \varnothing \) represents all intermediate nodes in binary tree decomposition.} approach, which considers all valid binary decompositions of an \( n \)-ary node during training. For instance, the constituent \( A \rightarrow B_1 B_2 B_3 \) can be binarized as:
• \( A \rightarrow A_1 B_3, A_1 \rightarrow B_1 B_2 \)
• \( A \rightarrow B_1 A_1, A_1 \rightarrow B_2 B_3 \)

This approach ensures consistency between the phrase structures and the learned dependency structures. In other words, for each head assignment, there exists a corresponding binary decomposition.

To clarify the notion of “latent” in this context: 1) In latent lexicalization, any two children of a parent node can form a potential dependency. 2) In latent binarization, for a span \( (i, j) \) in an \( n \)-ary tree, any span that crosses it is forbidden in the resulting binary tree. For example, the span \( (1, 3) \) crosses with span \( (2, 4) \), and such crossings are disallowed.
3.3 Incorporating Linguistic Knowledge
While the latent lexicalization framework is data-driven, incorporating linguistic knowledge can further refine the head selection.

Punctuation Exclusion  Punctuation marks are excluded as potential heads because they do not govern other words. This common linguistic constraint is applied across languages.

Content vs. Function Words  Content words (e.g., nouns, verbs) are more likely to serve as heads than function words (e.g., determiners, conjunctions). Introducing a bias toward content words helps the model prioritize semantically significant words when selecting heads.

Phrase-Level Preferences  Linguistic principles can guide head selection for specific phrase types. For example, in verb phrases VP → VP NP PP, objects (NP) or prepositions (PP) are less likely to serve as heads compared to the main verb (VP). Thus, NP and PP are forbidden as heads in this scenario. These preferences resemble traditional head-finding rules but are implemented as soft biases, allowing the model to adapt to diverse syntactic structures. For instance, the attachment of a PP to either a VP or an NP remains flexible, depending on the data.

For simplicity and cross-linguistic applicability, only punctuation exclusion is used in this work.

4 Model
In this section, we describe the scoring mechanism, model architecture, training, and decoding process.

4.1 Scoring factorization
We adopt a two-stage chart-based framework for lexicalized parsing, following Dozat and Manning (2017) and Zhang et al. (2020). In this framework, constituent brackets are predicted first, followed by the labeling of these constituents. The overall score for a (unlabeled) lexicalized tree \( t \) is decomposed into constituency and dependency scores:

\[
s(t) = s^{\text{con}}(t) + s^{\text{dep}}(t)
= \sum_{(i,j) \in t} s(i,j) + \sum_{(h \to m) \in t} s(h \to m)
\] (1)

where \( s(i,j) \) is the score of constituent span \( (i,j) \), and \( s(h \to m) \) denotes the score of a dependency arc from a head \( h \) to its dependent \( m \).

This decomposition enables the use of Eisner-Satta algorithm (Appendix A.2), a chart-based dynamic programming approach with a time complexity of \( O(n^4) \) for both training and decoding.\footnote{Alternatively, scores can be factored over headed spans \( (i,j,h) \), but this would require a CKY-style algorithm (Collins, 1997), leading to a higher computational complexity (\( O(n^5) \)).}

4.2 Model Architecture
For an input sentence \( w = w_0w_1...w_n \), where \( w_0 \) is an artificial ROOT token, we first obtain contextualized representation of each word \( w_i \) using a pre-trained language model (BERT in this work).

\[
x_i = \text{BERT}(w_i)
\] (2)

The representations\footnote{We use fencenpost representations for constituents, as described in Stern et al. (2017).} are then used to compute scores for candidate constituents and dependencies using biaffine attention mechanisms (Dozat and Manning, 2017).

\[
\begin{align*}
x_i^{\text{con-left}} &= \text{MLP}^{\text{con-left}}(x_i) \\
x_j^{\text{con-right}} &= \text{MLP}^{\text{con-right}}(x_j) \\
x_h^{\text{dep-head}} &= \text{MLP}^{\text{dep-head}}(x_h) \\
x_m^{\text{dep-mod}} &= \text{MLP}^{\text{dep-mod}}(x_m)
\end{align*}
\] (3)

Scores are computed for constituency spans \( (i,j) \) and dependency arcs \( h \to m \):

\[
s(i,j) = \text{BiAffine}^{\text{con}}(x_i^{\text{con-left}}, x_j^{\text{con-right}})
\]
\[
s(h \to m) = \text{BiAffine}^{\text{dep}}(x_h^{\text{dep-head}}, x_m^{\text{dep-mod}})
\] (4)

Additionally, constituent label scores \( s(i,j,l) \) are computed using another biaffine mechanism.

4.3 Training
We train the model using conditional random fields (CRF) to maximize the probability of the gold-standard tree. The probability of a lexicalized tree \( t \) is defined as:

\[
p(t|w) = \frac{\exp(s(t))}{Z(w) \equiv \sum_t \exp(s(t'))}
\] (5)

where \( Z(w) \) is the partition function summing over all possible trees. When headwords are explicitly specified, the tree probability directly supervises the model. When headwords are latent, we optimize over a forest \( \mathcal{T} \) of valid lexicalized trees:

\[
p(\mathcal{T}|w) = \frac{\sum_{t^* \in \mathcal{T}} \exp(s(t^*))}{Z(w)}
\]
\[
\mathcal{L}_{\text{bracket}}(\theta) = -\log p(\mathcal{T}|w)
\] (6)
The summation process is performed using a Inside-version of Eisner-Satta algorithm. See Algorithm 1 in Appendix A.2 for details.

For constituent labels, we use cross-entropy loss:

\[
\mathcal{L}_{\text{label}}(\theta) = - \sum_{(i,j,l) \in t} \log p(l|i,j)
\] (7)

The final loss combines both objectives:

\[
\mathcal{L}(\theta) = \mathcal{L}_{\text{bracket}}(\theta) + \mathcal{L}_{\text{label}}(\theta)
\] (8)

4.4 Inference

During inference, we use Eisner-Satta algorithm to decode the best lexicalized tree:

\[
\hat{t} = \arg\max_t s(t)
\] (9)

Then, the optimal label for each constituent \((i,j)\) is predicted as:

\[
\hat{l}_{i,j} = \arg\max_l s(i,j,l)
\] (10)

In addition, head-dependent relations can be directly extracted from the predicted lexicalized tree.

5 Experiments

5.1 Setup

Data. We conduct experiments on benchmarks for multiple languages. For English, we use Penn Treebank 3.0 (PTB) (Marcus et al., 1993), following the standard splits (sections 2–21 for training, section 22 for development, and section 23 for testing). For Chinese, the Chinese Treebank 5.1 (CTBS) (Xue et al., 2005), split according to the common practice. We also evaluate our model on six morphologically rich languages (French, German, Korean, Hungarian, Basque, and Polish) from the SPMRL dataset (Seddah et al., 2013), using the provided splits. The statistics of the datasets are shown in the Appendix B.1.

Evaluation Metrics Parsing performance is evaluated using the standard precision, recall and F1 score metrics with the EVALB tool6 for PTB and CTB, and the EVALB_SPMRL script7 for SPMRL.

Implementation Details For English and Chinese, we extract dependencies with Stanford parser v3.3.08. And for SPMRL, we use the provided gold-standard dependencies. Additional implementation details, including hyperparameter settings, optimization strategies, and computational resources, are provided in Appendix B.1.

5.2 Main Results

Effectiveness of Lexicalization We evaluate the impact of different lexicalization methods by comparing the following models, all trained with a CRF loss. The results of the max-margin training are presented in the Appendix B.2 for details.
• Con: A baseline unlexicalized parser, following the approach of Zhang et al. (2020).
• Con (Latent): An extension of Con, incorporating latent binarization (see Section 3.2).
• Lex (Head): A lexicalized parser that uses pre-specified lexical heads, extracted or annotated.
• Lex (Left): Assigns the leftmost child as the head for each constituent.
• Lex (Right): Assigns the rightmost child as the head for each constituent.
• Lex (Latent): Uses latent binarization and latent lexicalization to infer heads during training.

Table 1 presents the results on PTB, CTB5 and SPMRL datasets. Surprisingly, using predetermined lexical heads (Lex (Head)) degrades performance compared to the unlexicalized baseline (Con), suggesting that rigid heuristic rules may not align well with the implicit syntactic knowledge encoded in PLMs. Assigning the leftmost (Lex (Left)) or rightmost (Lex (Right)) child as the head yields slight improvements on some languages, despite being linguistically simplistic. Our latent lexicalization approach (Lex (Latent)) consistently outperforms other methods across all languages, except for two low-resource languages, Basque and Polish. This demonstrates the advantages of dynamically inferred heads over fixed head assignments. These findings highlight the limitations of traditional head-finding heuristics in the context of PLMs. While there may be room for improvement in leveraging rule-extracted headwords, the results emphasize the promise of latent lexicalization as a flexible and effective strategy for improving constituent lexicalization in neural parsing models.

Compared with previous work Tables 2, 3 and 4 compare our model with state-of-the-art parsers
<table>
  <tr>
    <th>Model</th>
    <th>English</th>
    <th>Chinese</th>
    <th>French</th>
    <th>German</th>
    <th>Korean</th>
    <th>Hungarian</th>
    <th>Basque</th>
    <th>Polish</th>
  </tr>
  <tr>
    <td>Con</td>
    <td>95.81</td>
    <td>91.96</td>
    <td>87.25</td>
    <td>90.36</td>
    <td>89.38</td>
    <td>94.54</td>
    <td>91.20</td>
    <td>95.94</td>
  </tr>
  <tr>
    <td>Con (latent)</td>
    <td>95.86</td>
    <td>91.68</td>
    <td>87.88</td>
    <td>90.41</td>
    <td>89.42</td>
    <td>94.79</td>
    <td>91.19</td>
    <td><b>95.98</b></td>
  </tr>
  <tr>
    <td>Lex (head)</td>
    <td>95.77</td>
    <td>91.67</td>
    <td>87.30</td>
    <td>90.33</td>
    <td>89.28</td>
    <td>94.32</td>
    <td>91.16</td>
    <td>95.62</td>
  </tr>
  <tr>
    <td>Lex (left)</td>
    <td>95.79</td>
    <td>92.00</td>
    <td>87.79</td>
    <td>90.34</td>
    <td>89.31</td>
    <td>94.69</td>
    <td><b>91.34</b></td>
    <td>95.68</td>
  </tr>
  <tr>
    <td>Lex (right)</td>
    <td>95.82</td>
    <td>92.07</td>
    <td>87.29</td>
    <td>90.52</td>
    <td>89.43</td>
    <td>94.82</td>
    <td>91.13</td>
    <td>95.76</td>
  </tr>
  <tr>
    <td>Lex (latent)</td>
    <td><b>96.02</b></td>
    <td><b>92.34</b></td>
    <td><b>87.91</b></td>
    <td><b>90.53</b></td>
    <td><b>89.51</b></td>
    <td><b>94.83</b></td>
    <td>91.28</td>
    <td>95.79</td>
  </tr>
  <tr>
    <td>Δ vs. Con (latent)</td>
    <td>+0.16</td>
    <td>+0.66</td>
    <td>+0.03</td>
    <td>+0.17</td>
    <td>+0.09</td>
    <td>+0.04</td>
    <td>+0.09</td>
    <td>-0.19</td>
  </tr>
</table>

Table 1: Results of different lexicalization methods on PTB (English), CTB5 (Chinese) and SPMRL (covering French, German, Korean, Hungarian, Basque and Polish). Only the F1 scores are reported.

<table>
  <tr>
    <th>Model</th>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Kitaev et al. (2019)</td>
    <td>95.46</td>
    <td>95.73</td>
    <td>95.59</td>
  </tr>
  <tr>
    <td>Zhou and Zhao (2019)</td>
    <td>95.70</td>
    <td>95.98</td>
    <td>95.84</td>
  </tr>
  <tr>
    <td>Zhang et al. (2020)</td>
    <td>95.85</td>
    <td>95.53</td>
    <td>95.69</td>
  </tr>
  <tr>
    <td>Xin et al. (2021)</td>
    <td><b>96.29</b></td>
    <td>95.55</td>
    <td>95.92</td>
  </tr>
  <tr>
    <td>Cui et al. (2022)</td>
    <td>95.70</td>
    <td><b>96.14</b></td>
    <td>95.92</td>
  </tr>
  <tr>
    <td>Yang and Tu (2022)</td>
    <td>96.19</td>
    <td>95.83</td>
    <td>96.01</td>
  </tr>
  <tr>
    <td>Yang and Tu (2023)</td>
    <td>96.21</td>
    <td>95.87</td>
    <td><b>96.04</b></td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>96.19</td>
    <td>95.85</td>
    <td>96.02</td>
  </tr>
</table>

Table 2: Results on PTB test set.

<table>
  <tr>
    <th>Model</th>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Kitaev et al. (2019)</td>
    <td>91.96</td>
    <td>91.55</td>
    <td>91.75</td>
  </tr>
  <tr>
    <td>Zhang et al. (2020)</td>
    <td>92.51</td>
    <td>92.04</td>
    <td>92.27</td>
  </tr>
  <tr>
    <td>Yang and Deng (2020)</td>
    <td><b>93.80</b></td>
    <td><b>93.40</b></td>
    <td><b>93.59</b></td>
  </tr>
  <tr>
    <td>Xin et al. (2021)</td>
    <td>92.94</td>
    <td>92.06</td>
    <td>92.50</td>
  </tr>
  <tr>
    <td>Cui et al. (2022)</td>
    <td>92.45</td>
    <td>92.17</td>
    <td>92.31</td>
  </tr>
  <tr>
    <td>Yang and Tu (2023)</td>
    <td>92.83</td>
    <td>91.97</td>
    <td>92.41</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>92.60</td>
    <td>91.87</td>
    <td>92.34</td>
  </tr>
</table>

Table 3: Results on CTB5 test set.

on PTB, CTB, and SPMRL datasets. On PTB, our model achieves a F1 score of 96.02, comparable to the best-performing parser (Yang and Tu, 2023). For CTB5, our method achieves competitive results, demonstrating its effectiveness across languages. On the multilingual SPMRL dataset, our parser outperforms previous methods on French and Korean, and achieves strong results on other languages, showcasing its robustness in morphologically rich settings.

6 Analysis

In this section, we analyze the prediction behavior of our model to better understand the effects of latent lexicalization. Specifically, we investigate the head selection patterns and its consistency with Stanford Dependencies (SD) on PTB test set.

6.1 Lexical Head Selection

We analyze the predicted head selection patterns from two perspectives: (1) the headword choices for different phrase types, and (2) the dependency relationships inferred between POS tags. The gold-standard references can be found in the Appendix B.4.

Phrase Head Preferences We investigate head selection for various types of constituents.9 The results, shown in Table 5, reveal several patterns:
• Noun Phrase (NP): The model predominantly selects the noun as the head, aligning with linguistic expectation.
• Verb Phrase (VP): The main verb is usually chosen as the head, but auxiliary verbs (e.g., is, has) are also frequently identified as heads.
• Prepositional Phrase (PP): The model frequently selects the noun phrase (NP) within the PP as the head. This behavior is consistent with the Universal Dependencies (UD) annotation scheme.
These results suggest that the model captures certain linguistic norms, though it does not always strictly adhere to them.

Dependency Relationship To further investigate head selection, we examine the predicted dependencies between POS tags. Table 6 outlines frequent dependency relationships. Some dependencies align with linguistic conventions, such as:
• The dependency between verbs and nouns (VB \( \rightarrow \) NN).

9Inherited heads from unary chains (e.g., NP \( \rightarrow \) NNS) have minimal impact on the results.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Rich Resource</th>
    <th colspan="4">Low Resource</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>French</th>
    <th>German</th>
    <th>Korean</th>
    <th>Hungarian</th>
    <th>Basque</th>
    <th>Polish</th>
  </tr>
  <tr>
    <td>Kitaev et al. (2019)</td>
    <td>87.42</td>
    <td>90.20</td>
    <td>88.80</td>
    <td>94.90</td>
    <td>91.63</td>
    <td><b>96.36</b></td>
    <td>91.55</td>
  </tr>
  <tr>
    <td>Nguyen et al. (2020)</td>
    <td>86.69</td>
    <td>90.28</td>
    <td>88.71</td>
    <td>94.24</td>
    <td><b>92.02</b></td>
    <td>96.14</td>
    <td>91.34</td>
  </tr>
  <tr>
    <td>Cui et al. (2022)</td>
    <td>87.51</td>
    <td>90.43</td>
    <td>89.07</td>
    <td>94.95</td>
    <td>91.73</td>
    <td>96.33</td>
    <td>91.67</td>
  </tr>
  <tr>
    <td>Yang and Tu (2023)</td>
    <td>87.89</td>
    <td><b>91.07</b></td>
    <td>89.31</td>
    <td><b>95.06</b></td>
    <td>91.72</td>
    <td>96.18</td>
    <td><b>91.87</b></td>
  </tr>
  <tr>
    <td>Ours</td>
    <td><b>87.91</b></td>
    <td>90.53</td>
    <td><b>89.51</b></td>
    <td>94.83</td>
    <td>91.28</td>
    <td>95.79</td>
    <td>91.64</td>
  </tr>
</table>

Table 4: Results on SPMRL test set. Only the F1 scores are reported.

<table>
  <tr>
    <th rowspan="2">Phrase</th>
    <th>Head</th>
    <th>Count</th>
    <th>Ratio (%)</th>
    <th>Headword</th>
    <th rowspan="2">Head \( \bowtie \) Dependent</th>
    <th>Count</th>
    <th>Ratio (%)</th>
  </tr>
  <tr>
    <td>NN</td>
    <td>5382</td>
    <td>29.0</td>
    <td>it, “%”,</td>
    <td>NN \( \bowtie \) DT</td>
    <td>3250</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td rowspan="3">NP</td>
    <td>NNS</td>
    <td>3123</td>
    <td>16.9</td>
    <td>the, company,</td>
    <td>NNS \( \bowtie \) NNP</td>
    <td>2837</td>
    <td>5.0</td>
  </tr>
  <tr>
    <td>NNP</td>
    <td>2795</td>
    <td>15.1</td>
    <td>he, market,</td>
    <td>NN \( \bowtie \) IN</td>
    <td>1871</td>
    <td>3.3</td>
  </tr>
  <tr>
    <td>NP</td>
    <td>2511</td>
    <td>13.6</td>
    <td>share, million</td>
    <td>NN \( \bowtie \) NN</td>
    <td>1506</td>
    <td>2.7</td>
  </tr>
  <tr>
    <td rowspan="4">VP</td>
    <td>VP</td>
    <td>2453</td>
    <td>28.0</td>
    <td>said, is,</td>
    <td>NNS \( \bowtie \) IN</td>
    <td>1231</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>VBD</td>
    <td>1220</td>
    <td>13.9</td>
    <td>be, are,</td>
    <td>NNS \( \bowtie \) NN</td>
    <td>973</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>VB</td>
    <td>973</td>
    <td>11.1</td>
    <td>have, was,</td>
    <td>NN \( \bowtie \) JJ</td>
    <td>959</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>VBZ</td>
    <td>690</td>
    <td>7.9</td>
    <td>says, for</td>
    <td>VBD \( \bowtie \) .</td>
    <td>901</td>
    <td>1.6</td>
  </tr>
  <tr>
    <td rowspan="2">S</td>
    <td>VP</td>
    <td>5265</td>
    <td>93.4</td>
    <td>said, is,</td>
    <td>VB \( \bowtie \) NN</td>
    <td>885</td>
    <td>1.6</td>
  </tr>
  <tr>
    <td>S</td>
    <td>221</td>
    <td>3.9</td>
    <td>are, was</td>
    <td>ROOT \( \bowtie \) VBD</td>
    <td>880</td>
    <td>1.5</td>
  </tr>
  <tr>
    <td rowspan="2">PP</td>
    <td>NP</td>
    <td>4570</td>
    <td>83.3</td>
    <td>in, of,</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>IN</td>
    <td>552</td>
    <td>10.1</td>
    <td>market, for</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">ADVP</td>
    <td>RB</td>
    <td>1027</td>
    <td>83.5</td>
    <td>also, ago,</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>RBR</td>
    <td>43</td>
    <td>3.5</td>
    <td>now, still</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">ADJP</td>
    <td>JJ</td>
    <td>420</td>
    <td>49.3</td>
    <td>“$”, million,</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>S</td>
    <td>51</td>
    <td>6.0</td>
    <td>more, billion</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">WHNP</td>
    <td>WDT</td>
    <td>272</td>
    <td>63.0</td>
    <td>that, which,</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>WP</td>
    <td>106</td>
    <td>24.5</td>
    <td>who, what</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Most frequent heads for each constituent label, along with high-frequency headwords (listed in left-to-right, top-down order).

<table>
  <tr>
    <th>Model</th>
    <th>UAS</th>
    <th>UUAS</th>
  </tr>
  <tr>
    <td>Ours</td>
    <td>37.0</td>
    <td>51.3</td>
  </tr>
</table>

Table 6: Dependency relationships between POS tags.

for reference

Wu et al. (2020)  41.7  52.1

Table 7: Dependency parsing performance. Wu et al. (2020) use a syntax probing method on the English Parallel Universal Dependencies (PUD) treebank.

• The tendency for verbs to serve as the head of sentences (ROOT \( \bowtie \) VBD).

However, certain dependencies contradict commonly used Stanford Dependencies. For instance:
• Dependencies between nouns and prepositions (NN \( \bowtie \) IN) contradict the expected head direction, where prepositions typically govern nouns.

These inconsistencies do not necessarily indicate a failure to learn dependency relationships. Intriguingly, the prediction preference for nouns as heads of prepositional phrases aligns with Universal Dependencies conventions. This alignment offers practical advantages for resolving classic PP attachment ambiguities. For example, in the sentence “They saw a cat with a telescope”, to facilitate disambiguation, the model requires prioritizing semantically meaningful associations (\( saw \leftrightarrow telescope, cat \leftrightarrow telescope \)) over syntactically functional connections (\( saw \leftrightarrow with, cat \leftrightarrow with \)).

6.2 Consistency with Stanford Dependencies

Parsing Performance  We compare the predicted dependencies against Stanford Dependencies using unlabeled attachment score (UAS) and undirected UAS (UUAS) (Klein and Manning, 2004). The results, presented in Table 7, show that the UAS value is relatively low, indicating that the predicted dependencies differ substantially from those in the SD. However, UUAS, which eliminates sensitive to annotation variation, gains a higher score.

Arc Direction  The directionality of dependency arcs offers valuable insight into language-specific syntactic patterns (note that arc direction is distinct from head-dependent direction). Dependency arc
<table>
  <tr>
    <th>Model</th>
    <th>Leftward Arc</th>
    <th>Rightward Arc</th>
  </tr>
  <tr>
    <td>SD</td>
    <td>51.8</td>
    <td>48.2</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>67.0</td>
    <td>33.0</td>
  </tr>
</table>

Table 8: Ratio of leftward arcs and rightward arcs.

<table>
  <tr>
    <th>Language</th>
    <th>Source</th>
    <th>UAS</th>
    <th>UUAS</th>
  </tr>
  <tr>
    <td>English</td>
    <td>C</td>
    <td>37.01</td>
    <td>51.32</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>C</td>
    <td>63.02</td>
    <td>70.34</td>
  </tr>
  <tr>
    <td>French</td>
    <td>C</td>
    <td>34.83</td>
    <td>52.76</td>
  </tr>
  <tr>
    <td>German</td>
    <td>C</td>
    <td>46.22</td>
    <td>58.22</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>C</td>
    <td>29.70</td>
    <td>54.45</td>
  </tr>
  <tr>
    <td>Hungarian</td>
    <td>C</td>
    <td>36.34</td>
    <td>49.22</td>
  </tr>
  <tr>
    <td>Polish</td>
    <td>C</td>
    <td>13.10</td>
    <td>33.21</td>
  </tr>
  <tr>
    <td>Basque</td>
    <td>D</td>
    <td>44.07</td>
    <td>58.92</td>
  </tr>
</table>

Table 9: Dependency parsing performance on PTB, CTB5, and SPMRL datasets. C and D denote converted and manually annotated dependency trees, respectively.

direction varies across languages. For example, English is a head-initial language, whereas Chinese is head-final.\footnote{The annotation formalism also influences arc direction. For instance, Universal Dependencies (UD) often prioritize right-to-left dependencies.}

To evaluate arc directionality, we calculate the ratio of leftward arcs and rightward arcs. As shown in Table 8, the predicted dependencies predominantly exhibit head-final patterns, aligning with the results in Table 6. Specifically, the model correctly predicts head-initial arcs for verbs governing objects (e.g., eat \( \sim \) apple). However, a significant proportion of noun phrases in the dataset follow a head-final structure (e.g., the \( \sim \) dog), which influences the overall prediction patterns.

\subsection*{6.3 Dependency Evaluation}

We expand the evaluation of dependency parsing to more datasets, including CTB5 and SPMRL. We distinguish between dependency trees that are manually annotated (D) and those converted from constituency trees (C). The results are shown in Table 9. We observe that for CTB5, the dependency performance is quite strong, while the model performs worst in Polish. Basque, with manually annotated dependency trees, shows a fair result.

\section*{7 Related Work}

Lexicalized Parsing Lexicalized parsing has been a cornerstone in syntactic analysis, underpinning many early parsing frameworks (Magerman, 1995; Charniak, 1997; Collins, 1996, 1997). Among these, the Collins formalism stands out as particularly influential, utilizing lexical heads to facilitate syntactic derivations (Collins, 2003). Although initially designed for English and chart-based parsing, these models have been successfully adapted to other languages and transition-based parsing paradigm (Crabbé, 2014, 2015).

Joint Parsing The inherent ability of lexicalized parsing to encode dependency relations has motivated efforts to jointly predict constituency and dependency structures within a unified framework. Some approaches maintain separate representations for dependency and constituency structures while enforcing consistency constraints between them (Strzyz et al., 2019; Fernández-González and Gómez-Rodríguez, 2022). Others adopt a single-tree representation to jointly predict both structures (Zhou and Zhao, 2019; Gu et al., 2024).

Modern Constituency Parsing Building on traditional methods, modern constituency parsers predominantly follow either chart-based (Kitaeve and Klein, 2018; Zhang et al., 2020; Xin et al., 2021) or transition-based (Cross and Huang, 2016; Liu and Zhang, 2017) approaches. Transition-based parsers, in particular, have seen various innovations. Some models transform input sentences into linearized tree representations (Wei et al., 2020; Amini and Cotterell, 2022) or pointer sequences (Nguyen et al., 2020; Yang and Tu, 2022). Another promising direction is combinatorial parsing (Chen et al., 2021; Chen and Komachi, 2023), where shorter spans are merged into longer spans in a layer-wise manner. Notably, their multi-branching architecture—similar to our work—can be adapted to select the head of each constituent.

Latent Parsing The use of latent representations has proven effective in various NLP tasks. Fu et al. (2020) model nested named entities using a constituency parsing framework. Lou et al. (2022) extend their work by introducing a latent lexicalized tree structure. Zhang et al. (2022) treat semantic arguments as latent single-root dependency trees. These approaches demonstrate the potential of latent parsing frameworks to handle complex linguistic structures without manual annotations.
Compared with these work, our work is the first to apply latent parsing to constituency parsing.

Syntax Probing Syntax probing has emerged as a key methodology for analyzing the linguistic knowledge encoded in PLMs (Tenney et al., 2019; Belinkov, 2022). Probing tasks often rely on linear transformations to map model representations to syntactic parse trees (Hewitt and Manning, 2019; Manning et al., 2020). Utilizing parameter-free methods, researchers have shown that PLMs implicitly learn rich syntactic structures, such as constituency and dependency parses, even without explicit supervision (Wu et al., 2020; Niu et al., 2022; Buder-Gröndahl, 2024). Our work complements syntax probing by explicitly incorporating lexicalized parsing into a neural framework.

8 Conclusion

In this work, we proposed a latent lexicalization framework for constituency parsing, designed to dynamically infer head words during training. By treating head selection as a latent variable, the model can learn lexical dependencies directly from data, eliminating the need for predefined head-finding rules and enhancing adaptability across languages and datasets.

Our experiments on PTB and CTB demonstrate that the latent lexicalization framework consistently outperform traditional lexicalization approaches, such as heuristic-based. Key findings include: 1) The heads inferred by the model successfully align with linguistic conventions for some syntactic structures, such as verbs governing their arguments. However, for preposition attachment, the head selection conflicts with Stanford Dependencies. 2) The model often predicts head-final arcs in English, which is common for nouns governing their modifiers. These results underscore the potential of latent lexicalization to utilize lexical dependency representations encoded in PLMs.

Limitations

Projectivity and Continuous Spans The parsing algorithm (Eisner-Satta) used in this work inherently restricts the model to projective dependency structures. Furthermore, our framework focuses exclusively on continuous spans, meaning it does not account for discontinuous constituents. Discontinuous spans are often associated with non-projective dependencies, which frequently occur in linguistic phenomena such as extraposition or topicalization. As a result, our model may not fully capture certain syntactic configurations that require non-projective or discontinuous representations. To address this limitation, future work could explore alternative parsing algorithms that support non-projective structures and discontinuous spans.

Computational Complexity While the proposed method supports efficient batch processing on GPUs, its theoretical complexity of \( O(n^3) \) may limit scalability in scenarios involving long sentences. Although parsing speeds are comparable to existing methods in practice (Appendix B.3), the increased computational overhead could pose challenges for resource-constrained environments.

Headword Consistency The discrepancies observed between the predicted heads and gold-standard heads suggest that the model does not always capture the correct headword information. This inconsistency may propagate errors in downstream tasks, which rely on accurate dependency structures. While incorporating linguistic constraints could improve consistency, doing so may introduce additional complexity and reduce the cross-linguistic generalizability.

Acknowledgements

We thank all the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foundation of China (Grant No. 62176173 and 62336006), and a Project Funded by the Priority Academic Program Development (PAPD) of Jiangsu Higher Education Institutions.

References

Afra Amini and Ryan Cotterell. 2022. On parsing as tagging. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8884–8900, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219.

Tommi Buder-Gröndahl. 2024. What does parameter-free probing really uncover? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 327–336, Bangkok, Thailand. Association for Computational Linguistics.
Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Innovative Applications of Artificial Intelligence Conference, AAAI 97, IAAI 97, July 27-31, 1997, Providence, Rhode Island, USA, pages 598–603. AAAI Press / The MIT Press.

Eugene Charniak. 2000. A maximum-entropy-inspired parser. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics.

Eugene Charniak. 2001. Immediate-head parsing for language models. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 124–131, Toulouse, France. Association for Computational Linguistics.

Zhousi Chen and Mamoru Komachi. 2023. Discontinuous combinatorial constituency parsing. Transactions of the Association for Computational Linguistics, 11:267–283.

Zhousi Chen, Longtu Zhang, Aizhan Imankulova, and Mamoru Komachi. 2021. Neural combinatorial constituency parsing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2199–2213, Online. Association for Computational Linguistics.

Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pages 16–23, Madrid, Spain. Association for Computational Linguistics.

Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637.

Michael John Collins. 1996. A new statistical parser based on bigram lexical dependencies. In 34th Annual Meeting of the Association for Computational Linguistics, pages 184–191, Santa Cruz, California, USA. Association for Computational Linguistics.

Benoit Crabbé. 2014. An LR-inspired generalized lexicalized phrase structure parser. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 541–552, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.

Benoit Crabbé. 2015. Multilingual discriminative lexicalized phrase structure parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1847–1856, Lisbon, Portugal. Association for Computational Linguistics.

James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1–11, Austin, Texas. Association for Computational Linguistics.

Leyang Cui, Sen Yang, and Yue Zhang. 2022. Investigating non-local features for neural constituency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2065–2075, Dublin, Ireland. Association for Computational Linguistics.

Anna Currey and Kenneth Heafield. 2019. Incorporating source syntax into transformer-based neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 24–33, Florence, Italy. Association for Computational Linguistics.

Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Robert MW Dixon. 2010. Basic Linguistic Theory: Volume 2 Grammatical Topics. Oxford University Press.

Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.

Jason Eisner and Giorgio Satta. 1999. Efficient parsing for bilexical context-free grammars and head automaton grammars. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 457–464, College Park, Maryland, USA. Association for Computational Linguistics.

Daniel Fernández-González and Carlos Gómez-Rodríguez. 2022. Multitask pointer network for multi-representational parsing. Knowl. Based Syst., 236:107760.

Yao Fu, Chuanqi Tan, Mosha Chen, Songfang Huang, and Fei Huang. 2020. Nested named entity recognition with partially-observed treecrfs. In AAAI Conference on Artificial Intelligence.
Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing.

Yanggan Gu, Yang Hou, Zhefeng Wang, Xinyu Duan, and Zhenghua Li. 2024. High-order joint constituency and dependency parsing. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8144–8154, Torino, Italia. ELRA and ICCL.

John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics.

Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational Linguistics, 19(1):103–120.

Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499–3505, Florence, Italy. Association for Computational Linguistics.

Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.

Dan Klein and Christopher Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 478–485, Barcelona, Spain.

Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. Association for Computational Linguistics.

Peng-Hsuan Li, Ruo-Ping Dong, Yu-Siang Wang, Ju-Chieh Chou, and Wei-Yun Ma. 2017. Leveraging linguistic structures for named entity recognition with bidirectional recursive neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2664–2669, Copenhagen, Denmark. Association for Computational Linguistics.

Jiangming Liu and Yue Zhang. 2017. Encoder-decoder shift-reduce syntactic parsing. In Proceedings of the 15th International Conference on Parsing Technologies, pages 105–114, Pisa, Italy. Association for Computational Linguistics.

Chao Lou, Songlin Yang, and Kewei Tu. 2022. Nested named entity recognition as latent lexicalized constituency parsing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6183–6198, Dublin, Ireland. Association for Computational Linguistics.

David M. Magerman. 1995. Statistical decision-tree models for parsing. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 276–283, Cambridge, Massachusetts, USA. Association for Computational Linguistics.

Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proc. Natl. Acad. Sci. USA, 117(48):30046–30054.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330.

Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2020. Efficient constituency parsing by pointing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3284–3294, Online. Association for Computational Linguistics.

Jingcheng Niu, Wenjie Lu, Eric Corlett, and Gerald Penn. 2022. Using roark-hollingshead distance to probe BERT’s syntactic competence. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 325–334, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics: Proceedings of the Main Conference, pages 404–411, Rochester, New York. Association for Computational Linguistics.

Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richard Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Galletebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics.

Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser.
In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 818–827, Vancouver, Canada. Association for Computational Linguistics.

Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Sequence labeling parsing by learning across representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5350–5357, Florence, Italy. Association for Computational Linguistics.

Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics.

Wei Wang, Kevin Knight, and Daniel Marcu. 2007. Binarizing syntax trees to improve syntax-based machine translation accuracy. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 746–754, Prague, Czech Republic. Association for Computational Linguistics.

Yang Wei, Yuanbin Wu, and Man Lan. 2020. A span-based linearization for constituent trees. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3267–3277, Online. Association for Computational Linguistics.

Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, Online. Association for Computational Linguistics.

Xin Xin, Jinlong Li, and Zeqi Tan. 2021. N-ary constituent tree parsing with recursive semi-Markov model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2631–2642, Online. Association for Computational Linguistics.

Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Natural language engineering, pages 207–238.

Kaiyu Yang and Jia Deng. 2020. Strongly incremental constituency parsing with graph neural networks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Songlin Yang and Kewei Tu. 2022. Bottom-up constituency parsing and nested named entity recognition with pointer networks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2403–2416, Dublin, Ireland. Association for Computational Linguistics.

Songlin Yang and Kewei Tu. 2023. Don’t parse, choose spans! continuous and discontinuous constituency parsing via autoregressive span selection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8420–8433, Toronto, Canada. Association for Computational Linguistics.

Yu Zhang, Qingrong Xia, Shilin Zhou, Yong Jiang, Guohong Fu, and Min Zhang. 2022. Semantic role labeling as dependency parsing: Exploring latent tree structures inside arguments. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4212–4227, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Yu Zhang, Houquan Zhou, and Zhenghua Li. 2020. Fast and accurate neural CRF constituency parsing. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4046–4053. ijcai.org.

Junru Zhou, Zuchao Li, and Hai Zhao. 2020. Parsing all: Syntax and semantics, dependencies and spans. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4438–4449, Online. Association for Computational Linguistics.

Junru Zhou and Hai Zhao. 2019. Head-Driven Phrase Structure Grammar parsing on Penn Treebank. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2396–2408, Florence, Italy. Association for Computational Linguistics.
ATTACH :
COMPLETE :

Figure 3: Deduction rules for Eisner-Satta algorithm (ATTACH and COMPLETE). We show only Left-rules, omitting the symmetric Right-rules as well as initial conditions for brevity.

A Method

A.1 Head Binarization

If the headwords are extracted with head-finding rules, the left binarization and right binarization strategies are no longer applicable. That is because the headword is not always the leftmost or right-most child. In this case, we can use the head binarization to binarize the constituency tree.

The basic idea is to split the constituent into two parts with the headword in the middle. However, it is not enough to simply split the constituent into two parts. We need to ensure that the dependency structures in both parts are valid. Specifically, if node \( A \) is binarized as \( A \rightarrow A_1 A_2 \): (1) there exists a dependency relation between the headword of \( A_1 \) and \( A_2 \), (2) dependency arcs within both \( A_1 \) and \( A_2 \) can form single-rooted dependency subtrees\footnote{This requires dependency trees to be projective.}.

A.2 Eisner-Satta Algorithm

The Eisner-Satta algorithm is a dynamic programming algorithm for finding the highest-scoring lexicalized tree. It iteratively combines smaller spans into larger ones to construct a complete lexicalized tree. It distinguishes between two types of spans: headed spans and hooked spans. A headed span \((i, j, h)\) is a span \((i, j)\) with a headword \(h\), while a hooked span \((i, j, h)\) is a span \((i, j)\) with an unknown headword, which is governed by a word \(h\) outside the span. The deduction process is shown in Figure 3.

During training, we use an Inside-version of the Eisner-Satta algorithm to compute the partition function \(Z(w)\), by replacing the max operator with the log-sum-exp operator. For the proposed latent lexicalization method, we use a masked version of the algorithm to enumerate all valid lexicalized trees. As shown in Algorithm 1.

<table>
  <tr>
    <th>Dataset</th>
    <th>Language</th>
    <th>#Train</th>
    <th>#Dev</th>
    <th>#Test</th>
  </tr>
  <tr>
    <td>PTB</td>
    <td>English</td>
    <td>39,832</td>
    <td>1,700</td>
    <td>2,416</td>
  </tr>
  <tr>
    <td>CTB5</td>
    <td>Chinese</td>
    <td>17,544</td>
    <td>352</td>
    <td>348</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>French</td>
    <td>14,759</td>
    <td>1,235</td>
    <td>2,541</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>German</td>
    <td>40,472</td>
    <td>5,000</td>
    <td>5,000</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>Korean</td>
    <td>23,010</td>
    <td>2,066</td>
    <td>2,287</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>Hungarian</td>
    <td>8,146</td>
    <td>1,051</td>
    <td>1,009</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>Basque</td>
    <td>7,577</td>
    <td>948</td>
    <td>946</td>
  </tr>
  <tr>
    <td>SPMRL</td>
    <td>Polish</td>
    <td>6,578</td>
    <td>821</td>
    <td>822</td>
  </tr>
</table>

Table 10: Data statistics. We present the number of sentences in the training, development, and test sets.

<table>
  <tr>
    <th>Params</th>
    <th>BERT</th>
  </tr>
  <tr>
    <td>MLP Input Size</td>
    <td>768/1024</td>
  </tr>
  <tr>
    <td>MLP Layers</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Con/Dep BiAffine Size</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Label BiAffine Size</td>
    <td>100</td>
  </tr>
  <tr>
    <td>BERT Dropout</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Other Dropout</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
  <tr>
    <td>BERT Learning Rate</td>
    <td>\( 5 \times 10^{-5} \)</td>
  </tr>
  <tr>
    <td>Other Learning Rate</td>
    <td>\( 1 \times 10^{-3} \)</td>
  </tr>
  <tr>
    <td>Batch Size (tokens)</td>
    <td>1000</td>
  </tr>
</table>

Table 11: Parameters settings.

B Experiments

B.1 Implementation Details

The data statistics for our experiments are summarized in Table 10. We adopt the majority of hyperparameters from Zhang et al. (2020). A key modification in our approach is the removal of the cascaded LSTM layers traditionally placed above BERT. Instead, we directly fine-tune the BERT model. Specifically, we employ bert-large-cased for English, bert-base-chinese for Chinese, and bert-base-multilingual-cased for the SPMRL dataset. For PTB and CTB, we fine-tune BERT for 10 epochs, and for SPMRL, we fine-tune for 15 epochs for high-resource languages and 100 epochs for low-resource languages. The detailed hyperparameters are listed in Table 11. Experiments are conducted on a single NVIDIA V100 GPU. Results are averaged over three runs with different random seeds.
Algorithm 1 Eisner-Satta Algorithm

1: Input: span scores \( s(i, j) \in \mathbb{R}^{n \times n} \)
2: ▷ scores of crossing spans are masked to \( -\infty \) (Section 3.2)
3: Input: arc scores \( s(h \to m) \in \mathbb{R}^{(n+1) \times n} \)
4: ▷ some arc scores are masked to \( -\infty \) based on linguistic constraints (Section 3.3)
5: Define: \( \alpha, \beta \in \mathbb{R}^{n \times n \times (n+1)} \)
6: Initialize: \( \alpha_{:, :, :} = 0, \beta_{:, :, :} = 0 \)
7: for \( w = 1, \ldots, n \) do
8:     for \( i = 1, \ldots, n - w \) do
9:         \( j = i + w \)
10:        for \( h = i, \ldots, j \) do
11:            \( \alpha_{i,j,h} = s(i, j) + \max_{k \leq h \leq j} (\alpha_{i,k,h} + \beta_{k+1,j,h}; \beta_{i,k,h} + \alpha_{k+1,j,h}) \)
12:            ▷ Inside version: \( \log \sum_{i \leq k \leq j} [\exp(\alpha_{i,k,h} + \beta_{k+1,j,h}) + \exp(\beta_{i,k,h} + \alpha_{k+1,j,h})] \)
13:        end for
14:        for \( h = 0, \ldots, n \) do
15:            \( \beta_{i,j,h} = \max_{l \leq m \leq j} (\alpha_{i,j,m} + s(h \to m)) \)
16:            ▷ Inside version: \( \log \sum_{i \leq m \leq j} \exp(\alpha_{i,j,m} + s(h \to m)) \)
17:        end for
18:    end for
19: end for
20: return \( \beta_{1,n,0} \)

<table>
  <tr>
    <th>Model</th>
    <th>English</th>
    <th>Chinese</th>
    <th>French</th>
    <th>German</th>
    <th>Korean</th>
    <th>Hungarian</th>
    <th>Basque</th>
    <th>Polish</th>
  </tr>
  <tr>
    <td>Con</td>
    <td>95.79</td>
    <td>91.93</td>
    <td>87.50</td>
    <td>90.45</td>
    <td>89.45</td>
    <td>94.71</td>
    <td>91.26</td>
    <td>95.97</td>
  </tr>
  <tr>
    <td>Lex (head)</td>
    <td>95.76</td>
    <td>91.86</td>
    <td>87.63</td>
    <td>90.46</td>
    <td>89.48</td>
    <td>94.38</td>
    <td>91.12</td>
    <td>95.68</td>
  </tr>
  <tr>
    <td>Lex (left)</td>
    <td>95.88</td>
    <td>91.94</td>
    <td>87.52</td>
    <td>90.42</td>
    <td>89.36</td>
    <td>94.48</td>
    <td>91.34</td>
    <td>96.00</td>
  </tr>
  <tr>
    <td>Lex (right)</td>
    <td>95.74</td>
    <td>91.96</td>
    <td>87.39</td>
    <td>90.57</td>
    <td>89.49</td>
    <td>94.80</td>
    <td>91.52</td>
    <td>95.87</td>
  </tr>
  <tr>
    <td>Lex (latent) (n=4)</td>
    <td>95.87</td>
    <td>92.16</td>
    <td><b>87.75</b></td>
    <td>90.03</td>
    <td>89.43</td>
    <td>94.78</td>
    <td>91.07</td>
    <td>95.89</td>
  </tr>
  <tr>
    <td>Lex (latent) (n=8)</td>
    <td><b>95.97</b></td>
    <td><b>92.20</b></td>
    <td>87.55</td>
    <td>90.08</td>
    <td><b>89.50</b></td>
    <td>94.44</td>
    <td>91.34</td>
    <td><b>96.17</b></td>
  </tr>
</table>

Table 12: Max-margin training results on PTB (English), CTB5 (Chinese) and SPMRL (covering French, German, Korean, Hungarian, Basque and Polish). For latent lexicalization, we report the results with different numbers of sampled trees (\( n \)).

<table>
  <tr>
    <th>Model</th>
    <th>Sent/Sec.</th>
  </tr>
  <tr>
    <td>Zhou and Zhao (2019)</td>
    <td>158.7</td>
  </tr>
  <tr>
    <td>Chen et al. (2021)</td>
    <td><b>411.2</b></td>
  </tr>
  <tr>
    <td>Gu et al. (2024)</td>
    <td>305.8</td>
  </tr>
  <tr>
    <td>Con</td>
    <td>254.4</td>
  </tr>
  <tr>
    <td>Lex (latent)</td>
    <td>249.7</td>
  </tr>
</table>

Table 13: Decoding speed comparison on PTB test set.

\[
\mathcal{L} = \max(0, \Delta(\hat{\mathbf{t}}, t^*) + s(\hat{\mathbf{t}}) - s(t^*))
\]

For latent lexicalization, a possible approach to apply max-margin loss is to enumerate all valid lexicalized trees and compute the margin loss for each tree.

\[
\mathcal{L} = \max(0, s(\hat{\mathbf{t}}) - \frac{\sum_{t^* \in \mathcal{T}} s(t^*)}{|\mathcal{T}|} + \frac{\Delta(\hat{\mathbf{t}}, \mathcal{T})}{|\mathcal{T}|})
\]

However, these exists two key challenges:
1. The number of trees in the forest (i.e. \( |\mathcal{T}| \)) grows exponentially, making it difficult to determine the exact count.
<table>
  <tr>
    <th>Phrase</th>
    <th>Head</th>
    <th>Count</th>
    <th>Ratio (%)</th>
    <th>Headword</th>
  </tr>
  <tr>
    <td rowspan="4">NP</td>
    <td>NN</td>
    <td>5823</td>
    <td>31.3</td>
    <td>“$”, “%”,</td>
  </tr>
  <tr>
    <td>NP</td>
    <td>4443</td>
    <td>23.9</td>
    <td>it, company,</td>
  </tr>
  <tr>
    <td>NNS</td>
    <td>3273</td>
    <td>17.6</td>
    <td>he, market,</td>
  </tr>
  <tr>
    <td>NNP</td>
    <td>2939</td>
    <td>15.8</td>
    <td>year, Corp.</td>
  </tr>
  <tr>
    <td rowspan="4">VP</td>
    <td>VP</td>
    <td>2745</td>
    <td>31.4</td>
    <td>said, have,</td>
  </tr>
  <tr>
    <td>VBD</td>
    <td>1416</td>
    <td>16.2</td>
    <td>expected, says,</td>
  </tr>
  <tr>
    <td>VB</td>
    <td>1318</td>
    <td>15.1</td>
    <td>is, buy,</td>
  </tr>
  <tr>
    <td>VBN</td>
    <td>900</td>
    <td>10.3</td>
    <td>take, had</td>
  </tr>
  <tr>
    <td rowspan="2">S</td>
    <td>VP</td>
    <td>5359</td>
    <td>94.6</td>
    <td>said, is,</td>
  </tr>
  <tr>
    <td>S</td>
    <td>195</td>
    <td>3.4</td>
    <td>have, says</td>
  </tr>
  <tr>
    <td rowspan="4">PP</td>
    <td>IN</td>
    <td>4971</td>
    <td>90.5</td>
    <td>of, in,</td>
  </tr>
  <tr>
    <td>TO</td>
    <td>394</td>
    <td>7.2</td>
    <td>for, to,</td>
  </tr>
  <tr>
    <td>VBG</td>
    <td>64</td>
    <td>1.2</td>
    <td>on, by,</td>
  </tr>
  <tr>
    <td>PP</td>
    <td>29</td>
    <td>0.5</td>
    <td>with, from</td>
  </tr>
  <tr>
    <td rowspan="2">SBAR</td>
    <td>S</td>
    <td>1758</td>
    <td>97.8</td>
    <td>have, is,</td>
  </tr>
  <tr>
    <td>SBAR</td>
    <td>27</td>
    <td>1.5</td>
    <td>had, expects</td>
  </tr>
  <tr>
    <td rowspan="4">ADVP</td>
    <td>RB</td>
    <td>1065</td>
    <td>87.8</td>
    <td>also, down,</td>
  </tr>
  <tr>
    <td>RBR</td>
    <td>50</td>
    <td>4.1</td>
    <td>ago, still,</td>
  </tr>
  <tr>
    <td>IN</td>
    <td>47</td>
    <td>3.9</td>
    <td>up, however,</td>
  </tr>
  <tr>
    <td>ADVP</td>
    <td>21</td>
    <td>1.7</td>
    <td>back, well</td>
  </tr>
  <tr>
    <td rowspan="2">ADJP</td>
    <td>JJ</td>
    <td>598</td>
    <td>67.0</td>
    <td>“$”, million,</td>
  </tr>
  <tr>
    <td>ADJP</td>
    <td>61</td>
    <td>6.8</td>
    <td>likely, higher</td>
  </tr>
  <tr>
    <td rowspan="2">QP</td>
    <td>$</td>
    <td>249</td>
    <td>50.8</td>
    <td>million, “%”</td>
  </tr>
  <tr>
    <td>CD</td>
    <td>196</td>
    <td>40.0</td>
    <td>1/2, billion</td>
  </tr>
  <tr>
    <td rowspan="2">WHNP</td>
    <td>WDT</td>
    <td>270</td>
    <td>62.9</td>
    <td>that, which,</td>
  </tr>
  <tr>
    <td>WP</td>
    <td>106</td>
    <td>24.7</td>
    <td>who, what</td>
  </tr>
</table>

Table 14: Most frequent heads for each constituent label on SD.

<table>
  <tr>
    <th>Head \( \bowtie \) Dependent</th>
    <th>Count</th>
    <th>Ratio (%)</th>
  </tr>
  <tr>
    <td>NN \( \bowtie \) DT</td>
    <td>3554</td>
    <td>6.27</td>
  </tr>
  <tr>
    <td>NNP \( \bowtie \) NNP</td>
    <td>2489</td>
    <td>4.39</td>
  </tr>
  <tr>
    <td>IN \( \bowtie \) NN</td>
    <td>2148</td>
    <td>3.79</td>
  </tr>
  <tr>
    <td>NN \( \bowtie \) JJ</td>
    <td>1706</td>
    <td>3.01</td>
  </tr>
  <tr>
    <td>NN \( \bowtie \) IN</td>
    <td>1667</td>
    <td>2.94</td>
  </tr>
  <tr>
    <td>NN \( \bowtie \) NN</td>
    <td>1304</td>
    <td>2.30</td>
  </tr>
  <tr>
    <td>IN \( \bowtie \) NNS</td>
    <td>1300</td>
    <td>2.29</td>
  </tr>
  <tr>
    <td>NNS \( \bowtie \) JJ</td>
    <td>1211</td>
    <td>2.14</td>
  </tr>
  <tr>
    <td>IN \( \bowtie \) NNP</td>
    <td>905</td>
    <td>1.60</td>
  </tr>
  <tr>
    <td>ROOT \( \bowtie \) VBD</td>
    <td>879</td>
    <td>1.55</td>
  </tr>
</table>

Table 15: Top ten dependency relationships on SD.

<table>
  <tr>
    <th>Model</th>
    <th>Arc Distance</th>
    <th>Root Distance</th>
  </tr>
  <tr>
    <td>SD</td>
    <td>3.4</td>
    <td>8.9</td>
  </tr>
  <tr>
    <td>Ours</td>
    <td>4.1</td>
    <td>9.6</td>
  </tr>
</table>

Table 16: Average dependency arc distance, with the root arc listed separately.

lexicalization is negligible in real-world scenarios.

B.4 Head Selection from Stanford Dependencies

In this section, we provide the gold-standard head selection by the Stanford Dependencies (SD). The phrase head selection is shown in Table 14, and the dependency relationships are shown in Table 15.

B.5 Arc Distance

We analyze the average distance between heads and their dependents, as well as the distance of the root to the main verb. Table 16 shows that predicted dependencies tend to span longer distances than gold-standard ones. While the dependency structures are constrained by the constituency structures, this results indicate that the inferred dependencies are not limited to local relationships.

2. Summing over all tree scores requires a dynamic programming algorithm (Eisner-Satta in our case). However, for numerical stability, it computes \( \log \sum_{t^* \in T} \exp(s(t^*)) \) rather than \( \sum_{t^* \in T} s(t^*) \), which is necessary for max-margin loss.

A potential solution involves sampling a subset of trees from the forest, and we present the results using 4 and 8 sampled trees.

B.3 Parsing Speed Comparison

Parsing efficiency is critical for practical applications. Table 13 compares the parsing speed of our model with Con on PTB. Although our model has a higher theoretical complexity (\( O(n^4) \) vs. \( O(n^3) \)), both achieve comparable speeds in practice due to the GPU-optimized batch processing. This demonstrates that the added computational cost of latent

Reviewer paper summaries: This paper proposes a latent lexicalization framework for constituency parsing, which dynamically infers headwords during training. The model takes head selection as a latent variable, so lexical dependencies can be acquired from data, not predefined head-finding rules. The authors employ BERT and BiAffine to compute tree scores, and TreeCRF loss to optimize the parser. Experimental results verify that the latent lexicalization framework outperforms other predefined head lexicalization methods.

This paper described a constituency parsing methodology that hybridised with lexical head information. The training process may not need annotated head information but the models manage to explore, find, and learn such information. The model performance is fair in terms of accuracy (F1 scores) and the results are interesting and well demonstrated with analysis.
Summary of strengths: 1. The proposed latent lexicalization method.
2. Thorough experiments and detailed analyses.

1. This work is a new instance of hybridize both phrase and lexical head to achieve a harmony mutualistic symbiosis for both tasks.
2. This work reused old models (BERT, TreeCRF, and Biaffine attention) to make fair comparison but the combination and methodology is new.
3. This work revisited old concepts such as parsing, lexicaliztation, and CNF. The introduction is friendly to read for new researchers. Both symbolic/explict and neural/implicit lexicaliztations are both mentioned.
4. The experiment is comprehensive on the major treebanks.
Summary of weaknesses: 1. The framework is based on TreeCRF loss, but it could be applied to max-margin loss (Kitaev and Klein, 2018) as well. Adding the corresponding results will make the paper very solid.
2.  Please conduct experiments on the SPMRL dataset for Table 1.

1. Although the introduction is easy to read, the landscape of constituency parsing is not complete. Constituency is not just about continuous spans, there is a branch of neural parsing methods focusing on discontinuous spans or phrases.
2. Attempts to joint constituency parsing and dependency parsing should also be mentioned in the Background section. And it is not clear why the authors choose this configuration of models for the task.
3. They talked about the complexity of decomposing trees but not the complexity of the parsing method. What type does the parser belong to? Chart or transition-based?
Comments, suggestions and typos: Line 221: To clarify the notion of “latent” in this context’, remove ' at the end.

There is a short series of similar work that is lacking:
Zhousi Chen, Longtu Zhang, Aizhan Imankulova, and Mamoru Komachi. Neural Combinatory Constituency Parsing. Findings of ACL-IJCNLP 2021.
Zhousi Chen and Mamoru Komachi. Discontinuous Combinatory Constituency Parsing. Transactions of the Association for Computational Linguistics 11 (2023): 267-283.
Their parsers also touched different similar strategies, such as CNF and random, and head word selection. Although their F1 scores are not as high as this work, their parsing speeds are the highest with low complexity. Head words are also a by-product, the same as this work. Besides, some parsers do not require binarization to find head words.
Overall assessment scores: 3.5

4
Metareview score: 5
Metareview: This work proposes latent lexicalization, a variant of lexicalized constituency parsing, in which lexical information associated with each constituency label is induced during training instead of a by-product of binarization, i.e., left, right or head binarization. Training is carried out by efficiently summing over all possible latent trees with Eisner-Satta algorithm built upon a BERT-based biaffine model. Experiments show gains over baselines on the standard benchmark datasets with diverse languages.
Summary of reasons to publish: * This work revives the lexicalized constituency parsing and introduces a novel approach of latent lexicalization. The approach is well-motivated and presentation is very clear.
* Experiments are carried out in diverse languages with detail comparisons against other approaches in a systematic manner. Analyses are comprehensive enough to justify the gains.