Improving Unsupervised Constituency Parsing via Maximizing Semantic Information

## Abstract

Abstract Unsupervised constituency parsers organize phrases within a sentence into a tree-shaped syntactic constituent structure that reflects the organization of sentence semantics.
However, the traditional objective of maximizing sentence log-likelihood (LL) does not explicitly account for the close relationship between the constituent structure and the semantics, resulting in a weak correlation between LL values and parsing accuracy.
In this paper, we introduce a novel objective for training unsupervised parsers: maximizing the information between constituent structures and sentence semantics (SemInfo).
We introduce a bag-of-substrings model to represent the semantics and apply the probability-weighted information metric to estimate the SemInfo.
Additionally, we develop a Tree Conditional Random Field (TreeCRF)-based model to apply the SemInfo maximization objective to Probabilistic Context-Free Grammar (PCFG) induction, the state-of-the-art method for unsupervised constituency parsing.
Experiments demonstrate that SemInfo correlates more strongly with parsing accuracy than LL.
Our algorithm significantly enhances parsing accuracy by an average of 7.85 points across five PCFG variants and in four languages, achieving new state-of-the-art results in three of the four languages.

## 1 Introduction

Unsupervised constituency parsing is a syntactic task of organizing phrases of a sentence into a tree-shaped and unlabelled constituent structure without relying on any linguistic annotations (Klein & Manning, ).
The constituent structure is a fundamental tool in analyzing sentence semantics (i.e., the meaning) (Carnie, ; Steedman, ) and can significantly improve performance for downstream Natural Language Processing systems, such as natural language inference (He et al., ), machine translation (Xie & Xing, ) and semantic role labeling (Chen et al., ).
Each constituent in the structure corresponds to a meaningful substring in the sentence, which guides us to progressively construct the sentence semantics.
Figure 1 illustrates the progressive semantic construction of the sentence “John has been working on a theory until late night”.
This example demonstrates that *constituent substrings in the sentence carry significant semantic information*, illustrating an alignment between syntax and semantics.

Figure: Figure 1: An illustration of the progressive semantics build-up in accordance with the constituent structure. The tree structure in the top-right shows the simplified constituent structure for illustration purposes. Constituent strings are highlighted in blue.
Refer to caption: x1.png

Maximizing sentence log-likelihood has traditionally been the primary training objective for unsupervised constituency parsers (Eisner, ; Kim et al., ).
However, Log-Likelihood (LL), the objective function, does not explicitly factor in the syntax-semantics alignment.
This leads to a poor correlation between the LL value and the parsing accuracy, which we will discuss further in Section 5.3.
As pointed out in previous research, training a Probabilistic Context-Free Grammar (PCFG) parser that outperforms trivial baselines with the LL maximization objective is challenging (Carroll & Charniak, ; Kim et al., ).
Successful training commonly involves altering the LL maximization objective, such as imposing sparsity constraints (Cohen et al., ; Johnson et al., ) or heuristically estimating the LL value (Spitkovsky et al., ).
Evidence suggests that the LL function might not provide robust information to distinguish between constituents and non-constituents, rendering LL an insufficient objective function for unsupervised parsing.

In this paper, we propose a novel objective for unsupervised constituency parsing: maximizing the semantic information that constituent structures carry.
We propose a novel method for measuring *SemInfo*, the information between constituent structures and sentence semantics.
Specifically, we introduce a bag-of-substrings model to represent the sentence semantics and quantify semantic information encoded in each substring using the probability-weighted information metric developed for bag-of-words models (Aizawa, ).
We calculate the SemInfo value by aggregating the substring-semantic information associated with the constituent structure.
Experiments show a much stronger correlation between SemInfo and the parsing accuracy than the correlation between LL and the parsing accuracy, suggesting that SemInfo is an effective objective function for unsupervised constituency parsing.
In addition, we develop a Tree Conditional Random Field (TreeCRF)-based model to apply the SemInfo maximization objective to PCFG induction, the state-of-the-art method for unsupervised constituency parsing.
Experiments demonstrate that the SemInfo maximization objective improves the PCFG’s parsing accuracy by 7.85 points across five latest PCFG variants and in four languages.

Our main contributions are: (1) Proposing a novel method for estimating SemInfo, the information between constituent structures and sentence semantics. (2) Demonstrating a strong correlation between SemInfo values and parsing accuracy. (3) Developing a TreeCRF model to apply the SemInfo maximization objective to PCFG induction, significantly improving parsing accuracy and setting new state-of-the-art results in three of the four languages tested.

## 2 Background

The idea that constituent structures reflect the organization of sentence semantics is central to modern linguistic studies (Steedman, ; Pollard & Sag, ).
A constituent is a substring $s$ in a sentence $x$ that can function independently (Carnie, ) and carries individual meaning (Heim & Kratzer, ).
A collection of constituents forms a tree-shaped structure $t$, which we can represent as a collection of its constituent strings $t=\{s_{1},s_{2},...\}$.
For example, the constituent structure in the top right of Figure 1 can be represented as $\{$“a theory”, “until late night”,…$\}$.
Previous research evaluates unsupervised constituency parsers using sentence-F1, the F1 score computed from the string collection representing the predicted structure and the collection representing the gold structure (Shen et al., ).

Probability-Weighted Information (PWI) (Aizawa, ) is fundamental to our SemInfo estimation method.
PWI provides an information-theoretic interpretation of the term frequency-inverse document frequency (tf-idf) statistics calculated in Bag-of-Words (BoW) models.
Let ${\mathcal{D}}$ denote a document corpus, $d_{i}$ the $i$-th document in the corpus, and $w_{ij}$ the $j$-th word in $d_{i}$.
The BoW model represents the document $d_{i}$ as an unordered collection of words occurring in the document (i.e., $d_{i}=\{w_{i1},w_{i2},...\}$).
Tf-idf, as shown in Equation 1, is the product of the term frequency $F(w_{ij},d_{i})$ (i.e. the frequency of $w_{ij}$ occurring in $d_{i}$) and the inverse document frequency (i.e. the inverse log-frequency of documents containing $w_{ij}$).
Tf-idf is an important feature in finding keywords in documents (Li et al., ) or in efficiently locating documents based on the given keyword (Mishra & Vishwakarma, ).
PWI interprets the term frequency as the word generation probability and the inverse document frequency as the piecewise word-document information (Equation 2).
Intuitively, a word with a high tf-idf value suggests that it carries a significant amount of information about the document (i.e., a keyword of the document).
Aizawa () further interpret scaling or smoothing of the term frequency as variations in estimating $P(w_{ij}|d_{i})$.

$$ $\displaystyle\text{tf-idf}(w_{ij},d_{i})$ $\displaystyle=\underbrace{\vphantom{\left(\frac{a^{0.3}}{b}\right)}F(w_{ij},d_ {i})}_{\text{term frequency}}\times\underbrace{\log\frac{|{\mathcal{D}}|}{|d^{ \prime}:d^{\prime}\in{\mathcal{D}}\wedge w_{ij}\in d^{\prime}|}}_{\text{ inverse document frequency}}$ (1) $\displaystyle\approx\underbrace{\vphantom{\left(\frac{a^{0.3}}{b}\right)}P(w_{ ij}|d_{i})}_{\text{word generation probability}}\times\underbrace{\log\frac{P( d_{i}|w_{ij})}{P(d_{i})}}_{\text{piecewise word-document information}}$ (2) $\displaystyle=PWI(w_{ij},d_{i})$ $$

PCFG induction is currently the state-of-the-art method for training unsupervised constituency parsers (Liu et al., ; Yang et al., ).
This method trains a binary PCFG parser over a text corpus by maximizing the average LL of the corpus.
A PCFG is a generative model defined by a tuple $(NT,T,R,S,\pi)$, where $NT$ is the set of non-terminal symbols, $T$ is the set of terminal symbols, $R$ is the set of production rules, $S$ is the start symbol, and $\pi$ is the probability distribution over the rules.
The generation process starts with the start symbol $S$ and iteratively applies non-terminal expansion rules ($A\rightarrow BC:A,B,C\in NT$) or terminal rewriting rules ($A\rightarrow w:A\in NT,w\in T$) until it produces a complete sentence $x$.
We can represent the generation process with a tree-shaped structure $t$.
The PCFG assigns a probability for each distinct way of generating $x$, defining a distribution $P(x,t)$.
The Inside-Outside algorithm (Baker, ) provides an efficient solution for computing the total sentence probability $P(x)=\mathop{\sum}_{t}P(x,t)$.
The algorithm constructs a $\beta(s,A)$ table that records the total probability of generating a substring $s$ of $x$ from the non-terminal $A$.
The sentence probability can be calculated as $P(x)=\beta(x,S)$, the probability of $x$ being generated from the start symbol $S$.
The $\beta(x,S)$ quantity is commonly referred to as $Z(X)$ (Eisner, ).
Besides the total sentence probability, the $\beta$ table can also be used to calculate the span-posterior probability of $s$ being a constituent (Eisner, ) (Equation 3).(^1^11We explain the derivation in more detail in Section A.1.)

$$ $P(s\text{ is a constituent}|x)=\sum_{A\in NT}\frac{\partial\log Z(x)}{\partial \log\beta(s,A)}$ (3) $$

Span-based TreeCRF models are widely adopted in constituency parsers (Kim et al., ; Stern et al., ) to model the parser distribution $P(t|x)$, the probability of constituent structure $t$ given $x$.
The span-based TreeCRF model determines the probability of $t$ by evaluating whether all substrings involved in the structure are constituents.
The TreeCRF model will assign a high score to a substring $s$ in its potential function $\phi(s,x)$ if $s$ is likely a constituent.
Subsequently, the TreeCRF model can represent the parser distribution as $P(t|x)\propto\sum_{s\in t}\phi(s,x)$.
Using the TreeCRF framework, we can interpret the Tree Minimum Bayesian Risk (TreeMBR) decoding of PCFG (Yang et al., ) as a Viterbi decoding of a TreeCRF model.
In this interpretation, the TreeCRF model uses the substring’s span-posterior probability as its potential value (i.e., $\phi(s,x)=P(s\text{ is a constituent}|x)$).
Intuitively, the more likely a substring is a constituent, the more likely the TreeCRF model will generate a structure including the substring as a constituent.
Consequently, the parser distribution derived from the TreeMBR decoding algorithm is defined as follows.

$$ $P(t|x)\propto\sum_{s\in t}\sum_{A\in NT}\frac{\partial\log Z(x)}{\partial\log \beta(s,A)}$ (4) $$

## 3 SemInfo: A Metric of Information between Constituent Structures and Semantics

In this section, we introduce our estimation method of SemInfo, the information between constituent structures and sentence semantics.
We first introduce a bag-of-substrings model, representing the semantics of a sentence by examining whether substrings of the sentence would be *regenerated* during a paraphrasing process.
We then apply the PWI (Aizawa, ) metric to measure the substring-semantics information.
For a given constituent structure, we estimate its SemInfo value by aggregating the substring-semantics information associated with the structure.

Figure: Figure 2: Bag-of-Substrings model derived from a paraphrasing model
Refer to caption: x2.png

Our bag-of-substrings model is based on the paraphrasing model $P(x^{p}|Sem(x))$ shown in Figure 2.
The paraphrasing model takes a source sentence $x$ as input, internally analyzes its semantics $Sem(x)$, and generates a paraphrase $x^{p}$.
We can repeatedly sample from the process, collecting a paraphrase set ${\mathbb{X}}^{p}=\{x^{p}_{1},x^{p}_{2},...\}$.
To estimate the semantic information for substrings of $x$, we examine how often these substrings are regenerated by the paraphrasing model.
Specifically, if a substring $s$ from $x$ is regenerated in any paraphrase $x^{p}_{i}$, we consider $s$ to be generated by $Sem(x)$.
We then define a bag-of-substrings representation of $Sem(x)$ by aggregating the regeneration observations across ${\mathbb{X}}^{p}$.

We apply the PWI metric to quantify the substring-semantics information $I(s,Sem(x))$ between $s$ and $Sem(x)$, leveraging the similarity between our bag-of-substrings model and the traditional bag-of-words model.
To facilitate the PWI calculation (Equation 5), we define two components: $P(s|Sem(x))$, the substring generation probability, and $\log\frac{P(Sem(x)|s)}{P(Sem(x))}$, the piecewise mutual information between $s$ and $Sem(x)$.
Following standard practices in bag-of-words models (Aizawa, ; Blei et al., ), we assume that sentences in the corpus ${\mathcal{D}}$ are equally likely and each represents distinct semantics.
The two assumptions yield an empirical distribution $P(Sem(x))=\frac{1}{{\mathcal{D}}}$, where $|{\mathcal{D}}|$ is the size of the corpus.

$$ $I(s,Sem(x))=\frac{1}{|{\mathcal{D}}|}\vphantom{\left(\frac{a^{0.3}}{b}\right)} P(s|Sem(x))\log\frac{P(Sem(x)|s)}{P(Sem(x))}$ (5) $$

### 3.1 Defining P ⁢ ( s | S ⁢ e ⁢ m ⁢ ( x ) ) 𝑃 conditional 𝑠 𝑆 𝑒 𝑚 𝑥 P(s|Sem(x)) italic_P ( italic_s | italic_S italic_e italic_m ( italic_x ) ) and log ⁡ P ⁢ ( S ⁢ e ⁢ m ⁢ ( x ) | s ) P ⁢ ( S ⁢ e ⁢ m ⁢ ( x ) ) 𝑃 conditional 𝑆 𝑒 𝑚 𝑥 𝑠 𝑃 𝑆 𝑒 𝑚 𝑥 \log\frac{P(Sem(x)|s)}{P(Sem(x))} roman_log divide start_ARG italic_P ( italic_S italic_e italic_m ( italic_x ) | italic_s ) end_ARG start_ARG italic_P ( italic_S italic_e italic_m ( italic_x ) ) end_ARG with Maximal Substring Frequency

Figure: Figure 3: An example for naive substring frequency among paraphrases failing to estimate $P(s|Sem(x))$.
Refer to caption: x3.png

Frequency is a simple but effective empirical estimator of distributions.
However, naively measuring substring frequency among paraphrases ${\mathbb{X}}^{p}$ will yield a misleading estimate of $P(s|Sem(x))$, the probability of $s$ being generated to carry information of $Sem(x)$.
The reason is that one substring can be nested in another substring.
If a substring $s$ is generated to convey semantic information, we will observe an occurrence of $s$ along with an occurrence of all its substrings.
Hence, the naive substring frequency will wrongly count substring occurrences caused by the generation of larger substrings as occurrences caused by $P(s|Sem(x))$.
Let’s consider the example illustrated in Figure 3.
All three substrings in the example have a frequency of 2, yet only the first substring carries significant semantic information.
This is because the occurrence of the first substring causes the occurrence of the second and third substrings.
The true frequency of the second and third substrings should be 0 instead of 2.

We introduce the notion of maximal substring to counter this problem.
Given a source sentence $x$ and a paraphrase $x^{p}_{i}$, the maximal substring between the two is defined in Equation 6.
Intuitively, a maximal substring is the largest substring that occurs in both $x$ and $x^{p}_{i}$.
Formally, we denote the partial order relationship of string $\alpha$ being a substring in string $\beta$ by $\alpha\leq\beta$, and denote the set of maximal substrings by $MS(x,x^{p}_{i})$.
Using maximal substrings, we can avoid over-counting substring occurrences caused by the generation of larger substrings.

$$ $MS(x,x^{p}_{i}):=\{\alpha:\alpha\leq x\wedge\alpha\leq x^{p}_{i}\wedge\forall \alpha^{\prime}(\alpha<\alpha^{\prime}\implies\neg\alpha^{\prime}\leq x\vee \neg\alpha^{\prime}\leq x^{p}_{i})\}$ (6) $$

We are now ready to define $P(s|Sem(x))$ using the paraphrasing distribution $P(x^{p}|Sem(x))$ and the notion of maximal substrings.
We define $P(s|Sem(x))$ to be proportional to $s$’s expected probability of being generated as a maximal substring in paraphrases (Eqaution 7).
Here, $C$ is the normalizing constant.
Intuitively, the more likely $s$ is generated as a maximal substring, the more likely $s$ is needed to carry semantic information.
We approximate the expectation using Monte-Carlo estimation with $N$ samples from the paraphraser.
We replace $P(s\in MS(x^{p}_{i},x))$ with an indicator function $\bm{1}(s\in MS(x^{p}_{i},x))$ since a substring is either a maximal substring ($P(s\in MS(x^{p}_{i},x))=1$) or a non-maximal substring ($P(s\in MS(x^{p}_{i},x))=0$).
As suggested in Equation 9, we can compute $P(s|Sem(x))$ by calculating $F(s,{\mathbb{X}}^{p})$, $s$’s maximal substring frequency in ${\mathbb{X}}^{p}$.

$$ $\displaystyle P(s|Sem(x))$ $\displaystyle=\frac{1}{C}\mathop{\mathbb{E}}_{x_{i}^{p}\sim Sem}P(s\in MS(x_{i }^{p},x))$ (7) $\displaystyle\approx\frac{1}{C}\frac{1}{N}\sum_{x_{i}^{p}\in{\mathbb{X}}^{p}} \bm{1}(s\in MS(x_{i}^{p},x))$ (8) $\displaystyle=\frac{1}{CN}F(s,{\mathbb{X}}^{p})$ (9) $$

Similarly, we define the inverse document frequency for maximal substrings (Equation 10).
The inverse document frequency can serve as an estimate of the piecewise substring-semantics information.
The inverse document frequency quantifies how infrequent $s$ is generated in sentences with distinct semantics.
A high inverse document frequency implies that we can easily identify the target semantics $Sem(x)$ among all possible semantics.

$$ $\displaystyle\log\frac{P(Sem(x)|s)}{P(Sem(x))}$ $\displaystyle\approx\log\frac{|{\mathcal{D}}|}{|\{x^{\prime}:x^{\prime}\in{ \mathcal{D}}\wedge s\in MS(x,x^{\prime})\}|}$ (10) $$

### 3.2 SemInfo Metric

A constituent structure $t$ can be represented as a set of constituent substrings.
We define SemInfo, the information between the structure $t$ and the semantics $Sem(x)$, as the cumulative substring-semantics information associated with the structure (Equation 11).
We estimate the substring-semantics information with the maximal substring frequency-inverse document frequency developed in the above sections.

$$ $\displaystyle I(t,Sem(x))$ $\displaystyle=\sum_{s\in t}I(s,Sem(x))$ (11) $\displaystyle\propto\sum_{s\in t}\underbrace{F(s,{\mathbb{X}}^{p})\log\frac{|{ \mathcal{D}}|}{|\{x^{\prime}:x^{\prime}\in{\mathcal{D}}\wedge s\in MS(x,x^{ \prime})\}|}}_{\text{maximal substring frequency-inverse document frequency}}$ (12) $$

## 4 SemInfo Maximization via TreeCRF Model

We apply the SemInfo maximization objective to PCFG induction models by maximizing Equation 13.
We consider the PCFG parser a one-step reinforcement learning policy and consider the SemInfo value $I(t,Sem(x))$ the reward function.
We model the parser using the TreeCRF model derived from the TreeMBR decoding algorithm.
We choose the TreeCRF model because we can efficiently sample from the tree distribution and calculate the entropy.
We apply the REINFORCE algorithm with average baseline (Williams, ) and maximum entropy regularization (Ziebart et al., ) to maximize the expected SemInfo.
We also include the traditional LL term $\log Z(x)$ in Equation 13 because we found the inclusion significantly stabilizes the training process.
The stabilization might be related to the correlation between LL values and parsing accuracy at the early training stage, as we will discuss in Section 5.3.2.

$$ $\displaystyle{\mathcal{J}}({\mathcal{D}})=\mathop{\mathbb{E}}_{x\sim\mathcal{D }}[\log Z(x)+\mathop{\mathbb{E}}_{t\in P(t|x)}[$ $\displaystyle\log P(t|x)(I(t,Sem(x))-$ $\displaystyle\mathop{\mathbb{E}}_{t\in P(t|x)}I(t,Sem(x))+\beta H(P(t|x)))]]$ (13) $$

## 5 Experiment

### 5.1 Experiment Setup

We evaluate the effect of the SemInfo maximization objective on five latest PCFG variants: Neural-PCFG (NPCFG), Compound-PCFG (CPCFG) (Kim et al., ), TNPCFG (Yang et al., ), Simple-NPCFG (SNPCFG), and Simple-CPCFG (SCPCFG) (Liu et al., ).(^2^22Our implementation is based on the source code released by Yang et al. () and Liu et al. ())
SNPCFG and SCPCFG represent the current state-of-the-art for non-ensemble unsupervised constituency parsing.
We use 60 NTs for NPCFG and CPCFG, and 1024 NTs for TNPCFG, SNPCFG, and SCPCFG in our experiment.
We conduct the evaluations in four languages, namely English, Chinese, German, and French, as Liu et al. () did previously.
We use the Penn TreeBank (PTB) (Marcus et al., ) for the English evaluation, the Chinese Treebank 5.1 (CTB) (Palmer et al., ) for the Chinese evaluation, and the SPMRL dataset (Seddah et al., ) for the German and French evaluation.
We adopt the standard data split for the PTB dataset,
using Sections 02-21 for training, Section 22 for validation, and Section 23 for testing (Kim et al., ).
We adopt the official data split for the CTB and SPMRL datasets.

Following Shen et al. (), we train the PCFG induction model on raw text and evaluate its parsing performance using corpus-level sentence-F1 scores.
The corpus-level score is computed by averaging the sentence-F1 score across the entire corpus.
We evaluate the sentence-F1 score for sentences longer than two words and drop punctuation and trivial spans (i.e., sentence-level spans and spans with only one word).

Following Chen et al. (), we use the gpt-4o-mini-2024-07-18 model as our paraphrasing model to facilitate multilingual experiments and apply normalization to the source sentence and the paraphrases.
We use eight semantic-preserving prompts for the paraphrasing model.(^3^33Detailed prompts are listed in Section A.5)
We apply the snowball stemmer (Bird & Loper, ) to normalize the source sentence and its paraphrases before calculating the maximal substring frequency and the inverse document frequency.
We apply the log-normalization (Sparck Jones, ) to the maximal substring frequency to avoid some high-frequency substrings dominating the SemInfo value.
We found that the log-normalization variant performs marginally but consistently better than the unnormalized variant in our preliminary experiment.
The application of log-normalization is aligned with the theoretical framework of Aizawa ().

### 5.2 SemInfo Maximization Significantly Improves Parsing Accuracy

Table: Table 1: Corpus-level sentence-F1 scores of five PCFG variants trained with SemInfo and LL. Each cell in the upper section reports the mean sentence-F1 score and the standard deviation across three *identical and independently trained* PCFG models. Average $\Delta$ indicates average improvements in the sentence-F1 score when training with SemInfo compared to LL. Improvements that are statistically significant ($p<0.05$) are highlighted in bold.

Table 1 compares SemInfo-trained PCFGs and LL-trained PCFGs for five PCFG variants and in four languages.
For each variant, we independently train three PCFG models on the SemInfo and LL objectives and report the mean and standard deviation of the model’s performance.
We can observe that the SemInfo-trained PCFGs achieve significantly higher parsing accuracy than the LL-trained PCFGs.
The average improvements are 13.09, 6.02, 7.31, and 4.92 sentence-F1 scores in English, Chinese, French, and German, respectively.
Two-tailed t-tests indicate the improvement to be statistically significant (p$<$0.05) in 17 out of 20 combinations.
Two of the three insignificant results are due to the high score variance of the LL-trained PCFGs.
The significant improvement demonstrates the benefit of the SemInfo maximization objective in the unsupervised constituency parsing task.
The result also confirms the importance of semantic factors in identifying the syntactic constituent structure.

Table 1 also compares the SemInfo trained PCFG with two baseline parsers: the Maximum Tree Decoding (MTD) parser that predicts the constituent structure by finding the structure with maximum SemInfo value, and the GPT4o-mini parser that asks the GPT4o-mini model to directly predict the constituent structure.
Among the two baselines, we see that the MTD parser has significantly higher sentence-F1 scores than the GPT4o-mini parser across the four languages.
The accuracy gap indicates that SemInfo is discovering non-trivial information about the constituent structure.
Comparing the SemInfo-trained PCFG and the MTD parser, we see that all SemInfo-trained PCFG variants outperform the MTD parser in English, Chinese, and French.
The accuracy improvement indicates that the constituent information provided by the SemInfo value is noisy, and the grammar learns to mitigate the noises.
In German, SemInfo-trained PCFGs perform worse than the MTD parser.
One possible reason is that the German validation/testing set has a significantly different word vocabulary compared to the training set, unlike the datasets in the other three languages.
This shift in word distribution might be a significant factor in German PCFGs’ poor parsing accuracy.

### 5.3 SemInfo Strongly Correlates with Sentence-F1

In this section, we evaluate two aspects of the SemInfo and LL functions: 1) Whether the function can accurately evaluate the model’s prediction. 2) Whether the function can approximately rank models in accordance with their performance.
We evaluate the two aspects using Spearman correlation (Spearman, ), examining whether the objective function can rank good predictions/models higher than the bad ones.
We evaluate the prediction ranking capability using a sentence-level correlation and the model ranking capability using a corpus-level correlation.

#### 5.3.1 SemInfo is an Accurate Metric of Parsing Accuracy

The sentence-level correlation assesses the objective function’s capability to evaluate the model prediction accurately.
We independently train eight *identical* PCFG models using the LL maximization objective.
Each model is trained with a unique random seed for 30k steps.
For every sentence, the eight models produce eight (sentence-F1, SemInfo, LL) tuples, which we use to calculate the sentence-level Spearman correlation coefficient.

Figure 4 illustrates the SemInfo-sentence-F1 correlation and the LL-sentence-F1 correlation using a random sentence in the English validation set.
We observed a strong positive correlation for the (SemInfo, sentence-F1) pairs but observed no positive correlation for the (LL, sentence-F1) pairs.
Table 2 demonstrates the correlation in a more statistically reliable way.
We perform mean-aggregation on the sentence-level correlation coefficient using Fisher’s Z transformation (Fisher, ).
Fisher’s Z transformation converts the correlation coefficient to a uni-variance space, reducing the negative impact caused by the skewness in the coefficient’s distribution(Silver & Dunlap, ).
The aggregated coefficients for the SemInfo-sentence-F1 correlation range from 0.6-0.9, whereas the aggregated coefficients for the LL-sentence-F1 correlation center around 0.
We can consistently observe the correlation coefficient gap in different stages of training, as discussed in Appendix A.2.
This sentence-level correlation analysis demonstrates a strong positive correlation between SemInfo and sentence-F1, while identifying no apparent correlation between LL and sentence-F1.
The high SemInfo-sentence-F1 coefficient indicates that SemInfo can serve as an accurate metric of parsing accuracy.
The gap in correlation coefficients suggests that SemInfo might be a better objective function for unsupervised constituency parsing than LL.

#### 5.3.2 SemInfo Ranks PCFG Models Better Than LL

The corpus-level correlation evaluates the objective functions’ capability to rank models in accordance with their performance.
We examine the correlation using model checkpoints collected over different training stages of the above eight PCFG models.
Each stage is represented by a window over the amount of training steps.
For example, a stage [1k, 10k] contains checkpoints from 1k to 10k steps.
These checkpoints produce corpus-level sentence-F1 scores, average SemInfo values, and average LL values, which we use to calculate the corpus-level coefficient at that training stage.

Figure: Figure 5: Spearman $\rho$ with sentence-F1 (denoted as SF1) in different training stages of NPCFG.
Refer to caption: extracted/5898961/figures/sw_corr.png

Figure 5 illustrates the SemInfo-sentence-F1 and LL-sentence-F1 correlation curves for NPCFG.(^4^44We include the correlation curve for the other four PCFG variants in Appendix A.3.)
We can observe that LL does have a strong corpus-level correlation with sentence-F1 at the early stage of training despite having a near-non-existent sentence-level correlation.
However, the strength of the corpus-level correlation diminishes quickly, dropping below 0.4 at the late training stage.
This result indicates that LL is able to identify a reasonable PCFG model among a set of poorly performing models in the early training stage, but such ability quickly degrades as the training goes on.
In comparison, SemInfo maintains a strong correlation across the whole training process, which indicates SemInfo’s superior capability in ranking PCFG models by their performance.

In summary, the two correlation analyses provide an empirical explanation for SemInfo’s success in improving PCFG induction.
The analyses demonstrate that SemInfo can serve as an accurate metric for parsing accuracy and that SemInfo can rank PCFG models better than LL.
The strong SemInfo-sentence-F1 correlation also underscores the usefulness of semantic factors in identifying the syntactic constituent structure.

### 5.4 Comparing with State-Of-The-Arts

Table: Table 3: Sentence-F1 on English, Chinese, French, and German test sets. The top section shows the score for SemInfo-trained PCFGs while the bottom section shows the result from previous work. Results from the previous work are taken as-is from the corresponding paper.

Table 3 compares three SemInfo-trained PCFG variants with the state-of-the-art methods for unsupervised constituency parsing.
The SemInfo-trained PCFGs achieved state-of-the-art level parsing accuracy in English, Chinese, and French, outperforming the second-best algorithm by 1.82, 11.02, and 4.47 sentence-F1 scores, respectively.
The SemInfo-trained PCFGs, while using less than half the parameters, perform on par or significantly better than the larger SCPCFG and SNPCFG reported by Liu et al. ().
The comparison showcases the strong parsing accuracy of the SemInfo-trained PCFGs, confirming the usefulness of semantic information in discovering the constituent structure.

## 6 Related works

Parsing with Substring Frequency To the best of our knowledge, Chen et al. () first found that the substring frequency calculated among paraphrases can be used to identify constituent structures.
We extend their findings, establishing a connection between semantic information and the frequency statistic.
We interpret the substring frequency as a term in the substring-semantics information and apply the substring-semantics information to improve PCFG induction.
Our method significantly outperforms theirs in three out of the four languages tested.

Parsing with PCFG
PCFG induction is a long-established (Klein & Manning, ) and state-of-the-art (Liu et al., ) approach for non-ensemble unsupervised constituency parsing.
Much research has dedicated to improving PCFG induction from the model perspective, such as scaling up the PCFG model (Yang et al., ; Liu et al., ), integrating lexical information (Yang et al., ), and allowing PCFG rule probabilities to condition on sentence embeddings through variational inferences (Kim et al., ).
Our improvement is from the model optimization perspective and can be combined with the above efforts.
Our experiments validate the effectiveness of the SemInfo maximization objective in improving unlexicalized PCFGs.
The SemInfo maximization objective is also applicable to lexicalized PCFGs, which we leave to future work.

Parsing with Semantics
Zhao & Titov () and Zhang et al. () have sought to improve PCFG induction by learning to identify visual features, maximizing the association between constituent structures and these visual features.
If we consider the visual features as semantic representations, their approach is effectively maximizing the semantic information of the constituent structure.
In comparison, our method shares the same underlying principle but represents the semantics with textual features.
Our method leverages large language models as semantic processors, utilizing their outstanding semantic processing capabilities (Minaee et al., ).
We believe that combining both textual and visual semantic representations presents a significant research direction for unsupervised parsing tasks.

## 7 Conclusion

In this paper, we proposed and validated a novel objective for unsupervised constituency parsing: maximizing the information between constituent structures and sentence semantics (SemInfo).
We developed a bag-of-substrings model to represent the semantics and applied the probability-weighted information metric to estimate the SemInfo.
We applied the SemInfo maximization objective by maximizing the expected SemInfo for a PCFG-based TreeCRF parser.
Experiments showed that SemInfo has a strong sentence-level correlation with sentence-F1 and that SemInfo maintains a consistent corpus-level correlation throughout the PCFG training process.
The strong sentence-level correlation indicates that SemInfo can serve as an accurate metric of parsing accuracy.
The consistent corpus-level correlation indicates that SemInfo can serve as a reliable training objective for unsupervised parsing.
As a result, SemInfo-trained PCFGs significantly outperformed LL-trained PCFGs across four languages and achieved new state-of-the-art results in three of them.
Our findings underscore the effectiveness of leveraging semantic information in unsupervised constituency parsing, paving the way for accurate semantically-informed unsupervised parsing methods.

## Appendix A Appendix

### A.1 Computing Span-posterior probability via back-propagation

In Section 2, we claimed that the span-posterior probability $P(s\text{ is a constituent}|x)$ can be computed using back-propagation.

$$ $P(s\text{ is a constituent}|x)=\sum_{A\in NT}\frac{\partial\log Z(x)}{\partial \log\beta(s,A)}$ (14) $$

###### Proof.

Firstly, we define the span-posterior probability as Equation 15. Here $s$ is a substring of $x$, spanning from the $i$-th word to the $j$-th word (i.e., $s:=(x_{i},...,x_{j})$). Intuitively, $s$ is a constituent if there exists a non-terminal $A$ that is expanded into $s$.

$$ $P(s\text{ is a constituent}|x)=\frac{\sum_{A\in NT}P(S\rightarrow x\wedge A \rightarrow s_{i,j})}{P(x)}$ (15) $$

We split $P(S\rightarrow x\wedge A\rightarrow s_{i,j})$ into two parts in Equation 16: $P(S\rightarrow x_{1},...,x_{i-1},A,x_{j+1},...)$, the probability of generating words *outside* $s$, and $P(A\rightarrow s)$, the probability generating words *inside* $s$.
The outside probability can be computed using back-propagation (Eisner, ).
The inside probability is already computed by the $\beta$ table.
Exploiting algebraic transformations shown in Equation 18, we can derive the formula shown in Equation 14.

$$ $\displaystyle P(s\text{ is a constituent}|x)$ $\displaystyle=\frac{1}{Z(x)}\sum_{A\in NT}P(S\rightarrow x_{1},...,x_{i-1},A,x _{j+1},...)P(A\rightarrow s)$ (16) $\displaystyle=\frac{1}{Z(x)}\sum_{A\in NT}\frac{\partial Z(x)}{\partial\beta(s ,A)}\beta(s,A)$ (17) $\displaystyle=\frac{1}{Z(x)}\sum_{A\in NT}Z(x)\frac{\partial\log Z(x)}{ \partial\log\beta(s,A)}\frac{1}{\beta(s,A)}\beta(s,A)$ (18) $\displaystyle=\sum_{A\in NT}\frac{\partial\log Z(X)}{\partial\log\beta(s,A)}$ (19) $$

∎

###### Proof.

### A.2 Sentence-level Correlation in different training stages

Figure: Figure 6: Sentence-level Spearman correlations for models trained for 10k steps, 20k steps, and 30k steps.
Refer to caption: extracted/5898961/figures/sentence_level-corr_plot.png

In Table 2, we showed a strong sentence-level correlation between SemInfo and sentence-F1 but a weak correlation between LL and sentence-F1.
Nevertheless, the question of whether the correlation gap is related to the number of training steps remains.
Figure 6 excludes the number of training steps as a factor in the correlation gap.
In this experiment, we calculate the correlation coefficient for models trained for 10k steps, 20k steps, and 30k steps.
We can observe that, for all PCFG variants, the correlation coefficients for (SemInfo, sentence-F1) are consistently over 0.6, while the coefficients for (LL, sentence-F1) are consistently below 0.1.
This result underscores our conclusion that SemInfo can serve as an accurate of parsing accuracy.

### A.3 More detailed analysis for corpus-level correlation

Figure: Figure 7: Corpus-level Spearman correlation in different training stages.
Refer to caption: extracted/5898961/figures/corpus_level-corr_plot.png

Figure 7 shows the corpus-level correlation in different training stages for all five PCFG variants.
We observe the same phenomenon explained in Section 5.3.2 for CPCFG, NPCFG, SNPCFG, and TNPCFG.
The correlation coefficients for (SemInfo, Sentence-F1) are consistently above 0.75, whereas the coefficients for (LL, Sentence-F1) drop quickly as the training progresses.
We can observe the stronger correlation between SemInfo and sentence-F1 in Figure 8.
The figure plots the training curves of the corpus-level sentence-F1 score, the average SemInfo value, and the average LL value over the English validation set.
For example, we can see that SemInfo ranks the NPCFG models represented by the green and grey lines as the lowest, and those represented by the purple and blue lines as the highest.
This largely agrees with the sentence-F1 scores, where the NPCFG models represented by the green and grey lines are among the bottom three worst-performing models, and the models represented by the blue and purple lines are among the top three best-performing models.
In comparison, we see that all models have similar LL scores, which indicates LL’s inability to rank models in accordance with their parsing performance.
These results underscore our conclusion that SemInfo ranks PCFG models better than LL.

In Figure 7, we observe that the correlation strength for (SemInfo, Sentence-F1) also drops as training processes in SCPCFG.
One reason is that SCPCFG failed to explore constituent structures with high SemInfo values.
As shown in Figure 8, the average SemInfo value across the eight models is around 42 for SCPCFG, while the average SemInfo value is greater or equal to 45 for the other four PCFG variants.
This result indicates that the constituent information provided in low SemInfo regions might contain more noise than the information provided in high SemInfo regions.

Figure: Figure 8: Training curves of SemInfo, LL, and Sentence-F1. Each line represents the curve for a single PCFG model.
Refer to caption: extracted/5898961/figures/training_dynamics.png

### A.4 Recall on six most-frequent constituent types

Table: Table 4: Recall on six most frequent constituent types. The recall data is calculated over the English test set. $\Delta$ by Type indicates the average recall improvement for the constituent type. $\Delta$ by Model indicates the average recall improvement for the PCFG variant.

Table 4 shows the recall of the six most frequent constituent types on the English test set, following Yang et al. ().
We see that PCFGs trained with SemInfo achieves significant improvement in Noun Phrases (NP), Verb Phrase (VP), and Subordinate Clauses (SBAR).
These three constituents are the most typical constituents that carry semantic information.
The significant improvement underscores the importance of semantic information in identifying the constituent structure.

### A.5 Paraphrasing prompts

We use the below prompts to generate paraphrases from the gpt-4o-mini-2024-07-18 model. {lang} is a placeholder for langauges. For example, we set {lang}=“English” when collecting English paraphrases.

- •
Create grammatical sentences by shuffling the phrases in the below sentence. The generated sentences must be in {lang}. Use the same word as in the original sentence
- •
Create grammatical sentences by changing the tense in the below sentence. The generated sentences must be in {lang}. Use the same word as in the original sentence.
- •
Create grammatical sentences by restating the below sentences in passive voice. The generated sentences must be in {lang}. Use the same word as in the original sentence.
- •
Create grammatical sentences by restating the below sentences in active voice. The generated sentences must be in {lang}. Use the same word as in the original sentence.
- •
Create grammatical clefting sentences based on the below sentence. The generated sentences must be in {lang}. Use the same word as in the original sentence.
- •
Create pairs of interrogative and its answers based on the below sentence. The generated sentences must be grammatically correct and be explicit. The sentences must be in {lang}. Use the same word as in the original sentence. The answer to the questions should be a substring of the given sentence.
- •
Create pairs of confirmatory questions and its answers based on the below sentence. The generated sentences must be grammatically correct and textually diverse. The sentences must be in {lang}. Use the same word as in the original sentence. The answer to the questions should be a substring of the given sentence.
- •
Create grammatical sentences by performing the topicalization transformation to the below sentence. The sentences must be in {lang}. Use the same word as in the original sentence.
- •
Create grammatical sentences by performing the heavy NP shift transformation to the below sentence. The sentences must be in {lang}. Use the same word as in the original sentence.

### A.6 Examples of the collected paraphrases

The below list contains examples of our collected paraphrases for Such agency ‘ self-help ’ borrowing is unauthorized and expensive , far more expensive than direct Treasury borrowing , said Rep. Fortney Stark -LRB- D. , Calif. -RRB- , the bill ’s chief sponsor ..

- •
’Self-help’ borrowing by such agency is unauthorized and expensive, far more expensive than direct Treasury borrowing,’ said Rep. Fortney Stark -LRB- D., Calif. -RRB-, the bill’s chief sponsor.
- •
Far more expensive than direct Treasury borrowing is such agency ’ self-help ’ borrowing, unauthorized and expensive, said Rep. Fortney Stark -LRB- D., Calif. -RRB-, the bill ’s chief sponsor.
- •
Yes, he said it is far more expensive than direct Treasury borrowing.
- •
What is unauthorized and expensive is such agency ’self-help’ borrowing, far more expensive than direct Treasury borrowing, according to Rep. Fortney Stark.
- •
’Self-help’ borrowing by such agency is considered unauthorized and is regarded as expensive, far more expensive than direct Treasury borrowing,” said Rep. Fortney Stark -LRB- D., Calif. -RRB-, who is the chief sponsor of the bill.
- •
According to Rep. Fortney Stark -LRB- D. , Calif. -RRB- , the bill ’s chief sponsor , such agency ’self-help’ borrowing is unauthorized and far more expensive than direct Treasury borrowing.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Potential typo: Equation 13: t \in P(t | x) -> t ~ P(t | x)
soundness: 4
strengths: (1) Overall, the paper is strong, and the proposed method is a fundamental improvement in unsupervised parsing. In addition to providing strong results, it provides insight into what is missing in log likelihood maximization and how one might augment log likelihood with other objectives.

(2) The evaluation is very thorough, with 5 PCFG variants and 4 languages, and the method produces very large gains.

(3) The paper is well-written: it is well-organized, does a good job of motivating and explaining the method, and provides useful analysis.
summary: Unsupervised parsing methods typically involve maximizing sentence log likelihood, which does not correlate well with parsing accuracy. In contrast, this paper uses a paraphrase model to define SemInfo, an alternate objective that involves maximizing a mutual information metric between tree spans and sentence paraphrases. This alternate objective can then be used as a replacement for log likelihood when doing PCFG induction. Experiments show that this objective produces substantially better F1 scores than log likelihood maximization across a wide range of PCFG variants and languages.
weaknesses: No major weaknesses; some minor weaknesses are discussed below. I support the acceptance of this paper regardless of whether the additional experiments mentioned below are run or not.

(1) From what I can tell, the paper builds upon Chen et al. (2024), which is a simpler instantiation of the idea that spans are likely to be constituents if they tend to be regenerated by paraphrase models. While this is acknowledged in the related works section, perhaps it is also worth acknowledging the influence of their paper in Sections 2 and/or 3 (unless this paper was concurrent).

(2) I would be curious about ablations for each of the additional components of the method (naive substrings -> maximal substrings, the average baseline and entropy regularization in REINFORCE, the addition of the LL term in SemInfo), for both SemInfo as a training objective and when used for MaxTreeDecoding.

(3) While the paper is motivated as an alternate objective for unsupervised parsing, it instead sort of feels like a method for distilling the grammatical knowledge of a large model (in this case, gpt-4o-mini) into a PCFG. From that perspective, the experiments are not that surprising given that we know that large models have mastery of grammar. The promise of the motivation would be better met if the method used a weaker paraphrase method (e.g., something hand-coded) that does not require already "knowing" the grammar of the language.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Please address the following questions during the rebuttal:

- Does the proposed model produce different parse trees over multiple runs?  Could the uncertainty impact the performance of the model?
- Could the method use similar sentences (e.g. nearest neighbours) instead of synthetic data? or other sources of data (e.g. translations, multi-lingual). Could this change have a strong effect on performance? 
- Please speculate on the combination of the proposed semantic metric with contextual embeddings for taking into account paraphrasing. For example: https://aclanthology.org/P15-1030.pdf
soundness: 3
strengths: - Clear description of background knowledge and related work needed to understand the proposed method.  
- The authors perform a  comprehensive comparison of the proposed metric with different models and languages.
summary: The authors propose a novel method for unsupervised constituency parsing. The method is based on semantic information from text statistics (e.g. tf-idf, and bag-of-words). The main contributions are: i) novel metric to measure the information between syntactic structures and semantic, and ii) grammar induction model based on the semantic metric as a learning objective. The method shows a high correlation between the proposed metric and parsing performance, and the proposed model shows competitive performance compared to the state-of-the-art.
weaknesses: - Dependence of the model on synthetic data (e.g. gpt) for paraphrasing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: What is the tree probability (line 152) proportional to the sum of potentials? Normally, it is proportional to the product (or exponentiated sum) of potentials.

Can you say a bit more about how the Tree CRF is parameterized? What are the features and grammar?

How stable are the results of changes in the paraphrasing model? 

At the top of page 6, you suggest that adding log Z term to the objective stabilizes during *early* training; you could better support that claim if you dropped the log Z term after pre-training. Have you considered trying this?

Please explain in section 3 that the paraphrasing model is taken as a given.  I was confused about that until I got to the experiments section (page 6).
soundness: 3
strengths: This paper presents a new approach to learning an unsupervised constituency parser.  The approach works by a novel distillation procedure that leverages an LLM-based paraphrasing model.  It is a very clever idea!  The connection between paraphrasing and syntax is nicely motivated by linguistic theory.  Their approach combines a nice mix of old (keyword extraction, conditional random fields) and new (LLMs).  They provide strong empirical results in five languages over state-of-the-art methods.
summary: This paper presents a new approach to learning an unsupervised constituency parser.  The approach works by a novel distillation procedure that leverages an LLM-based paraphrasing model.  The connection between paraphrasing and syntax is nicely motivated by linguistic theory.  Their approach combines a nice mix of old (keyword extraction, conditional random fields) and new (LLMs).  They provide strong empirical results in five languages over state-of-the-art methods.
weaknesses: I found the mathematical exposition imprecise and hard to follow. This paper would benefit significantly from iteration with colleagues to identify and address these issues.  This is my main hesitation about this paper and why I gave it an overall score of 6 instead of 8.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 4
questions: * All the questions in the 3rd point in weaknesses.
* Why is the IDF term included in the SemInfo definition? Is there a specific reason for incorporating this term?
* In Section 5.3.1, why was 30k steps chosen as the evaluation point? Is there a particular rationale behind this choice?
soundness: 3
strengths: * Demonstrates the effectiveness of utilizing semantic information in parsing.
* Proposes a novel approach for measuring semantic information through paraphrasing.
* Exhibits significant performance improvements across various setups (different PCFG variants as base models and multiple languages).
* Includes experiments in four languages: English, French, Chinese, and German.
* Focuses on model optimization, making it complementary to other efforts in the field.
* The paper is well-written and easy to follow.
summary: This work investigates enhancing the performance of Probabilistic Context-Free Grammar (PCFG) parsers in the Unsupervised Constituency Parsing task by incorporating semantic information into the training objective. To achieve this, the authors use the `gpt-4o-mini-2024-07-18` model as an external paraphrasing tool to generate paraphrases for input sentences, assigning semantic scores to individual phrases based on these paraphrases. The parser is then trained with the Reinforce algorithm to maximize the expected total semantic scores of the induced constituents.

This work demonstrates non-ensemble non-distilled state-of-the-art (SOTA) unsupervised parsing performance for English, Chinese, and French, along with substantial performance on German. The authors also conduct additional experiments to show strong correlations between the proposed objective and parsing performance, motivating further investigation of this objective function for non-PCFG models in future work.
weaknesses: **I would be happy and am strongly willing to increase my rating if most of the following issues, especially the last one, are fixed.**

* **[Minor Weakness]** The authors frequently use “sentence-F1,” “corpus-level sentence-F1,” and “mean sentence-F1” interchangeably, which may lead to confusion. If I understand correctly, each of these terms refers to the “corpus-level sentence-F1” defined on line 299: “The corpus-level score is computed by averaging the sentence-F1 score across the entire corpus.” In prior research, this metric is commonly referred to as the sentence-level F1 score.

    The commonly accepted distinction, as stated in Footnote 8 of [Kim et al. (2019)](https://aclanthology.org/P19-1228/), is as follows: Corpus-level F1 calculates precision and recall at the corpus level to derive the F1 score, whereas sentence-level F1 calculates F1 for each sentence and then averages across the corpus.

    **Suggestions:** Although this paper defines its terminology, it is recommended to (1) avoid using multiple terms for the same concept, and (2) adopt standard terminology to prevent confusion. If the intent is to differentiate between sentence-level F1 scores used in correlation analysis and the average score used in main experiments, a preferable approach might be to use “sentence-level F1” and “mean sentence-level F1.”

* The paper claims to be “unsupervised” and compares its results with other “unsupervised” models. However, it relies on an external paraphrasing model that may involve either supervised or unsupervised methods of paraphrasing supervision. If this paraphrasing model is supervised, then the entire approach would more accurately be considered an “indirectly” or “distantly supervised” method, which is not directly comparable to fully unsupervised baselines. Specifically, the paraphrasing model used here, `gpt-4o-mini-2024-07-18`, incorporates reinforcement learning from human feedback (RLHF; [OpenAI source](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)), which makes it a supervised model due to its reliance on human feedback/labels.

    Additionally, one key motivation for claiming an “unsupervised” approach is applicability to low-resource languages or other structure-prediction tasks where GPT models may not be available. This limitation should be more clearly addressed in the paper.

    **Suggestions:** To support the “unsupervised” claim, I suggest conducting experiments with an unsupervised paraphrasing model. A comparable approach is found in [He et al. (2018)](https://arxiv.org/pdf/1808.09111), where their dependency parser model, which requires POS labels, reports two sets of results: one using gold POS tags and the other using induced unsupervised POS tags. This dual-setup approach acknowledges the compounded error introduced by unsupervised POS tagging, providing a clearer comparison.

* Regarding the paraphrasing model, several important questions and concerns should be addressed:
    1. Quality Requirements: What level of performance is required from the paraphrasing model to effectively support the proposed method? Is there a minimum paraphrasing accuracy or quality that significantly impacts the results, and how is this threshold determined?
    2. Robustness to Model Choice: How robust is the proposed model to variations in the paraphrasing model used? For example, would different paraphrasing models (with varying architectures, training datasets, or language capabilities) yield comparable outcomes, or are the results highly sensitive to the choice of the paraphrasing model? This could be demonstrated through experiments with multiple paraphrasing models, both high- and low-performing, to understand the potential variability in results.
    3. Efficiency in Training: How efficient is the training process, given that it requires multiple calls to the paraphrasing model? Discussing any optimizations or alternatives could strengthen the method’s practicality.

    Each of the above points warrants at least some discussion in the paper.

* There is no need for probability notation in Equation (7); the `P` notation can be safely replaced with an indicator function. It is recommended to avoid probability notation here, as `s ∈ MS(x^p, x)` is not a random variable, given that `x^p` serves as the expectation argument.
* In Table 1, the statistical test is conducted using only three samples, which affects its reliability. Since training multiple model instances is resource-intensive, I suggest performing the test at the sentence level. Specifically, consider each sentence’s F1 score as a separate sample, following [Shayegh et al. (2024)](https://aclanthology.org/2024.acl-long.808.pdf).
* The final argument in Section 5.2, which discusses the performance on the German dataset, could benefit from further elaboration. Currently, the claim lacks sufficient support.
* **[Minor Weakness]** In Appendix A.2, the correlation is reported at 10k, 20k, and 30k steps. However, Figure 5 shows that LL-SF1 diminishes significantly after 10k steps, while achieving better correlation at earlier stages. Although this does not affect the conclusion that SemInfo-SF1 maintains a higher correlation throughout the entire training process, it would be beneficial to include results for earlier steps in Appendix A.2 for completeness.
* **[Minor Weakness]** Figure 4 appears to depict a worst-case scenario for LL-SF1 correlation, showing a large negative value. However, Table 2 shows that the correlation is nearly zero for all models and slightly positive on average, which may make Figure 4’s presentation somewhat misleading.
* The paper introduces PCFG models as the state-of-the-art (SOTA) in the task five times, yet only twice (and neither in the Approach nor Introduction sections) specifies that this refers to non-ensemble SOTA. Additionally, the authors claim to achieve new SOTA results four times without clarifying that this pertains only to non-ensemble models.
    1. Transparency of Setup: This approach somewhat obscures the actual setup. It should be clearly stated as non-ensemble, and more importantly, **non-distilled** SOTA, especially as works like [Kim et al. (2019)](https://aclanthology.org/P19-1228/), [Cao et al. (2020)](https://aclanthology.org/2020.emnlp-main.389.pdf), and [Shayegh et al. (2024b)](https://arxiv.org/pdf/2310.01717) demonstrate significant performance improvements by distilling induced knowledge into a URNNG model ([Kim et al.; 2019b](https://arxiv.org/abs/1904.03746)), all of which outperform the best results in this paper. The first two works achieve this by distilling the best model (which requires a validation set), and the latter by distilling ensemble knowledge (non-best, thus without a validation set).
    2. Comparison Justification: It is unclear why the authors chose to compare only with non-ensemble and non-distilled models. This choice is not discussed in the paper. Since distilled models have demonstrated efficiency during inference (Table 5 in [Shayegh et al.; 2024b](https://arxiv.org/pdf/2310.01717)), inference efficiency cannot be the reason. If training efficiency is the concern, then an efficiency analysis on the paraphrasing models is warranted.
    3. Discussion of Related Work: Even if the authors choose not to compare directly with the mentioned works, these studies are significant enough to merit discussion.

    In general, there is an overemphasis on claiming SOTA. This is unnecessary, as the paper’s novelty and valuable contributions stand on their own and are beneficial to the field. Notably, [Shayegh et al. (2024b)](https://arxiv.org/pdf/2310.01717) show that each individual model with specialized expertise can contribute substantially to final parsing performance when part of an ensemble. Specifically, they take advantage from [Li & Lu (2024)](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/2306.00645&hl=en&sa=T&oi=gsr-r&ct=res&cd=0&d=15034760626359817941&ei=bG4mZ7CxJuG86rQPgN2emQk&scisig=AFWwaeZgwSDS3qGcVtN0rBG9whiK) despite their lower standalone performance, because of its unique approach to Unsupervised Constituency Parsing.

    **Suggestions:**
    * Instead of heavily emphasizing standalone SOTA status, consider including a correlation analysis with other baselines to demonstrate the unique insights offered by the new model, particularly due to the proposed new objective.
    * Incorporating the model into the ensemble approach from [Shayegh et al. (2024b)](https://arxiv.org/pdf/2310.01717) and measuring any resulting performance boost could also provide strong support for the model’s contribution.

Meta-review: This paper presents a novel method for learning constituency parsers in an unsupervised fashion.  The reviewers are overwhelmingly supportive of acceptance as the consensus is that the paper presents a substantial improvement in this area beyond what exists in the literature.  There are no explicit large weaknesses given the discussion.

Decision: Accept (Spotlight)