Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction (2024)

Abstract: Neural parameterization has significantly advanced unsupervised grammar induction. However, training these models with a traditional likelihood loss for all possible parses exacerbates two issues: 1) structural optimization ambiguity that arbitrarily selects one among structurally ambiguous optimal grammars despite the specific preference of gold parses, and 2) structural simplicity bias that leads a model to underutilize rules to compose parse trees. These challenges subject unsupervised neural grammar induction (UNGI) to inevitable prediction errors, high variance, and the necessity for extensive grammars to achieve accurate predictions. This paper tackles these issues, offering a comprehensive analysis of their origins. As a solution, we introduce sentencewise parse-focusing to reduce the parse pool per sentence for loss evaluation, using the structural bias from pre-trained parsers on the same dataset. In unsupervised parsing benchmark tests, our method significantly improves performance while effectively reducing variance and bias toward overly simplistic parses. Our research promotes learning more compact, accurate, and consistent explicit grammars, facilitating better interpretability.

heading: 1 Introduction
text: Grammar induction (GI) has been intensively studied due to its critical role in understanding, controlling, and utilizing the structural information of internal symbols called constituents in linguistic analysis. Unsupervised approaches to GI are continuously studied for their benefits of avoiding structured data as parses, which requires enormous human efforts and expertise to generate supervised data. In the literature of this field, neural parameterization for representing grammars (Kim et al., 2019a) has shown significant improvement over traditional probabilistic models (Klein and Manning,
*Corresponding author.
2002, 2001; Clark, 2001). Furthermore, advanced models that incorporate conventional approaches, such as lexicalization (Zhu et al., 2020; Yang et al., 2021a) and symbol expansion (Yang et al., 2021b, 2022) have been proposed.
However, these Unsupervised Neural Grammar Inductions (UNGIs) lack analyses of the traditional challenges, particularly regarding the training loss. Some studies, such as Kim et al. (2019a), have highlighted the difficulty of optimization presented by the complex loss landscape in UGIs (Lari and Young, 1990; Carroll and Charniak, 1992; Klein and Manning, 2001). Subsequent efforts (Yang et al., 2021b, 2022) have alleviated these issues by only enhancing expressiveness through overparameterization rather than the cause analysis.
In this paper, we aim to reveal two unhandled problems in UNGIs: 1) structural optimization ambiguity, which increases the number of optima of the same quality that prefer different parse structures, leading to prediction errors by random selection for branching and high variance, and 2) structural simplicity bias, which induces the grammar to use only a few rules in parse trees and resulting in simplistic parse trees even with a large capacity. Both problems are caused by training a neural grammar with sentence probability.
To address these issues, we propose a simple but effective method to inject focusing-bias that accounts for only a few selected parses for training loss in the inside algorithm. This approach reduces the ambiguity by selecting more distinct parses across sentences and restricts the simplicity bias by excluding of unnecessary simplistic parse trees by sentences. We also introduce a method of generating structural bias by parse selection from single or multiple parsers pre-trained on the same training data.
Our method outperforms state-of-the-art implicit and explicit grammar induction models in English parsing tasks on the Penn Treebank (PTB) (Marcus
15124
et al., 1993), as well as in ten other languages, including the Chinese Penn Treebank (CTB) (Xue et al., 2005) and the SPMRL datasets (Seddah et al., 2014). In-depth analysis shows a reduction in high variance and overly simplistic parses. We provide the investigation results of various structural biases, recommending heterogeneous multi-parsers.
Our contributions are summarized as follows:
• We raise and clarify structural optimization ambiguity and structural simplicity bias, which lead to high variance and overly simplistic parse problems in UNGIs.
• We propose sentence-wise parse-focusing that employs biases from pre-trained parsers using the same training data. This approach enables the stable learning of more compact and accurate explicit grammars.
• Through in-depth empirical analysis, we demonstrate the effectiveness of our approach in reducing the identified causes, investigating various focusing-biases, and achieving significant performance improvements compared to state-of-the-art UNGIs.

heading: 2 Background

heading: 2.1 Notations of Probabilistic Context-Free Grammar
text: In the following analysis, we use the notations below for a PCFG G = (S,N, P,Σ, R,Π): the root symbol S, finite sets of nonterminals N , preterminals P , terminal words Σ, production rules R, and production rule probabilities Π. R consists of three types of rules:
S → A where A ∈ N A → α β where α, β ∈ (N ∪ P ) B → ω where B ∈ P and ω ∈ Σ
Note that the left-hand side (parent) of the rules is distinguished between nonterminal and preterminal categories.

heading: 2.2 Negative Log Likelihood Loss
text: The loss of grammar induction is well-known to increase the probability mass of only observed parse trees (Chi and Geman, 1998) and to maximize rule probability for representing correct parses of sentences (Corazza and Satta, 2006). The loss of UGI is the same as SGI in that they both maximize sentence probability for a given sentence; however, it
is different in that UGI uses all possible parse trees to calculate sentence probability.
The likelihood loss for a given sentence is:
L(s) = − log ∑
τ∈T(s) exp
∑
r∈R(τ) f(r|τ) log p(r|G)
where the exponential term is the probability of a parse tree τ , and f(r|τ) denotes the frequency of rules used in the tree. In the case of SGI, T(s) only contains the gold parse tree for s. However, for UGI, T(s) contains all parse trees that can generate s.
In the equation 2.2, the loss reaches its minimum when the exponential term within the summation, which is always negative or equal to zero, approaches zero. This can be rewritten as ∑ f(A → α) logP (A|α) ≃ 0.

heading: 3 Limits on Unsupervised Learning of Neural Grammar Induction
text: As explained earlier, UNGI Loss has the advantage of inducing grammar without the information of gold parse trees. However, UNGIs still inherit the limitations of probabilistic UGIs. In this section, we reveal two issues arising from these limitations in UNGIs:structural optimization ambiguity and structural simplicity bias, and discuss derived negative effects and their causes. For precise analysis, we focus on inducing explicit neural grammars based on PCFGs, mainly Neural PCFGs (NPCFGs).

heading: 3.1 Structural Optimization Ambiguity (SOA)
text: Structural optimization ambiguity causes a huge variance of performance by arbitrary branching bias that depends on random seeds. In this section, we clarify the definition of SOA, and explain the arbitrary convergence caused by SOA. After that, we mathematically prove the existence of SOA under the single pre-terminal condition and extend this to the multiple pre-terminal condition using empirical evidences.
Definition of Structure Optimization Ambiguity by UGI Loss SOA is the ambiguity that disrupts the distinction between two optima that predict different parse trees for the same sentence. For clear analysis, we define the ambiguity as follows.
Definition 1. Given G1 = (S,N, P,Σ, R,Π1) and G2 = (S,N, P,Σ, R,Π2) that assign different rule probabilities from the same G, the grammar G is
structural-optimization-ambiguous if and only if p(s|G1) = p(s|G2) for all sentences s ∈ S in the training data, and there exists a sentence s whose best parses respectively derived through G1 and G2 are different.
As mentioned in Section 2.2, the sentence probability is the loss function of GI. This means that if two grammars have the same sentence probability, they are not distinguished in the optimization landscape. In other words, they are ambiguous in the perspective of optimization.
Arbitrary Convergence by Structural Optimization Ambiguity The discrepancy between structure and loss function leads to that each grammar randomly initialized favors different optimal points. In Figure 1, highlighted two optimal points are equally probable yet structurally divergent. At the blue optimum, the parse tree t1 has a higher probability than the parse tree t2, whereas, at the red optimum, the situation is reversed, with t2 having a higher probability than t1. Consequently, it becomes unclear which optimum will be chosen based solely on sentence probabilities from an optimization perspective. Moreover, it is intuitively understood that the reachable optimum may vary with the randomness, such as initial states.
Proof in Single Pre-terminal Condition We prove that SOA always exists under the single preterminal condition by using equivalence of sen-
tence probability between two grammars that have different probability distributions.
In the first step, we consider constraints on the grammars: 1) both grammars share the same configuration (S,N, P,Σ, R), and 2) the rule probabilities of both grammars are identical except for a single type of flipped rule pair (Ni → NjT and Ni → TNj). Under these constraints, suppose that the two grammars yield the same sentence probability for a given sentence; we can derive the following equation in general condition from the formula of the inside algorithm:
pG(Ni → NjT ) + pG(Ni → TNj)α = pG′(Ni → NjT ) + pG′(Ni → TNj)α, (1)
Where,
α = pG1(T → w1)IG1,j(2, r)
IG1,j(1, r − 1)pG1(T → wr)
Then, we also consider that each grammar includes only one pre-terminal1. Under the aforementioned constraints, α = 1 is always satisfied; therefore, the above equation 1 can be simplified as follows:
pG(Ni → NjT ) + pG(Ni → TNj) = pG′(Ni → NjT ) + pG′(Ni → TNj)
(2)
Under the identical probability distribution except for the flipped rule pair, the sum of the probabilities for the flipped rule pair is always equal in a proper grammar (Nederhof and Satta, 2006). Thus, equation 2 is always satisfied, and the two grammars always yield the same probability for the same sentence. Furthermore, due to the strict inequality of the flipped rule pair, the two grammars can derive different parse trees. This signifies that the two grammars are structural-optimizationambiguous. Appendix A.1 provides a more detailed proof of this.
Empirical Evidence in General Condition: Low Correlation of Loss to S-F1 Because proving the existence is significantly complex under general condition using multiple pre-terminals, we prove it indirectly by presenting empirical evidences. The
1The single pre-terminal condition is also established when 1) rule probabilities are uniformly initialized or 2) all tokens in sentences cannot be distinguished (e.g. all tokens are <unk>).
0.45 0.50 0.55 0.60 0.65
F1
110.0
110.5
111.0
111.5
112.0
112.5
113.0
113.5
114.0
N eg
at iv
e Lo
g Li
ke lih
oo d
r=−0.17 P=0.35
0
5
0 5
(a) Correlation of S-F1 score and likelihood
0.45 0.50 0.55 0.60 0.65
F1
110
112
114
116
118
120
N eg
at iv
e Lo
g Li
ke lih
oo d
(N LL
)
NT=4500 NT=250 NT=30
(b) Amplification for the variance of S-F1 by symbol size
Figure 2: (a) Low correlation between S-F1 score and UGI likelihood for PTB test set. We evaluate trained FGG-TNPCFGs with 4500 nonterminal symbols and 9000 preterminal symbols using 32 random seeds over 10 epochs. (b) Amplification of variance by symbol size increasing. Circles are the same as in (a) and others are trained with the same settings as in (a) except for the number of symbols.
first evidence is a low correlation between sentencelevel F1 (S-F1) scores2 and negative log likelihood loss, which implies that the optimization of grammars is inconsistent, and can lead to generating structures different from those desired by gold parses.
In Figure 2a, the negative log likelihood and SF1 scores are shown to be almost uncorrelated, as indicated by a low P-value and Pearson coefficient. This low correlation means that the distinction between the grammars that induce different structures is hard to determine using negative log likelihood, which causes the large variance of S-F1 scores due to randomness. Through this analysis, we demonstrate the existence of SOA under general conditions.
Empirical Evidence in General Condition: Variance Amplification by Symbol Size Another piece of evidence is the variance amplification of S-F1 scores with the increase in the number of symbols, as shown in Figure 2b. It appears that more nonterminals create more local optima of similar quality that derive different parses, leading to a greater variety of parses compared to a single gold parse. This demonstrates the variance amplification of SOA as the number of symbols increases under general conditions.
2The S-F1 score is one of the common metrics to evaluate parsing quality. It evaluates the similarity between predicted parses and gold parses. In this paper, we only use unlabeled S-F1 scores that do not consider the symbol labels the same as N-PCFG.

heading: 3.2 Structural Simplicity Bias (SSB)
text: The second problem, structural simplicity bias, causes a decrease in expressive power for parse trees by leading the grammar to have low structural diversity. In this section, we discuss what SSB is, the problems it causes, and why it occurs. Lastly, we present empirical evidence for how SSB is revealed during training.
What is SSB? SSB is the bias that induces a grammar to utilize as few rules as possible to compose parse trees. This bias is a natural and intended effect of the objective to maximize sentence probability. However, it leads to learning undesirable structures that are different from gold parses in UNGI.
Specific Conditions of UGI UGI is affected by different biases, unlike SGI, due to two properties: 1) the loss maximizes the probabilities of all possible trees for a given sentence. 2) all rules can be used in any position that has the same type3. Therefore, UGI prefers the structure that maximizes tree probability rather than optimizing the given parses as SGI does.

heading: Rule Simplification for Expressing a Parse If
text: all rules in the tree are the same, it is intuitively easy to maximize tree probability because the model only needs to optimize one rule. Therefore, maximization using tree probability prefers to train utilizing a small number of rules. This preference leads to a grammar that includes simple structures in parses. This means that the grammar has low expressive power for parse trees. Specifically, only rules of same type can share their position in the tree due to symbol types; therefore, parses prefer structures that utilize structural repetition. For these reasons, it prefers left- or right-binarized structures with many repetitions.
Space Inefficiency and Performance Degradation by the Potential Expressive Limit The grammar affected by SSB utilizes only a small part of its potential expressive power for parse trees, which causes two problems: 1) space inefficiency, as expressive power grows slowly compared to an increase in grammar size, and 2) low expressive power compared to gold parses, leading to decreased performance. The low rule diversity of
3The type of rules means the four types, N → NN |NP |PN |PP , distinguished by the type of symbols: nonterminal, preterminal.
5 10 15 20 25 30 35 40 Sentence Length
0
5
10
15
20
25
30
N um
be r o
f R ul
e Ty
pe s
Binarized gold FGG-NT30 FGG-NT15 FGG-NT5
(a) Comparison between the gold parse trees and FGGTNPCFGs.
5 10 15 20 25 30 35 40 Sentence Length
0
5
10
15
20
25
30
N um
be r o
f R ul
e Ty
pe s
FGG-NT30 FGG-NT15 FGG-NT5 PF-NT30 PF-NT15 PF-NT5
(b) Comparison between FGG-TNPCFGs and parsefocused N-PCFGs.
Figure 3: The average number of unique rules in each parse for each sentence length. We evaluate the average for models with 5 / 15 / 30 nonterminals trained with four different random seeds. These evaluations use the WSJ train set. (a) The gold uses non-binarized gold parses. (b) Parse-focused N-PCFGs use models trained by Structformer, NBL-PCFG, and FGG-TNPCFG.
grammar compared to the necessity of gold parses cannot represent the necessary tree structures for a given sentence. Induced parses that are not sufficiently represented cause low performance.
Empirical Evidence: Average Diversity in Each Parse Tree We demonstrate that existing models induce simplified grammars by analyzing the predicted parses. We provide evidence that SSB leads a model to learn simplistic diversity compared to gold parses, as shown in Figure 3a, which simplifies the induced tree structures and results in similar structures among trees of the same length. In other words, this can damage diversity. This decrease in diversity appears to lead to a decrease in performance.
In Figure 3b, the grammars of the same size with parse-focusing have greater rule diversity than FGG-TNPCFGs. This shows that 1) parse-focusing relaxes the SSB, and 2) the decrease in rule diversity is not caused by the upper bound for grammar capacity. We verify the loss of rule diversity in other languages in Appendix A.2 and come to the same conclusion. Additionally, we provide the qualitative analysis of overly simplistic parses in Appendix A.3.

heading: 4 Method

heading: 4.1 Preliminaries
text: Motivation SOA and SSB have not yet been reported as critical problems, even though they can also exist in conventional probabilistic GIs (Klein and Manning, 2001). The reason is that conven-
tional methods can avoid equal frequency assignment to all parses, which is the basic condition that causes these problems. For example, the EM algorithm in UGIs reassigns different rule frequencies at every iteration (Petrov et al., 2006), and SGIs use only gold parse trees while implicitly assigning zero probability to all incorrect parses. Motivated by this, we propose sentence-wise parse-focusing to learn from a few parses for each sentence and structural bias generation to select the parses from unsupervised multi-parsers pre-trained on the same training data.
Overview of Parse-Focusing Figure 4 presents an example application of sentence-wise parsefocusing to N-PCFGs. Compared to N-PCFG, the distinguishing feature is selecting a few parses from unsupervised parsers pre-trained on the same training data and deriving the training loss from these parses. Details of the algorithm are provided in Appendix A.4.

heading: 4.2 Sentence-wise Parse-Focusing
text: Learning from Focused Parses To learn grammar from a few selected parses, we simply modify the inside algorithm to count constituent frequency only for observed parses. However, this strict bias injection completely excludes unselected parses, resulting in losing potential structures. To alleviate this limitation, we use soft weights derived from softmax over all constituent counts, which maintains the focus on the selected parses and also leaves small weights on potential structures. The process of applying the parse-focusing is detailed in Algorithm 1.
Given a finite set of sentences s ∈ S, we represent a finite set of selected parse trees for each sentence s as τs ∈ Ts. We denote the frequency
for span (p, q) of word p to q for parse tree τs as f((p, q)|τs). Then, the soft weights are
w(p, q|s) = exp ∑ τs∈Ts f((p, q)|τs)∑
p,q exp ∑ τs∈Ts f((p, q)|τs)
Why Does It Alleviate High Variance and Overly Simplistic Parse Issues? As mentioned in Section 4.1, the base condition for causing SOA and SSB is that all parse trees are considered equally.
This equal consideration establishes conflicting pairs of flipped rules with equal frequency, which causes SOA. Sentence-wise parse-focusing avoids this equal frequency by using soft weights for spans. Additionally, sentence-wise parse-focusing has no specific dynamics to consider them equally. These reasons significantly decrease the possibility of establishing Equation 1 under general condition, which avoids the problems caused by SOA in training.
This consideration also makes a model maximize advantageous simplistic parse trees that are not related to gold parse trees, which causes SSB. These simplistic parse tree structures are shared across sentences. Sentence-wise parse-focusing avoids the maximization of the simplistic parse trees by assigning selected parse trees for given sentences. The selected parse trees may not involve the simplistic parse trees and are also different across sentences.

heading: 4.3 Focusing-Bias Generation
text: Bias Generation from Pre-Trained Unsupervised Parsers The performance of grammar relies on the quality of selected parses because focused parses control the loss for given sentences. Therefore, we can expect to further improve performance when we use high-quality parses. However, in unsupervised learning, finding parses close to gold parse trees without supervision-related information is a challenging issue. As an alternative solution, we use additional parsers pre-trained on the same data with unsupervised learning. Note that our method allows any type of parser as long as it provides parse span information.
Data-Specific Bias Induction via Heterogeneous Multi-Parsers A potential approach to generate good parses is to use parsers induced by different models, which we call heterogeneous parsers. According to Ishii and Miyao (2023), the branching bias consists of model-specific inherent bias
and data-specific inductive bias. They also show that different parsers have different model-specific inherent biases. Therefore, if we use heterogeneous parsers when counting constituents in parsefocusing, the opposing inherent biases between parsers are offset, while the same structures by data-specific bias are enhanced.
In Table 1, we compare the structural differences between parsers within pairs of heterogeneous parsers and within pairs of homogeneous parsers. Homogeneous parsers are trained using the same model but with different random seeds. To directly quantify structural heterogeneity, we use IoU scores on parse spans defined as:
⋂ Gp∈Gp TGp⋃ Gp∈Gp TGp
The IoU scores of heterogeneous parsers are consistently lower than those of homogeneous parsers, which implies that these parsers have more different branching biases. Therefore, when using heterogeneous parsers, the different model-specific inherent biases are offset, making the shared part more likely to indicate data-specific bias.

heading: 5 Experiments

heading: 5.1 Settings
text: Datasets and Hyperparameters We use the Penn Treebank (PTB) (Marcus et al., 1994), an English constituency parsing task dataset, to evaluate parsing performance. Additionally, we use the CTB (Xue et al., 2005) and SPMRL dataset (Seddah et al., 2014) for multilingual evaluation.
We use the same hyperparameters as Yang et al. (2022); for the analysis of the number of symbols, we follow the ratio of 1:2 between nonterminals and preterminals. The details for the dataset and training are in Appendix A.6
Inference We primarily evaluate grammars with MBR decoding (Yang et al., 2021b). However, we
use CYK decoding when it is necessary to distinguish each rule in the grammar.
Base Model We use the model from Yang et al. (2022), which we refer to as Factor Graph Grammar-based TN-PCFGs (FGG-TNPCFGs), as our base model. Our model is implemented based on the open-sourced code from Yang et al. (2022)4. There are two reasons for choosing this model: 1) The strict assumption of PCFGs is preserved. In the case of Compound PCFGs (C-PCFGs) and Lexicalized PCFGs (Zhu et al., 2020; Yang et al., 2021a), because they are dependent on words or sentences, which makes the induced grammar complex to understand. We avoid this to verify whether better performance can be achieved without relaxed assumptions. 2) FGG-TNPCFGs rapidly train largescale grammars through efficient use of memory and computation. This is favorable for obtaining diverse results in different conditions. Additionally, we note that while FGG-TNPCFGs provide efficient computation, the fundamental mathematical principle for grammar remains the same.
Selected Parsers for Focusing-Bias We adopt three parsers that use different algorithms; 1) Structformer (Shen et al., 2021) is a transformer-based and implicit grammar model. 2) NBL-PCFG (Yang et al., 2021a) is a lexicalized explicit grammar model. 3) FGG-TNPCFG (Yang et al., 2022) is the largest and state-of-the-art grammar model. Note that all of these parsers use the same hyperparameter settings as in their original papers, and they are trained only with PTB. The primary model trained by parse-focusing consists of these three parsers, which shows the best performance. Additionally, we use DIORA (Drozdov et al., 2019), which is a self-supervised model utilizing a recursive autoencoder architecture. However, it is not used as the main parser because it is trained with the additional dataset MultiNLI (Williams et al., 2018b) besides PTB.

heading: 5.2 Performance
text: Performance in English We evaluated the performance of state-of-the-art UNGI methods on English PTB data, as shown in Table 2. Our approach significantly outperforms the state-of-the-art model (FGG-TNPCFGs) with only 30 nonterminals and 60 preterminals. When using 4500 nonterminals, as with FGG-TNPCFGs, the performance further
4https://github.com/sustcsonglin/TN-PCFG
increases significantly. Note the difference between reproduced models and the variance among models. Instead of running only four runs, we performed 32 runs, resulting in lower performance and higher variance, especially in FGG-TNPCFG and NBLPCFG. The significant performance difference between DIORA and DIORA† is due to differing evaluation criteria. While DIORA utilizes a test set of binarized trees by Stanford CoreNLP, DIORA† employs non-binarized trees. This approach is intended to assess performance accurately within the evaluation environment of N-PCFG.
Performance in Multilingual Table 3 shows the performance of explicit grammar-based UNGIs in multilingual parsing on CTB and SPMRL data. The results indicate that our model significantly outperforms other models across most languages. While a few languages, such as Hungarian, Polish, and Swedish, show second or third-best performance, our model ranks the highest overall. Note that our method has consistently and significantly lower variance compared to all other methods, which indicate that it is a more reliable estimator.

heading: 5.3 In-depth Analysis

heading: Performance Consistency about Randomness
text: We investigate the impact of our method on the high variance problem, which is the main limitation of UNGIs. To observe the variance change caused solely by parse-focusing, we also evaluate the variance using the generated sets of parse trees as the focusing-bias: randomly, left-only branched, and right-only branched. As shown in Table 4, all of the variances of S-F1 scores are significantly reduced compared to those of FGG-TNPCFG, while their negative log likelihoods exhibit a similar bias. This means that sentence-wise parse-focusing reduces variance without depending on the selected parses. The results indicate that applying parse-focusing reduces the variance problem and indirectly supports the idea that the cause is due to SOA under uncorrelated S-F1 and NLL. Moreover, they show that focusing-bias from pre-trained parsers can lead to higher performance than algorithmically generated parses.

heading: Performance and Variance by Symbol Size In
text: Section 3.2, we show that parse-focusing relaxes the low diversity of unique rules in conventional methods. Here, we demonstrate how the relaxation
from parse-focusing affects the increase in performance according to changes in symbol size. In Table 5, although our method boosts performance in the larger model, it significantly enhances performance compared to FGG-TNPCFGs in the smaller grammar model. These results imply that our method can stably induce more compact grammars by reducing SSB while achieving more accurate models using the same model capacity.
Rule Utilization We show that parse-focusing reduces the extreme reliance on a small number of rules in FGG-TNPCFG. Figure 5 shows the sorted frequency of all rules in a grammar. After applying our method, the frequency of the most frequent
three rules decreases from 46.1% to 38.7%, and the reduced frequency is redistributed to other less frequently used rules. These results imply the effective enhancement of rule utilization.
Practical Multi-Parser Combinations We show the significance of using heterogeneous multiparsers that have different model-specific inherent biases. Figure 6 shows that the performance increase in heterogeneous parsers is generally more considerable compared to homogeneous ones. These parsers exhibit more significant variances in performance, which supports the idea that the inherent biases of models are offset. Consequently, the results implies that heterogeneity is a practically helpful strategy.

heading: 6 Related Works
text: TN-PCFG/FGGs Yang et al. (2021b) and Yang et al. (2022) verify that overparameterization leads to better grammars with many symbols. Conversely, we highlight the inefficiency in learning UGIs caused by simplicity and ambiguity issues. Solving this inefficiency enables learning even a small grammar with significantly high performance and low variance.
S-DIORA Drozdov et al. (2020) shows that structural ambiguity between subtrees can affect locally greedy training, and that this problem can be resolved by single tree encoding through beam search. This work supports the impact of the ambiguity issue, although our research addresses the issue of learning explicit grammars in UGIs differently.
Ensemble Distillation Shayegh et al. (2023) introduce a variant of the CYK algorithm that gains
refined parse trees by combining trees from the different pre-trained parsers. Our approach differs in searching for structure through training trees in a relaxed condition using soft weights from the count of spans. It utilizes the model’s search capability. In addition, it is different that we resolve the optimization problem for the loss function.
Tree-shape Uncertainty The tree-shape uncertainty referred to in Ishii and Miyao (2023) is the existence of similar grammars that assign different tree structures for a given sentence. In our work, the issue is analyzed from an optimization perspective, highlighting how undifferentiated optima can cause unstable learning.

heading: What Do Implicit Grammar Models Learn?
text: Williams et al. (2018a) investigate how consistently latent tree learning models learn in the presence of randomness. In our work, we focus on identifying the causes of inconsistency and its negative effects, which are primarily due to likelihood loss over all parses in UGI.

heading: 7 Conclusion
text: In this paper, we tackle the challenges of high variance and overly simplistic parse in unsupervised neural grammar induction (UNGI). We identify structural optimization-ambiguity and structural simplicity bias as primary causes, arising from UGI loss on all possible parses of all sentence parses. We introduce a parse-focusing approach that narrows down the parse set per sentence and incorporates biases from parses generated by pre-trained parsers on the same dataset. This method has been shown to effectively mitigate these issues, significantly surpassing state-of-the-art methods in both English and multilingual parsing benchmarks. Our contributions facilitate the consistent learning of more compact, explicit, accurate, and therefore interpretable grammar models in unsupervised learning environment.

heading: Acknowledgements
text: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2022R1A2C2012054, Development of AI for Canonicalized Expression of Trained Hypotheses by Resolving Ambiguity in Various Relation Levels of Representation Learning).

heading: Limitations
text: Our work is limited to offline learning due to additional computational costs for pre-training parsers and for the hyper-parameter search of better combinations. In the future, more light and adaptive structural bias generation is required.

heading: A Appendices

heading: A.1 Detail Proof of Structural Optimization Ambiguity with the Inside Algorithm
text: We incorporate the three stipulated conditions into the conventional inside algorithm as per Equation 3: 1) distinguishing between nonterminals and preterminals, 2) allowing all symbols to occupy identical positions in parse trees, and 3) encompassing all structurally possible binary trees for given sentences. Therefore, we can reformulate the inside algorithm as shown in Equation 4.
We examine a special case where a grammar contains only one preterminal symbol, as detailed in Equation 5, to investigate the potential for structural optimization ambiguity. We recursively derive the conditions leading to ambiguity.
Before we begin the proof, let us consider two grammars, G1 and G2. The relationship between them is defined as follows:
pG1(Ni → NjT ) ̸= pG2(Ni → NjT ) pG1(Ni → TNj) ̸= pG2(Ni → TNj)
∀r ∈ Rc, pG1(r) = pG2(r)
Here, Ni, Nj represent nonterminal symbols, T denotes a preterminal symbol, pG1(Ni → NjT ) represents the probability of the rule Ni → NjT assigned by G1, and Rc is the complement set of rules that do not involve Ni → NjT and Ni → TNj .
Initially, we can directly determine the probability of a span with width w = 2:
IG,i(p, p+ 1) = pG(Ni → TT )pG(T → wp)pG(T → wp+1) (9)
Conventional inside algorithm
Ii(1, l) =
n∑
i
n∑
j
n∑
k
p(Ni → NjNk)Ij(1, p)Ik(p+ 1, l) (3)
Inside algorithm in Unsupervised Neural Grammar Induction
Ii(1, l) =
n∑
j
m∑
k
p(Ni → NjTk)Ij(1, l − 1)p(Tk → wl) + p(Ni → TkNj)p(Tk → w1)Ij(2, l)
+ n∑
j
n∑
k
n−2∑
a=2
p(Ni → NjNk)Ij(1, a)Ik(a+ 1, l) (4)
Inside algorithm in UNGI with one preterminal symbol
Ii(1, l) =
n∑
j
p(Ni → NjT )Ij(1, l − 1)p(T → wl) + p(Ni → TNj)p(T → w1)Ij(2, l)
+ n∑
j
n∑
k
n−2∑
a=2
p(Ni → NjNk)Ij(1, a)Ik(a+ 1, l) (5)
Inside probability for span width 3
IG,i(1, l) =
n∑
j
pG(Ni → NjT )IG,j(1, l − 1)pG(T → wl) + pG(Ni → TNj)pG(T → w1)IG,j(2, l) (6)
Equivalence between two different grammars
For both grammars G1 and G2, the equality IG1,i(p, p+1) = IG2,i(p, p+1) consistently holds for all p because the probability of a span with width w = 2 does not involve the rules Ni → NjT and Ni → TNj .
For spans of width w = 3, we refer to Equation 6. We set the inside probabilities for grammars G1 and G2 to be equal, such that IG1,i(p, p+ 2) = IG2,i(p, p+ 2). Noting that only the probabilities of Ni → NjT and Ni → TNj differ, we see that the other terms cancel out, as shown in Equation 7. Therefore, Equation 7 can be simplified to Equation 8, given that IG1,i(p, p+ 1) = IG2,i(p, p+ 1). Consequently:
α = pG1(T → w1)IG1,j(2, r)
IG1,j(1, r − 1)pG1(T → wr) (10)
It must be noted that pG1(T → w1)IG1,j(2, r) = IG1,j(1, r− 1)pG1(T → wr) since the preterminal symbol T is unified, leading to α = 1.
This process can be applied recursively to spans of any width. As a result, pG1(Ni → NjT ) + pG1(Ni → TNj) = pG2(Ni → NjT ) + pG2(Ni → TNj) if and only if IG1,i(p, q) = IG2,i(p, q). In simpler terms, the sentence probabilities calculated by G1 and G2 will be identical as long as the sum of probabilities pG1(Ni → NjT ) + pG1(Ni → TNj) matches that of G2.

heading: A.2 Structural Simplicity Bias in Multilingual
text: We demonstrate that structural simplicity bias is observed in all language datasets. For this, we train and evaluate FGG-TNPCFGs with 5, 15, and 30 nonterminals using CTB and SPMRL datasets. In Figure 7, it is shown that in all languages, the average number of unique rules decreases as the number of symbols decreases, and verifying that structural simplicity bias is a general issue occurring in all languages, not just limited to PTB.
A.3 Qualitative analysis of tree structure
In Figure 8, we qualitatively analyze the structural differences between parses induced by different grammars (Gold, FGG-TNPCFGs and parsefocused N-PCFG) with 5 nonterminal symbols for the same sentence. We verify that the FGGTNPCFGs utilize a small number of rule types to induce parses, which leads a different structure compared to the gold parse.

heading: A.4 Algorithm of Proposed Method
text: Algorithm 1 presents the pseudocode for our entire methodology. We calculate the weights to be applied in the Inside Algorithm using softmax, based on the frequency of each span within a sentence as provided by the parse trees from pre-trained parsers.
Algorithm 1 Parse-Focusing Inside algorithm 1: function PARSE-FOCUSING(s ∈ S, Tp) 2: Tp(s)← Tp ▷ Subset of parse trees 3: C(s)← COUNTSPANS(τpi , |s|) 4: W (s)← exp f(c(p, q)|τGp(si))∑
p,q exp f(c(p, q)|τGp(si)) 5: for w ← 2 to |s| do 6: for n← 0 to |s| do 7: Ii(n, n+ w)←∑
j,k ∑ m p(i, j, k)Ij(n,m)Ik(m+ 1, n+ w)
8: Ii(n, n+ w)← ws(n, n+ w)Ii(n, n+ w) 9: return p(s)
10: function COUNTSPANS(τpi , |s|) 11: Initialize c(i, j) = 0, for all 0 ≤ i, j ≤ |s| 12: for i← 0 to |s| do 13: for j ← i+ 2 to |s| do 14: for each τpi ∈ Tp do 15: if span (i, j) in τpi then 16: c(i, j)← c(i, j) + 1 17: return C(s)

heading: A.5 Quality of Focusing-Bias by the number of Multiple Parsers
text: In Figure 9, the common structures across the selected parsers are more similar to gold parse trees. This gives the intuition that assigning strong weights to parts that commonly appear across multiple parsers will lead to better performance.

heading: A.6 Experiment Details
text: Dataset Detail We follow the standard setup and preprocessing for the PTB5, utilizing sections 02- 21 as the training set, section 22 as the validation set, and section 23 as the test set excluding punctuation and trivial constituents. The most frequent 10,000 words were selected as the vocabulary, and the remaining words were treated as <unk>. To process the data, we followed the pipeline used by models employing base models and parsers as outlined in (Kim et al., 2019a; Yang et al., 2022; Shen et al., 2021; Yang et al., 2021a; Drozdov et al., 2019). We also use CTB6 and Basque, French, Ger-
5The license of PTB is LDC User Agreement for NonMembers. https://catalog.ldc.upenn.edu/LDC99T42
6The license of CTB is LDC User Agreement for NonMembers. https://catalog.ldc.upenn.edu/LDC2005T01.

heading: NP-SBJ PRP
text: man, Hebrew, Hungarian, Korean, Polish, Swedish dataset in SPMRL7.
Implementation Detail To implement our methodology on top of the base model FGGTNPCFGs, we utilized PyTorch version 2.2 (Paszke et al., 2019). For smooth processing and analysis of tree structures, we employed NLTK8.
Training Detail The hyperparameter details are identical to those found in Yang et al. (2022). The ratio of nonterminal to preterminal symbols is primarily set at 1:2, and to analyze performance based on the number of symbols, experiments were conducted using a minimum of 1 to a maximum of 4500 nonterminal symbols. Training was carried out for up to 10 epochs, with early stopping applied based on validation likelihood. For optimization, the Adam optimizer was utilized with a learning rate of 2e−3, β1 = 0.75, and β2 = 0.999. The model’s hyperparameters were set to a rank size of 1000, a symbol embedding size of 256, and a word embedding size of 200. The same hyperparameters were used for all languages without any specific tuning. All experiments were performed using an NVIDIA RTX 2080ti, and 32 experiments were conducted to verify variance for both the original FGG-TNPCFGs and our method, as shown in Table 2, while the rest of the experiments were carried out four times each. Each run takes approximately 3 hours.
7The license of SPMRL is Creative Commons Attribution 4.0 International License. https://www.spmrl.org/spmrl2013sharedtask.html
8https://www.nltk.org/

heading: A.7 Focusing-Bias of Single Parser
text: To validate the learning of various types of focusing-bias in the N-PCFG, we assessed the Intersection over Union (IoU) scores between the pre-trained single parsers and our parse-focused N-PCFGs, as shown in Table 7. The results in Table 7 indicate that our parse-focusing N-PCFGs can replicate the tree structures generated by the pretrained parsers. They show the highest congruence with FGG-TNPCFGs and the least congruence with Structformer. Despite this, the Structformer, which has the lowest IoU score, still surpasses the highest IoU score listed in Table 1. This indirectly indicates the extent of similarity between the parse trees derived from the pre-trained parsers and those obtained from the parse-focused N-PCFG.

heading: A.8 Numerical Result for Figure 10
text: In Table 8, the S-F1 and IoU scores for combinations of homogeneous parsers and heterogeneous parsers. However, heterogeneous parsers show significantly lower IoU scores across all combinations compared to homogeneous parsers. A lower IoU score signifies that the two grammars make different decisions for more spans, allowing the model to handle a wider variety of structural information in parse-focusing. The model can select the most advantageous structural information from the given options, opening up opportunities for exploration, potentially leading to higher performance.

heading: A.9 Recent Studies
text: With the dramatic growth of neural networks, various approaches have been developed to integrate the concepts of neural networks and grammars. (Shen et al., 2018, 2019) utilized tree structures within their models, hierarchically learning implicit structural information from sentences using tree structures. Meanwhile, Wang et al. (2019); Shen et al. (2021) aimed to leverage structural information within transformers by inferring structural information from sentences and using it to
mask the attention matrix to control relationships between words. Notably, Shen et al. (2021) proposed a method for simultaneously learning both constituency and dependency parsing.
There have been studies applying structural concepts, either by mimicking grammars using recurrent neural networks as in Dyer et al. (2016); Kim et al. (2019b) or by combining the inside algorithm with the concept of autoencoders as in Drozdov et al. (2019).
In particular, Kim et al. (2019a) proposed neural parameterization, assigning rule probabilities using neural networks to grammar component embeddings. Kim et al. (2019a) utilized explicit grammatical structures and the inside algorithm to induce grammars, thus enabling traditional grammat-
ical approaches. This led to the emergence of lexicalized N-PCFGs (Zhu et al., 2020; Yang et al., 2021a), which leverage lexicons. However, the inside algorithm still demanded high computational complexity, which limited the expansion of grammar sizes.
Yang et al. (2021b, 2022) addressed the high computational complexity of the inside algorithm by utilizing tensor decomposition and FGGs, which allowed for the learning of larger grammars. This approach, which solves issues through the number of parameters, aligns with the methods of Petrov et al. (2006).

heading: Future Work
text: In the future, more light and adaptive structural bias generation is required