Tree-Averaging Algorithms for Ensemble-Based Unsupervised Discontinuous Constituency Parsing (2024)

Abstract: We address unsupervised discontinuous constituency parsing, where we observe a high variance in the performance of the only previous model in the literature. We propose to build an ensemble of different runs of the existing discontinuous parser by averaging the predicted trees, to stabilize and boost performance. To begin with, we provide comprehensive computational complexity analysis (in terms of P and NP-complete) for tree averaging under different setups of binarity and continuity. We then develop an efficient exact algorithm to tackle the task, which runs in a reasonable time for all samples in our experiments. Results on three datasets show our method outperforms all baselines in all metrics; we also provide in-depth analyses of our approach.1

heading: 1 Introduction
text: Unsupervised parsing has been attracting the interest of researchers over decades (Klein and Manning, 2002; Klein, 2005; Snyder et al., 2009; Shen et al., 2019; Cao et al., 2020). Compared with supervised methods, unsupervised parsing has its own importance: (1) It reduces the reliance on linguistically annotated data, and is beneficial for low-resource languages and domains (Kann et al., 2019). (2) Discovering language structures in an unsupervised way helps to verify theories in linguistics (Goldsmith, 2001) and cognitive science (Bod, 2009). (3) Unsupervised parsing methods are applicable to other types of streaming data, such as motion-sensor signals (Peng et al., 2011).
A constituency tree, a hierarchy of words and phrases shown in Figure 1, is an important parse structure (Carnie, 2007). Researchers have proposed various approaches to address unsupervised constituency parsing, such as latent-variable methods (Clark, 2001; Petrov and Klein, 2007;
1Code available at https://github.com/MANGA-UOFA/ TAA4EUDCP
Kim et al., 2019a) and rule-based systems (Cao et al., 2020; Li and Lu, 2023). In our previous work (Shayegh et al., 2024), we reveal that different unsupervised parsers have low correlations with each other, and further propose an ensemble approach based on dynamic programming to boost performance.
One limitation of most existing studies is that they only address continuous constituency parsing, that is, a constituent can only be a continuous span of words (Figure 1a). However, a constituent may be discontinuous (McCawley, 1982).2 In Figure 1b, for example, it is linguistically arguable that wake and up form a constituent in the sentence “Wake your friend up.” Such discontinuous constituents are more common in certain languages (such as German shown in Figure 1c) than in English (Skut et al., 1997), but they have been less tackled in the literature. Very recently, Yang et al. (2023) proposed an unsupervised parsing method based on a mildly context-sensitive grammar (Joshi, 1985) that allows discontinuous parse structures, known as discontinuous trees (Tomita, 1990). As an early attempt, their performance appears to be low, and we find that their approach is noisy and exhibits low
2In the literature, a continuous parse structure is also known to be projective, and a discontinuous parse structure is non-projective (Versley, 2014).
15135
correlation in different runs with random seeds. In this work, we propose an ensemble method to discontinuous constituency parsing by tree averaging, following our previous work (Shayegh et al., 2024), in which we address the ensemble of continuous constituency parsing by CYK-like dynamic programming (Younger, 1967). However, the previous approach is not directly applicable to this paper because of the discontinuous structures in our setup. Consequently, the seemingly small change in the setting (continuous vs. discontinuous) leads to a completely different landscape of problems, requiring advanced algorithmic analysis and development.
Specifically, we first analyze the computational complexity of tree averaging under different setups, such as binarity and continuity. We show certain problems are in the P category, whereas general tree averaging is NP-complete. Then, we develop an algorithm utilizing the meet-in-the-middle technique (Horowitz and Sahni, 1974) with effective pruning strategies, making our search practical despite its exponential time complexity.
Our experiments on German and Dutch show that our approach largely outperforms previous work in terms of continuous, discontinuous, and overall F1 scores. To the best of our knowledge, our algorithm is the first that can handle non-binary constituents in the setting of unsupervised constituency parsing.
To sum up, our main contributions conclude: (1) proposing an ensemble approach to unsupervised discontinuous constituency parsing, (2) theoretically analyzing the computational complexity of various tree-averaging settings, and (3) conducting experiments on benchmark datasets to verify the effectiveness of our approach.

heading: 2 Approach

heading: 2.1 Unsupervised Discontinuous Constituency Parsing
text: In linguistics, a constituent is one or more words that act as a semantic unit in a hierarchical tree structure (Carnie, 2007). Discontinuous constituents are intriguing, where a constituent is split into two or more components by other words (Figure 1; McCawley, 1982; Tomita, 1990).
Unsupervised discontinuous constituency parsing aims to induce—without using linguistically annotated data for training—a parse structure that may contain discontinuous constituents. As men-
tioned in §1, unsupervised parsing is an important research topic, as it potentially helps low-resource languages, the development of linguistic and cognitive theory, as well as the processing of non-textual streaming data.
Yang et al. (2023) propose the only known unsupervised discontinuous parsing approach, based on linear context-free rewriting systems (LCFRS; Vijay-Shanker et al., 1987), a type of mildly context-sensitive grammars (Joshi, 1985), which can model certain discontinuous constituents. They focus on binary LCFRS and limit the fan-out of their grammar to be at most 2, known as LCFRS-2 (Stanojević and Steedman, 2020), meaning that each constituent can contain up to two nonadjacent components3 (illustrated in Figure 1). Maier et al. (2012) observe such structures cover most cases in common treebanks. Essentially, the LCFRS-2 grammar used in Yang et al. (2023) is a 6-tuple G = (S,N1,N2,P,Σ,R), where S is the start symbol, N1 a finite set of non-terminal symbols with fan-out being 1, N2 fan-out being 2, P preterminals, and Σ terminals. R is a finite set of rules, following one of the forms:
R1 : S(x)→ A(x) A ∈ N1 R2 : A(xy)→ U(x)U ′(y) A ∈ N1 R3 : A(xyz)→ U(y)B(x, z) A ∈ N1 R4 : A(x, y)→ U(x)U ′(y) A ∈ N2 R5 : A(xy, z)→ U(x)B(y, z) A ∈ N2 R6 : A(xy, z)→ U(y)B(x, z) A ∈ N2 R7 : A(x, yz)→ U(y)B(x, z) A ∈ N2 R8 : A(x, yz)→ U(z)B(x, y) A ∈ N2 R9 : T (w)→ w T ∈ P, w ∈ Σ
for B ∈ N2, and U,U ′ ∈ N1 ∪ P . Note that x in a rewriting rule A(x) is not an input string. Instead, it is a placeholder suggesting that A(x) has a fanout 1. Likewise, A(x, y) has a fan-out 2, with x and y being the two nonadjacent components.
Such a grammar may handle certain types of discontinuous constituents. Take R3 as an example. The left-hand side A(xyz) implies the placeholder string xyz is a constituent that can be split into three adjacent components: x, y, and z in order. The notation B(x, z) in the rule asserts that x and z form a constituent, although they are not adjacent in the original string xyz.
We follow Yang et al. (2023) and train a
3A component refers to a span of one or more consecutive words in a sentence.
probabilistic LCFRS-2, parametrized by a tensor decomposition-based neural network (TN-LCFRS). The objective is to maximize the likelihood of sentence reconstruction by marginalizing the grammar rules. Compared with general context-sensitive grammar, LCFRS-2 balances the modeling capacity and polynomial-time inference efficiency; thus, it is also used in supervised discontinuous parsing (Maier, 2010; Cranenburgh et al., 2016). We refer interested readers to Yang et al. (2023) for the details of TN-LCFRS training and inference.
In our work, we observe that TN-LCFRS exhibits high variance in performance, especially for discontinuous constituents (Figure 2). In addition, there is a negative correlation between performance on continuous and discontinuous constituents among different runs: some are better at continuous constituents, while others are better at discontinuous ones.
In our previous work (Shayegh et al., 2024), we show that an ensemble model can utilize different expertise of existing continuous parsers and smooth out their noise. A natural question now is: Can we build an ensemble of discontinuous constituency parsers?
In the rest of this section, we perform in-depth theoretical analysis of the problem, and show that with seemingly harmless alteration of setups, the problem may belong to either P or NP-complete complexity categories. We further develop a meetin-the-middle search algorithm with efficient pruning to solve our problem efficiently in practice.

heading: 2.2 Averaging over Constituency Trees
text: We adopt the tree-averaging notion in Shayegh et al. (2024), which suggests the ensemble output should be the tree with the highest average F1-score against the ensemble components (referred to as individuals):
T ∗ = argmax T∈T
K∑
k=1
F1(T, Tk) (1)
where T is the search space given an input sentence, and Tk is the parse tree predicted by the kth individual. The F1 score is commonly used for evaluating constituency parsers, and is also our ensemble objective, given by F1(Tprd, Tref) =
2|C(Tprd)∩C(Tref)| |C(Tprd)|+|C(Tref)| ,
where Tprd and Tref denote the predicted and reference trees, respectively, while C(T ) represents the set of constituents in a tree T .
It is noted that existing unsupervised parsers can only produce binary trees in both continuous and discontinuous settings (Shen et al., 2018a; Kim et al., 2019a; Yang et al., 2023). The binary property asserts that, given a length-n sentence, |C(Tk)| = 2n − 1 for every individual k, thus simplifying Eqn. (1) to
T ∗ = argmax T∈T
∑ c∈C(T ) h(c)
|C(T )|+ 2n− 1 (2)
Here, h(c) counts the occurrences of a constituent c in the trees predicted by the individuals. We call h(c) the hit count of c.
We point out that the output of our approach does not have to be a binary tree, and we will empirically analyze the output binarity in §3.2.
We would like to examine the computational complexity for tree averaging, such as P and NP categories. To begin with, we consider the decision problem (i.e., whether there exists a tree satisfying some conditions) corresponding to the search problem (i.e., finding the best tree), which is standard in complexity analysis (Arora and Barak, 2009).
Problem 1 (Averaging binary trees with bounded fan-out). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the trees are binary and have a fan-out of at most F . Is there a constituency tree T such that∑K
k=1 F1(T, Tk) ≥ z?
Theorem 1. Problem 1 belongs to P.
Proof sketch. In our previous work (Shayegh et al., 2024), we present a polynomial-time dynamic programming (DP) algorithm to average continuous binary trees, where the outputs are restricted to binary trees as well; here, continuous trees can be seen as having max fan-out 1. For non-binary outputs, the DP table can be augmented by an additional axis whose size is bounded by the sentence
length n. To handle discontinuity, we may augment the DP table with additional axes based on the maximum fan-out F . Overall, the time complexity of the DP algorithm is O(n4F+1); thus, Problem 1 is in P. See Appendix A.1 for the detailed proof.4
A follow-up question then is whether there is a polynomial-time algorithm when we relax the assumptions of input being binary and having bounded fan-out. Having non-binary inputs is intriguing since the above DP relies on Eqn. (2), which only holds for binary inputs.
Problem 2 (Averaging trees with bounded fan-out). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the fan-out is at most F but the trees may be nonbinary. Is there a constituency tree T such that∑K
k=1 F1(T, Tk) ≥ z?
Theorem 2. Problem 2 belongs to P.
Proof sketch. If we fix the number of nodes τ of the output tree, the above DP can be reused to solve this problem. We may enumerate all possible values of τ , which must satisfy n < τ < 2n. Thus, Problem 2 can be solved in O(n4F+2) time and belongs to P. See Appendix A.2 for the proof.
However, if the fan-out is unbounded, the difficulty is that the above DP table grows exponentially with respect to F . Since the problem is polynomialtime verifiable, it surely belongs to NP, but whether it belongs to P, NP-complete, or both remains unknown for binary inputs.
Open Problem 1 (Averaging binary trees). What is the exact complexity category (P or NP-complete) of averaging binary trees with unbounded fan-out?
For averaging general trees (non-binary inputs with unbounded fan-out), we can show that it belongs to NP-complete.
Problem 3 (Averaging trees). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the fan-out is unbounded and the trees may be non-binary. Is there a constituency tree T such that ∑K k=1 F1(T, Tk) ≥ z?
Theorem 3. Problem 3 belongs to NP-complete. 4Corro (2023) develops a CYK-like algorithm for spanbased nested named-entity recognition that takes discontinuous and non-binary structures into account. However, their algorithm limits its search space to constituents that contain at most one smaller multi-word constituent inside, making it not applicable to our scenario.
Proof sketch. It is easy to show that, given a certificate, Problem 3 is polynomial-time verifiable. For the completeness, we reduce the max clique problem, a known NP-complete problem, to Problem 3. See Appendix A.3 for the detailed proof.
The above theoretical analysis provides a deep understanding of building tree ensembles, summarized in Table 1. In our experiments, the task falls into Problem 1 because the only existing unsupervised discontinuous parser (Yang et al., 2023) produces binary trees with fan-out at most 2. The problem belongs to P.
That being said, a practical approach should consider not only algorithmic complexity, but also the specific properties of the task at hand. A DP algorithm that one may develop for our task has a complexity of O(n9), as discussed in the proof of Theorem 1, which appears impractical despite its polynomial time complexity.
In the rest of this section, we develop a more general search algorithm that works at the level of Open Problem 1 but also solves Problem 1 more efficiently in practice than high-order polynomial DP.

heading: 2.3 Our Search Algorithm
text: Our work concerns building an ensemble of binary, bounded-fan-out, discontinuous trees generated by the unsupervised parser in Yang et al. (2023).
We will develop a general search algorithm (regardless of fan-out) with strong pruning that only needs to consider a few candidate constituents, bringing down the O(22n) complexity of exhaustive search5 to O(2n). We will further halve the exponent using a meet-in-the-middle technique, resulting in an overall complexity of O(2n2 n2). As a result, our algorithm remains exact search while being efficient for all samples in the dataset; when a sentence is short, our algorithm is even much faster than the DP with O(n9) complexity.
5For a length-n sentence, there are 2n − 1 possible constituents because an arbitrary non-empty set of words can be a constituent. To build a constituency tree, the exhaustive search needs to look into any combination of constituents.
To solve the tree-averaging problem, we first convert it into an equivalent graph problem for clarity. Specifically, we construct an undirected graph G with vertices being all possible constituents (although they can be largely pruned), weighted by corresponding hit counts, as defined after Eqn. (2). We put an edge between two vertices if and only if their corresponding constituents can coexist within a constituency tree, that is to say, they are either disjoint or inclusive of one another. Then, we formulate the below graph-search problem, which can solve the original tree-search problem for ensemble discontinuous parsing.
Problem 4 (Normalized Max Weighted Clique). Consider a weighted undirected graph G = ⟨V,E⟩ and an objective function f(Q;α1, α2) =∑
v∈Q w(v)+α1 |Q|+α2 , where α1, α2 ∈ R, Q ⊆ V , and w(v) is the weight of v. What is the clique Q that maximizes f(Q;α1, α2)?
Lemma 1. Given G as described above, the clique Q∗ maximizing f(Q;α1, α2) corresponds to a constituency tree, if α1 ≤ Kα2 where K is the number of individuals. (Proof in Appendix A.4.)
Theorem 4. The average constituency tree T ∗ in Eqn. (2) is polynomial-time reducible from Q∗ in Problem 4 with α1 = 0 and α2 = 2n − 1, if G is built as above.
Proof. Suppose the constituents corresponding to a clique Q in the graph G form a constituency tree, denoted by TQ. We have ∑ v∈Qw(v) =∑
c∈TQ h(c) by the construction of G. Thus, the objective function of Problem 4 is
f(Q; 0, 2n− 1) = ∑ v∈Qw(v) + 0
|Q|+ 2n− 1 (3)
=
∑ c∈C(TQ) h(c)
|C(TQ)|+ 2n− 1 (4)
which is the same as the argmax objective in Eqn. (2). It is easy to see that each constituency tree corresponds to a clique in G. Therefore, the corresponding constituency tree to Q∗, guaranteed by Lemma 1, maximizes Eqn. (2). In other words, we have T ∗ = TQ∗ .
Problem 4 generalizes a standard max clique problem (Arora and Barak, 2009), which may be solved in O(2|V |) time complexity by exhaustive search. The meet-in-the-middle technique (Horowitz and Sahni, 1974) can be used to address the max clique problem, reducing the
complexity from O(2|V |) to O(2 |V |2 |V |2). In Appendix B, we develop a variant of the meet-in-themiddle algorithm to solve Problem 4.

heading: 2.4 Candidate Constituents Pruning
text: The efficiency of our algorithm for Problem 4 depends on the number of vertices in the graph. In our construction, the vertices in G correspond to possible constituents, which we call candidates.
A naïve approach may consider all O(2n) possible constituents, which are non-empty combinations of words in a length-n sentence. Thus, our meet-in-the-middle algorithm has an overall complexity of O(22n−1+2n) for tree averaging.
In this part, we theoretically derive lower and upper bounds for a candidate’s hit count. If a constituent has a lower hit count than the lower bound, it is guaranteed not to appear in the average tree. On the other hand, a constituent must appear in the average tree, if it has a higher hit count than the upper bound. We may exclude both cases from our search process, and directly add the must-appear candidates to the solution in a post hoc fashion.
Let P be a set of constituents that are known to be in the average tree T ∗. Here, P may even be an empty set. We may derive a hit-count lower bound for other constituents (not in P ), as stated in the following theorem.
Theorem 5 (Lower bound). For every constituent c ∈ C(T ∗)\P , where P ⊆ C(T ∗), we have
h(c) > min |P |≤j≤2n−2
∑j−|P | i=1 λ + i + ∑ c′∈P h(c
′) j + 2n− 1 (5)
where λ+i is the ith smallest positive hit count. (Proof in Appendix A.5.)
The theorem suggests that constituents having a hit count lower than the above threshold can be removed from the search. If we set P = ∅, we have h(c) > 0, immediately pruning all zero-hit constituents. In other words, we may only consider the constituents appearing in at least one individual, which largely reduces the graph size from O(2n) toO(nK) for K individuals. In fact, the graph can be further pruned with the below theorem.
Theorem 6 (Upper bound). Let c be a constituent with a hit count of K, where K is the number of individuals. (a) The constituent c is compatible— i.e., may occur in the same constituency tree—with every constituent in the average tree. (b) The constituent c appears in the average tree. (Proof in Appendix A.6.)
Theorem 6b suggests that the search process may exclude the constituents that occur in all individuals, denoted by P = {c : h(c) = K}. In this case, we may solve a reduced version of Problem 4 with the pruned graph and α1 = ∑ c′∈P h(c
′), α2 = |P |+2n−1. The α1 and α2 in Theorem 4 can be modified accordingly.6 When we add P back to the solution, the connectivity of P is guaranteed by Theorem 6a, forming the solution (guaranteed to be a clique) for the original Problem 4.
To analyze the worst-case complexity, we notice that single words are constituents that must occur in all individuals. At least, we may set P = {c : c is a single word}, and Theorem 5 yields a lower bound of
h(c) > min n≤j≤2n−2
0 + nK j + 2n− 1 > nK 4n = K 4 (6)
Then, we can find an upper bound for the number of candidates kept for the search process. Since the sum of all hit counts is always (2n− 1)K, the number of candidates with a higher hit count than the threshold in Eqn. (6) is bounded by
|V | ≤ ⌊(2n− 1)K
K/4
⌋ < 8n (7)
This shows that our pruning mechanism, profoundly, reduces the graph size from O(2n) to O(n), which in turn reduces the overall time complexity of our tree averaging from O(22n−1+2n) to O(2n2 n2) in the worst-case scenario.
In practice, the pruning is even more effective when the individuals either largely agree or largely disagree with each other. In the former case, we will have more all-hit constituents excluded from the search and consequently the lower bound increases, whereas in the latter case, many constituents will fall short of the lower bound. Empirically, our algorithm is efficient for all samples in our experiments.

heading: 3 Experiments

heading: 3.1 Settings
text: Datasets. We evaluated our method on Dutch and German datasets, where discontinuous constituents are relatively common. In particular, we used the LASSY treebank (Van Noord et al., 2013) for Dutch. For German, we trained our individuals on the union of NEGRA (Skut et al., 1997) and TIGER (Brants et al., 2002) treebanks, while test-
6The proof is parallel to that of the original version of Theorem 4, and we leave it as an exercise for readers.
ing and reporting the performance on their test sets respectively. Our settings strictly followed Yang et al. (2023) for fair comparison.
Evaluation Metrics. F1 scores are commonly used for evaluating constituency parsing performance (Klein, 2005; Shen et al., 2018a, 2019; Kim et al., 2019a,b). In our work of unsupervised discontinuous constituency parsing, we report corpuslevel F1 scores of all constituents, continuous constituents, and discontinuous constituents, denoted by F overall1 , F cont 1 , and F disco 1 , respectively. We followed the previous work (Yang et al., 2023), which discards punctuation and excludes trivial constituents (the whole sentence and single words) when calculating the F1 scores.
Setups of Our Ensemble. As mentioned in §2.1, we used different runs of the tensor decompositionbased neural LCFRS (TN-LCFRS; Yang et al., 2023) as our individual models. Our experiment was based on the released code7 with default training hyperparameters. However, we find the TNLCFRS is highly unstable, with a very high variance and lower overall performance than Yang et al. (2023). As a remedy, we trained the model 10 times and selected the top five based on validation F overall1 , and this yields performance close to Yang et al. (2023). In addition, we observe that random weight initializations lead to near-zero correlation of the predicted discontinuous constituents, but our ensemble method expects the individuals to at least agree with each other to some extent. Therefore, our different runs started with the same weight initialization8 but randomly shuffled the order of training samples to achieve stochasticity.
Our ensemble method does not have hyperparameters. However, F overall1 and F disco 1 may not correlate well. Since our ensemble objective is solely based on F overall1 , we may enhance F disco 1 by weighting the individuals with validation F disco1 scores in hopes of achieving high performance in all aspects. Note that weighting individuals does not hurt our theoretical analysis and algorithm, because weighting is equivalent to duplicating individuals (since all the weights are rational numbers) and can be implemented by modifying hit counts without actual duplication.
Baselines. Our ensemble individual, TN-
7https://github.com/sustcsonglin/TN-LCFRS 8Our pilot study shows the proposed ensemble method is not sensitive to the initialization, as long as it is shared among different runs. This is also supported by the evidence that our approach performs well consistently on three datasets.
LCFRS (Yang et al., 2023), is naturally included as a baseline. In addition, we consider different variants of continuous PCFG parsers and discontinuous LCFRS parsers, based on vanilla neural networks, compound prior (Kim et al., 2019a), and tensor decomposition-based neural networks (Yang et al., 2021), denoted by [N|C|TN]-[PCFG|LCFRS].

heading: 3.2 Results and Analyses
text: Main Results. Tabel 2 presents the main results on three datasets. We replicated TN-LCFRS with five runs as our ensemble individuals; our results are similar to Yang et al. (2023), showing the success of our replication (Rows 9 vs. 8).
In our study, we observe TN-LCFRS behaves inconsistently in different runs: some runs are good at continuous constituents (Row 11), while others are good at discontinuous ones (Row 12); both may disagree with the best run according to F overall1 (Row 10).
Our ensemble approach (Rows 14) achieves F cont1 and F disco 1 scores comparable to, or higher than, all individuals. This eventually leads to a much higher F overall1 score, with a p-value < 0.01 in an Improved Nonrandomized Sign (INS) test (Starks, 1979) with the binomial comparison over sentence-level performance.9 Results are consistent on all three datasets.
Binary vs. Non-Binary Ensemble. To the best of our knowledge, we are the first to address
9We chose INS test because it can properly handle a large number of ties (neutral cases), when they do not necessarily suggest the equivalency of the models but the inadequacy of test samples (e.g., no discontinuous constituents in a sentence).
non-binary unsupervised constituency parsing. A natural question is whether and how non-binarity in the output improves the performance, given that all individuals are binary. To this end, we conducted an experiment by restricting the output to binary trees.10 We compare the results with the non-binary setting in Rows 13–14, Table 2, showing the consistent superiority of our non-binary ensemble in all metrics and datasets. To further understand the effect of non-binarity, we report in Table 3 true positive and false positive counts on the LASSY dataset.11 Results suggest that the non-binary setting largely eliminates false positive constituents, as the model may opt to predict fewer constituents than that in the binary setting.
Number of Ensemble Individuals. We varied the number of individuals in the ensemble to analyze its effect. In Figure 3a, we picked a random subset of individuals from a pool of 10 and av-
10For binary outputs, we may still perform the meet-in-themiddle search for non-binary trees, with the hit count as the objective, and binarize them by post-processing. Note that Theorem 5 for pruning does not hold in this case, but we may still safely ignore zero-hit constituents, making the algorithm affordable (although it is not as efficient as the non-binary setting where Theorem 5 can be applied).
11We chose LASSY as the testbed for all our analyses due to the limit of space and time, because it is relatively more stable in terms of F disco1 than other datasets.
eraged the results over 30 trials. Overall, more individuals lead to higher performance and lower variance in all F1 scores, although the performance may be saturated if there are already a large number of individuals.
In addition, we present an analysis of the best-toworst incremental ensembles in Figure 3b, where we add individuals to the ensemble from the best to worst based on F overall1 . As seen, adding weak individuals to the ensemble does not hurt, if not help, the performance in our experiments, demonstrating that our ensemble approach is robust to the quality of individuals.
Efficiency Analysis. We analyze the effect of our pruning mechanism and the meet-in-the-middle algorithm. Pruning serves as preprocessing for our search, and we show the number of remaining candidate constituents in Figure 4. The empirical results confirm Theorem 5 bounding the number of candidates linear in both sentence length and the number of individuals, as well as Theorem 6 further bounding it by a constant with respect to the number of individuals.
We further analyze wall clock run time in Figure 5, as the actual execution time may be different from algorithmic complexity. In §A.1, we provide
a dynamic programming (DP) algorithm with time complexity of O(n9) for a length-n sentence. As shown in the figure, the DP fails to serve as a practical algorithm due to the high-order polynomial. By contrast, our exponential-time search algorithm, even without the meet-in-the-middle technique, is able to run in a reasonable time for many samples (especially short ones) because of the strong pruning mechanism. Our meet-in-the-middle technique further speeds up the search, making our algorithm efficient and faster than DP for all the data samples.
It is emphasized that this experiment also empirically verifies the correctness of all our algorithms, as they perform exact inference and we have obtained exact results by using different algorithms.
Additional Results. We provide additional results in the appendix. C: Performance by constituency types; and D: Case study.

heading: 4 Related Work
text: Constituency parsing carries a long history in natural language processing research (Charniak, 2000; Klein, 2005; Kallmeyer and Maier, 2010; Li et al., 2019). Different setups of the task have been introduced and explored, including supervised and unsupervised, continuous and discontinuous constituency parsing (Shen et al., 2018a,b, 2019; Corro, 2020; Fernández-González and Gómez-Rodríguez, 2020, 2021, 2023; Chen and Komachi, 2023; Yang et al., 2023; Yang and Tu, 2023). In the unsupervised setup, researchers typically define a joint distribution over the parse structure and an observable variable, e.g., the sentence itself, and maximize the observable variable’s likelihood through marginalization (Kim et al., 2019a,b). To the best of our knowledge, previous unsupervised parsing studies are all restricted to binary structures to squeeze the marginalization space, and we are the first to address non-binary unsupervised parsing and show
non-binarity is beneficial to parsing performance. Ensemble methods strategically combine multiple models to improve performance, rooted in the bagging concept where different data portions are used to train multiple models (Breiman, 1996a; Hastie et al., 2009). To build an ensemble, straightforward methods include averaging and voting (Breiman, 1996a,b). For outputs with internal structures, minimum Bayes risk decoding (MBR; Bickel and Doksum, 2015) can be used to build an ensemble, where the vote is the negative risk in MBR. However, existing MBR approaches are mostly selective, where the output is selected from a candidate set (Kumar and Byrne, 2004; Titov and Henderson, 2006; Shi et al., 2022). Petrov and Klein (2007) formulate MBR for supervised constituency parsing and propose to search for the global optimum decoding when the risk allows for dynamic programming. Smith and Smith (2007) extend the idea to non-projective dependency parsing. In our previous work (Shayegh et al., 2024), we formally introduce generative MBR and develop an algorithm that searches for a binary continuous tree. This paper extends our previous work and searches in the space of discontinuous constituency trees, leading to significant algorithmic design and theoretical analysis.

heading: 5 Conclusion
text: In this work, we address ensemble-based unsupervised discontinuous constituency parsing by tree averaging, where we provide comprehensive complexity analysis and develop an efficient search algorithm to obtain the average tree. Our experiments on Dutch and German demonstrate the effectiveness of our ensemble method. To the best of our knowledge, we are also the first to address, and show the importance of, non-binary structures in unsupervised constituency parsing.

heading: 6 Limitations
text: Our work demonstrates both theoretical depth and empirical effectiveness, but may also have limitations.
First, our work is focused on syntactic parsing, and we have provided a series of theoretical analyses and algorithmic designs for averaging constituency trees under different setups. Following the trajectory, our ensemble approach may be extended to other structures beyond parsing. We are happy to explore this direction as future work. For
example, a concurrent study of ours develops algorithms for ensemble-based text generation (Wen et al., 2024).
Second, our model is only tested on Dutch and German datasets. This is partially because of the established setups in previous work (Yang et al., 2023) and the lack of annotated treebanks. Notice that English is usually excluded from the study of unsupervised discontinuous parsing, because English discontinuous structures are too rare for any model to discover. A potential future direction is multilingual linguistic structure discovery, perhaps, with ensembles of different languages.
Third, our theoretical analysis leaves an open problem about averaging binary trees. However, pointing to open problems is usually considered a contribution (instead of a weakness) in theoretical computer science. Our theoretical analysis is also crucial to the understanding of our algorithms, because we now know that our proposed method works at the level of Open Problem 1 and can be easily extended to the level of Problem 3 following the technique proposed in Appendix A.2.

heading: Acknowledgments
text: We would like to thank all reviewers and chairs for their valuable and constructive comments. The research is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Amii Fellow Program, the Canada CIFAR AI Chair Program, an Alberta Innovates Program, a donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca). We also thank Yara Kamkar for providing advice on the algorithms.

heading: A Proofs
text: A.1 Proof of Theorem 1 Problem 1 (Averaging binary trees with bounded fan-out). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the trees are binary and have a fan-out of at most F . Is there a constituency tree T such that ∑K k=1 F1(T, Tk) ≥ z? Theorem 1. Problem 1 belongs to P.
Proof. Following our previous work (Shayegh et al., 2024), we may design a DP table with two axes being the start and end of every component of a constituent, thus 2F -axes for fan-out F . For outputs being non-binary, our DP algorithm requires one additional axis to indicate the number of nodes in a subtree.12
We define a DP variable H(c, τ) as the best total hit count for a τ -node constituency substree over a constituent c, which may be discontinuous. A constituent with fan-out f has f components, each being a span of consecutive words. Therefore, c can be represented by 2F numbers indicating the beginnings and ends of the components, as the fan-out is bounded by F . For τ , it is upper-bounded by 2n− 1 for a length-n sentence. Overall, the DP table has a size ofO(n2F+1). Specially, we further define H(c, 0) = 0 for every c.
For initialization, we consider every single-word constituent c and set H(c, 1) = h(c), where h(c) is the hit count of c in T1, · · · , TK . We also set H(c, i) = −∞ for every i > 1.
For recursion, we divide a constituent c into smaller sub-constituents based on alternation points j = (0, j1, j2, · · · , j2F−1, |c|), where |c| is the number of words in c, shown in Figure 6. We further join sub-constituents based on the parity (even or odd) of their indexes.
oj = ⊕
i=1,3,··· ,2F−1 c[ji−1 : ji] (8)
ej = ⊕
i=2,4,··· ,2F c[ji−1 : ji] (9)
where c[b : e] denotes a sub-constituent containing bth to eth words in c, and ⊕ symbol is the joint of sub-constituents. We also distribute the capacity of the number of nodes into ej and oj branches by setting s as the oj’s share and assigning the rest to ej .
To find the best alternation points and shares, we iterate over all possible values for them:
j∗τ , s ∗ τ = argmax
j, 0≤s≤τ
[ H(oj , s) +H(ej , τ − s) ] (10)
where j = (0, j1, · · · , j2F−1, |c|) must satisfy ji ≤ ji+1 ≤ |c|, j1 ̸= |c|, and if ji ̸= |c|, then ji < ji+1. We discard the cases that oj or ej have a fan-out greater than F , because they are guaranteed not to appear in the output (Theorem 5, whose proof does not rely on this theorem).
Our tree-building search processing first assumes a node is binary and then decides whether to join its parent and children to achieve non-binarity. Therefore, we calculate the best hit count when c itself is excluded, denoted by Hexcl, or included, denoted by Hincl. We have
Hexcl(c, τ) = H(oj∗τ , s ∗ τ ) +H(ej∗τ , τ − s∗τ ) (11)
Hincl(c, τ) = Hexcl(c, τ − 1) + h(c) (12) Finally, the recursion is
H(c, τ) = max{Hexcl(c, τ), Hincl(c, τ)} (13)
The best score of ∑K
k=1 F1(T, Tk) for the sentence S is
argmax n<τ<2n
H(S, τ)
τ + 2n− 1 (14)
12If the output is restricted to binary trees, we do not need to track the number of nodes in a subtree, because it is always 2n− 1 for n leaves.
according to Eqn. (2). Problem 1, as a decision problem, can then be solved trivially by checking whether the score in (14) is greater than or equal to z.
Overall, the time complexity for a recursion step isO(n2F ) to find j∗τ and s∗τ . With anO(n2F+1)-sized DP table, the complexity of the entire DP algorithm is O(n4F+1), which is polynomial. Therefore, Problem 1 belongs to P.
The above proof concerns the decision problem (whether the score reaches or exceeds a threshold). To search for the average constituency tree, we may backtrack the best corresponding constituency subtrees during the recursion.
Notice that restricting output to be binary tree simplifies the algorithm by always setting s∗τ = 2|oj |− 1, H(c, τ) = Hincl(c, τ), and only considering the computation of H(c, τ) if τ = 2|c| − 1. It reduces the complexity of a recursion step to O(n2F−1) and that of the DP-table size to O(n2F ), resulting in an overall complexity of O(n4F−1).
The soundness of the DP can be cross-validated by our meet-in-the-middle search (§2.3 and §B). As both are exact algorithms, they should output exactly the same results. This is what we observed in our experiments, providing strong empirical evidence that both of our algorithms are sound.
A.2 Proof of Theorem 2
Problem 2 (Averaging trees with bounded fan-out). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the fan-out is at most F but the trees may be non-binary. Is there a constituency tree T such that ∑K k=1 F1(T, Tk) ≥ z?
Theorem 2. Problem 2 belongs to P.
Proof. In §A.1, we provide a DP algorithm and by backtracking we can easily obtain, for any τ , the constituency tree T that maximizes ∑ c∈C(T ) h(c) such that |C(T )| = τ (i.e., the tree having τ nodes). The DP algorithm, in fact, works for any scoring function h over constituents. Therefore, we have a polynomial-time solver for the following problem:
Fixed-Size Maximization. Given a scoring function h over constituents of a length-n sentence and a natural number τ such that n < τ < 2n, what is the tree T that maximizes ∑ c∈C(T ) h(c) with |C(T )| = τ?
For Problem 2, we may enumerate all possible values of τ , i.e., n < τ < 2n. Given a fixed τ , we have
T (τ) = argmax T :|T |=τ
K∑
k=1
F1(T, Tk) (15)
= argmax T :|T |=τ
K∑
k=1
|C(T ) ∩ C(Tk)| |C(T )|+ |C(Tk)|
(16)
= argmax T :|T |=τ
∑
c∈C(T )
K∑
k=1
1[c ∈ C(Tk)] τ + |C(Tk)| ︸ ︷︷ ︸ ĥ(c)
(17)
= argmax T :|T |=τ
∑
c∈C(T ) ĥ(c) (18)
In Eqn. (18), we define a generalized scoring function ĥ , which can be thought of as a weighted hit count. In other words, the overall-F1 maximization in Eqn. (15) can be reduced to the above fixed-size maximization problem where the number of nodes is given, which can be solved in polynomial time. We may repeatedly solve the problem for (n− 2)-many values of τ and find the best answer among them:
τ∗ = argmax n<τ<2n
K∑
k=1
F1(T (τ), Tk) (19)
T ∗ = T (τ ∗) (20)
This does not push the complexity beyond polynomial. Finally, we can answer Problem 2 by checking whether ∑K k=1 F1(T ∗, Tk) ≥ z.
A.3 Proof of Theorem 3
Problem 3 (Averaging trees). Consider a number z and constituency trees T1, · · · , TK with the same leave nodes, where the fan-out is unbounded and the trees may be non-binary. Is there a constituency tree T such that ∑K k=1 F1(T, Tk) ≥ z?
Theorem 3. Problem 3 belongs to NP-complete.
Proof. To show a problem is NP-complete, we need to first show it is an NP problem, i.e., polynomialtime solvable with a non-deterministic Turing machine. Then, we need to show its completeness, i.e., any NP problem, or a known NP-complete problem, can be reduced to this problem in polynomial time.
[Being NP] Being non-deterministic polynomial-time solvable is equivalent to the ability to be verified with a certificate in polynomial time (Arora and Barak, 2009). In our case, verifying the score of any candidate tree (which may serve as the certificate) can be done in polynomial time, proving the NP part.
[Being complete] To show the completeness, it suffices to reduce a known NP-complete problem to the problem at hand. In particular, we would reduce the max clique problem, known to be NP-complete (Arora and Barak, 2009), to our problem.
Max Clique. Consider a number z and a graph G. Is there a clique in G having at least z vertices?
For reduction, we use a solver for Problem 3 to solve the max clique problem with G = ⟨V,E⟩ and z being given.
Without loss of generality, we may assume that G does not contain any vertex that is connected to all other vertices for the rest of this proof. This is because such a vertex is guaranteed to appear in any maximal clique and can be pruned. Then the original problem is equivalent to determining whether the pruned graph has a clique with (z − a)-many vertices, where a is the number of pruned vertices. Notice that such pruning can be accomplished in polynomial time, thus not changing the complexity category.
We construct an instantiation of Problem 3 as follows. For every ui ∈ V , construct a set Si initialized by a special symbol ti. For every disconnected vertex pair (ui, uj) such that i < j, place a special symbol tji in both Si and Sj , shown in Figure 7. We can view each ti and t j i as words in a sentence. Each Si represents a possible constituent containing the words corresponding to its elements.
For each Si, construct a constituency tree Ti by all the single words, the whole-sentence constituent, and Si. Note that Si cannot be a single word because there is no vertex in G connected to all other vertices. Moreover, Si cannot be the whole-sentence constituent either, because Si does not contain the word ti′ for i′ ̸= i. Therefore, |C(Ti)| = n+ 2 where n is the number of words.
We next show that Problem 3 given {Ti}i:ui∈V and the value z̃ = 2z+2|V |(n+1)z+2n+3 is equivalent to the max clique problem given the graph G and the number z.
For every ui, uj ∈ V with i < j, ui and uj are connected if and only if Si ∩ Sj = ∅; otherwise tji is in both Si and Sj . In addition, Si ̸⊆ Sj for i ̸= j because there is a special symbol ti in each Si. In other words, Si and Sj are compatible constituents if and only if ui and uj are connected in G. As a result, every constituency tree—constructed by single words, the whole-sentence constituent, and a subset of {Si}i:ui∈V —corresponds to a clique in G. We refer to such a tree as a clique tree.
We can further find the correspondence of the scores between Problem 3 and the max clique problem. For every clique tree T , we have
∑
i:ui∈V F1(T, Ti) =
∑
i:ui∈V
2(1[C(Ti) ⊆ C(T )] + n+ 1) |C(T )|+ n+ 2 (21)
= 2(|C(T )| − n− 1) + 2|V |(n+ 1)
|C(T )|+ n+ 2 (22)
where |C(T )| − n− 1 is the same as the number of vertices in the corresponding clique. This shows a clique tree T with a score of z̃ = 2z+2|V |(n+1)z+2n+3 corresponds to a clique with z vertices.
Now, let us consider using a solver for Problem 3 to solve the max clique problem. This is discussed by cases.
[Case 1] If the answer to Problem 3 is “no,” it means that there does not exist a tree (either a clique tree or a general tree) T satisfying ∑ i:ui∈V F1(T, Ti) ≥ z̃. Thus, there does not exist a clique Q satisfying |Q| ≥ z.
[Case 2] If the answer to Problem 3 is “yes,” there must exist a tree T satisfying the condition∑ i:ui∈V F1(T, Ti) ≥ z̃. Notice that such a tree is not guaranteed to be a clique tree, but in this case, we can show there must also exist a clique tree satisfying the condition.
In general, a tree is a clique tree if and only if it does not include any zero-hit constituents (zero hit means not appearing in any Ti). This is because, other than single words and the whole-sentence constituent, a clique tree selects a subset of {Si}i:ui∈V , whereas each Ti only selects one element of this
set. Let C0(T ) denote zero-hit constituents in T . We construct T ′ by removing C0(T ) from T . We have
z̃ ≤ ∑
i:ui∈V F1(T, Ti) (23)
= ∑
i:ui∈V
2 ∑
c∈C(T ) 1[c ∈ C(Ti)] |C(T )|+ |C(Ti)|
(24)
≤ ∑
i:ui∈V
2 ∑
c∈C(T )\C0(T ) 1[c ∈ C(Ti)] |C(T )| − |C0(T )|+ |C(Ti)|
(25)
= ∑
i:ui∈V
2 ∑
c∈C(T ′) 1[c ∈ C(Ti)] |C(T ′)|+ |C(Ti)|
(26)
= ∑
i:ui∈V F1(T
′, Ti) (27)
Here, Inequality (23) is due to the assumption of Case 2 that the answer to Problem 3 is “yes.” Inequality (25) is because |C0(T )| ≥ 0 and h(c) = 0 for c ∈ C0(T ), potentially increasing the denominator while the numerator is the same.
Since T ′ is a clique tree that satisfies ∑
i:ui∈V F1(T ′, Ti) ≥ z̃, we find the solution to the max clique
problem is also “yes.” The given reduction can be done in polynomial time, as it iterates over all vertices and vertex pairs. This concludes that Problem 3 belongs to NP-complete.
A.4 Proof of Lemma 1
Lemma 1. Given a graph G as described in §2.3, the clique Q∗ maximizing f(Q;α1, α2) corresponds to a constituency tree, if α1 ≤ Kα2 where K is the number of individuals.
Proof. A selection of constituents corresponds to a constituency tree over a sentence if and only if (1) it includes all the single words and the whole-sentence constituent, referred to as trivial constituents, and (2) all of its constituents are compatible with each other (already satisfied by the clique definition).
Trivial constituents are compatible with any possible constituent, and thus their corresponding vertices are connected to all vertices in G. Moreover, the hit count of such a constituent equals K, the number of individual trees, because trivial constituents appear in all constituency trees.
We assume by way of contradiction that a trivial constituent does not appear in the constituency tree corresponding to Q∗. This is equivalent to having a vertex u—connected to every vertex in G with w(u) = K, i.e., having a weight of K—not appearing in Q∗; that is,
f(Q∗ ∪ {u};α1, α2) < f(Q∗;α1, α2) (28) With the definition of f , we have
∑ v∈Q∗ w(v)+w(u)+α1 |Q∗|+ 1 + α2 < ∑ v∈Q∗ w(v) + α1 |Q∗|+ α2
(29) ∑
v∈Q∗ w(v)+w(u)+α1∑ v∈Q∗ w(v) + α1 < |Q∗|+ 1 + α2 |Q∗|+ α2
(30)
1 + w(u)∑
v∈Q∗ w(v) + α1 < 1 +
1
|Q∗|+ α2 (31)
w(u) < ∑ v∈Q∗ w(v) + α1 |Q∗|+ α2
(32)
w(u) < K|Q∗|+Kα2 |Q∗|+ α2 = K (33)
Here, the last inequality is due to α1 < Kα2 and w(v) ≤ K for every v. But this inequality contradicts our assumption that w(u) = K.
A.5 Proof of Theorem 5
Theorem 5 (Lower bound). For every constituent c ∈ C(T ∗)\P , where P ⊆ C(T ∗), we have
h(c) > min |P |≤j≤2n−2
∑j−|P | i=1 λ + i + ∑ c′∈P h(c
′) j + 2n− 1 (5)
where λ+i is the ith smallest positive hit count.
Proof. Consider any constituent c0 ∈ C(T ∗)\P . We construct T ′ by removing c0 from T ∗, i.e., C(T ∗) = C(T ′) ∪ {c0}. Based on Eqn. (2), we have
∑ c∈C(T ∗) h(c) |C(T ∗)|+ 2n− 1 > ∑ c∈C(T ′) h(c)
|C(T ′)|+ 2n− 1 (34)∑ c∈C(T ∗) h(c)∑ c∈C(T ′) h(c) > |C(T ∗)|+ 2n− 1 |C(T ′)|+ 2n− 1 (35)
1 + h(c0)∑
c∈C(T ′) h(c) > 1 +
1
|C(T ′)|+ 2n− 1 (36)
h(c0) >
∑ c∈C(T ′) h(c)
|C(T ′)|+ 2n− 1 (37)
Since the right-hand side is non-negative, we must have h(c0) > 0. In other words, zero-hit constituents do not appear in C(T ∗)\P . Thus, m-many constituents in C(T ∗)\P have a total of hit count greater than or equal to ∑m i=1 λ (+) i , because λ (+) i is the ith smallest positive hit count.
On the other hand, we have c0 /∈ P , implying that P ⊆ C(T ′). Since a constituent c ∈ C(T ′) is either in P or not, we can lower-bound the total hit count of T ′ by
∑
c∈C(T ′) h(c) ≥
∑ c′∈P h(c′) +
|C(T ′)\P |∑
i=1
λ+i (38)
Putting (38) into (37), we have
h(c0) > ∑|C(T ′)|−|P | i=1 λ + i + ∑ c′∈P h(c
′) |C(T ′)|+ 2n− 1 (39)
Moreover, P ⊆ C(T ′) also implies |P | ≤ |C(T ′)| = |C(T ∗)| − 1 ≤ 2n− 2 (40)
which derives (39) to a T ′-independent lower bound for h(c0) if we consider all the values |C(T ′)| can take, given by
h(c0) > min|P |≤j≤2n−2
∑j−|P | i=1 λ + i + ∑ c′∈P h(c
′) j + 2n− 1 (41)
concluding the proof.
The lower bound can be applied for pruning the search. We may immediately prune zero-hit constituents by setting P = ∅, and more importantly, it yields a much tighter lower bound together with Theorem 6.
A.6 Proof of Theorem 6
Theorem 6 (Upper bound). Let c be a constituent with a hit count of K, where K is the number of individuals. (a) The constituent c is compatible—i.e., may occur in the same constituency tree—with every constituent in the average tree. (b) The constituent c appears in the average tree.
Proof. [Part (a)] A hit count of K indicates that the constituent appears in all the individuals. Therefore, c is compatible with every constituent in every individual. On the other hand, Theorem 5 shows that every constituent in the average tree has a positive hit count (appears in at least one individual) and thus is compatible with c.
[Part (b)] We assume by way of contradiction that a constituent c0 with a hit count of K does not appear in the average tree T ∗. As shown in (a), c0 is compatible with every constituent in T ∗; hence, there exists a constituency tree T̂ such that C(T̂ ) = C(T ∗) ∪ {c0}. Based on Eqn. (2), we have
∑ c∈C(T̂ ) h(c)
|C(T̂ )|+ 2n− 1 <
∑ c∈C(T ∗) h(c)
|C(T ∗)|+ 2n− 1 (42)∑ c∈C(T̂ ) h(c)∑ c∈C(T ∗) h(c) < |C(T̂ )|+ 2n− 1 |C(T ∗)|+ 2n− 1 (43)
1 + h(c0)∑
c∈C(T ∗) h(c) < 1 +
1
|C(T ∗)|+ 2n− 1 (44)
h(c0) <
∑ c∈C(T ∗) h(c)
|C(T ∗)|+ 2n− 1 (45)
h(c0) < K|C(T ∗)|
|C(T ∗)|+ 2n− 1 ≤ K (46)
Here, the last inequality is due to h(c) ≤ K for every c. But this contradicts our assumption that h(c0) = K.

heading: B Our Meet-in-the-Middle Algorithm
text: The meet-in-the-middle technique (Horowitz and Sahni, 1974) has been used to tackle the max clique problem. This method halves the exponent of the time complexity of exhaustive search, cutting it down from O(2|V |) to O(2 |V |2 |V |2) given a graph with |V | vertices. However, the exact same algorithm is not sufficient for our needs, because Problem 4 is a generalization of the max clique problem and requires additional consideration. In this section, we develop an extended version of the algorithm that solves Problem 4 within the same time complexity.
Problem 4 (Normalized Max Weighted Clique). Consider a weighted undirected graph G = ⟨V,E⟩ and an objective function f(Q;α1, α2) = ∑ v∈Q w(v)+α1 |Q|+α2 , where α1, α2 ∈ R, Q ⊆ V , and w(v) is the weight of v. What is the clique Q that maximizes f(Q;α1, α2)?
To find the solution to Problem 4, we utilize the meet-in-the-middle technique, which splits the vertex set V into two parts V1 and V2, whose sizes are as equal as possible.
For every V ′1 ⊆ V1 and every 0 ≤ j ≤ |V ′1 |, we define
best(V ′1 , j) = argmax V ′′1 : V ′′ 1 ⊆V ′1 ,|V ′′1 |=j,
V ′′1 is a clique
∑
v∈V ′′1
w(v) (47)
which is the best j-vertex clique in V ′1 , and can be computed recursively. For initialization, we have best(V ′1 , 0) = ∅ for every V ′1 ⊆ V1 and best({u}, 1) = {u} for every u ∈ V1. The recursion is to compute the best set for any multi-vertex V ′1 assuming we have the best set for every subset of V1, whose size is smaller than V ′1 . To achieve this, we pick any u ∈ V ′1 , and the best(V ′1 , j) may or may not include u, which is discussed by cases.
If u ̸∈ best(V ′1 , j), we have best(V ′1 , j) = best(V ′ 1\{u}, j) (48)
If otherwise u ∈ best(V ′1 , j), then the other j − 1 vertices must be the neighbors of u in V ′1 , because
best(·, ·) must be a clique. Denoting by A(u) V ′1 all the vertices in V ′1 connected to u, we have
best(V ′1 , j) = {u} ∪ best(A(u)V ′1 , j − 1) (49)
Combining the two cases, we have best(V ′1 , j) be either Eqn. (48) or Eqn. (49), depending on which yields a higher weight.
For the V2 part, we would like to evaluate every clique V ′2 ⊆ V2 and its corresponding best match in V1 to obtain Q∗ ⊆ V = V1 ∪ V2 that maximizes our scoring function f(Q;α1, α2).
We enumerate all the subsets of V2. For every clique V ′2 ⊆ V2, we denote by A (V ′2) V1
the vertices in V1 that are connected to every vertex in V ′2 . We compute
val(V ′2 , j) = ∑
v ∈ V ′2 ∪ best ( A (V ′2) V1
,j ) w(v) (50)
being the weight of the best clique, which in V1 has j vertices and in V2 involves V ′2 only. Since the V = V1 ∪ V2, we can find the vertices of Q∗ in V1 and V2 separately:
Q∗2, j ∗ =argmax
V ′2 , j val(V ′2 , j) + α1 |V ′2 |+ j + α2
(51)
Q∗1 =best ( A (Q∗2) V1 , j∗ )
(52)
The final answer is Q∗ = Q∗1 ∪Q∗2. We analyze the time complexity of our algorithm. For the V1 part, our algorithm enumerates O(2 |V | 2 )- many subsets V ′1 and O(|V |)-many j values; in each iteration, O(|V |)-many vertices are visited for A (u) V ′1 . For the V2 part, we enumerate O(2 |V | 2 )-many subsets of V2, each verified for being a clique in O(|V |2) time; after that, A(V ′ 2)
V1 is retrieved with complexity O(|V1| · |V ′2 |) = O(|V |2), which can be absorbed in
the O-notation. Finally, retrieving Q∗ happens in O(2 |V |2 |V |) time, and the overall time complexity of our algorithm is O(2 |V |2 |V |2).

heading: C Performance by Constituency Types
text: We would like to analyze our ensemble method’s performance from a linguistic point of view. We provide a performance breakdown by constituency types, e.g., noun phrases (NP) and verb phrases (VP). Note that our unsupervised models offer unlabeled constituency structures, and thus we can only compute recall scores for different constituency types.
In Figures 8a and 8b, we report the recall scores for all constituents and discontinuous constituents, respectively. We can see for most of the types, the ensemble can retain the best individual’s performance in that type, suggesting that the ensemble is able to leverage the diverse strengths of different individuals.

heading: D Case Studies
text: In Figure 9, we present a case study on the LASSY dataset to show how the ensemble can take advantage of different individuals. In the figure, a groundtruth constituent is annotated by its type (e.g., CONJ representing conjunction). However, our unsupervised parsing is untyped, and we use “o” to denote a constituent. A skipped line “-|-” indicates discontinuity.
Consider the last three words “moeten worden opgesteld” (must be established).13 We see the ensemble output follows the majority votes (three out of five, namely Individuals 2, 4, and 5) and detects the correct structure of this phrase. Moreover, the ensemble is able to detect the discontinuous constituent “in sommige gevallen ... niet toereikend zijn” (in some cases ... not sufficient be), suggested by Individuals 4 and 5. On the other hand, we also notice that every individual produces incorrect discontinuous constituents, but our ensemble is able to smooth out such noise and achieve high F1 scores.
13We provide English interpretations by ChatGPT-4 with the prompt: “Translate the following Dutch sentence. Give word-by-word literal translation.”

heading: Future Work
text: We are happy to explore this direction as future work. For
example, a concurrent study of ours develops algorithms for ensemble-based text generation (Wen et al. , 2024). Second, our model is only tested on Dutch and German datasets. This is partially because of the established setups in previous work (Yang et al. , 2023) and the lack of annotated treebanks. Notice that English is usually excluded from the study of unsupervised discontinuous parsing, because English discontinuous structures are too rare for any model to discover. A potential future direction is multilingual linguistic structure discovery, perhaps, with ensembles of different languages. Third, our theoretical analysis leaves an open problem about averaging binary trees. However, pointing to open problems is usually considered a contribution (instead of a weakness) in theoretical computer science. Our theoretical analysis is also crucial to the understanding of our algorithms, because we now know that our proposed method works at the level of Open Problem 1 and can be easily extended to the level of Problem 3 following the technique proposed in Appendix A. 2