Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing (2024)

Abstract: We present the structured average intersectionover-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the groundtruth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).1

heading: 1 Introduction
text: Automatic constituency parsing of written text (Marcus et al., 1993, inter alia) and speech transcriptions (Godfrey and Holliman, 1993, inter alia), as representative tasks of automatic syntactic analysis, have been widely explored in the past few decades. Appropriate evaluation metrics have facilitated the comparison and benchmarking of different approaches: the PARSEVAL F1 score (Black et al., 1991; Sekine and Collins, 1997) has served as a reliable measure of text parsing across scenarios; for speech transcription parsing, the SPARSEVAL (Roark et al., 2006) metric extends PARSEVAL and accounts for speech recognition errors by allowing word-level editing with a cost.
1We open-source the code of STRUCT-IOU at https: //github.com/ExplorerFreda/struct-iou.
NP
PRP
Your
NN
turn
Forced Alignment
NP (2.56–3.01)
PRP (2.56–2.72)
Your
NN (2.72–3.01)
turn
(a) Ground-truth speech parse tree (right), obtained by forced alignment between the ground-truth text parse tree (left, top) and the spoken utterance (left, bottom).
VP (2.55–3.01)
VBP (2.55–2.56) NP (2.56-3.01)
PRP (2.56–2.72) NN (2.72–3.01)
(STRUCT-IOU = 0.75)
NP (2.51–3.10)
PRP (2.51–2.70) NN (2.70–3.10)
(STRUCT-IOU = 0.81)
(b) Predicted tree with good word boundaries and an errorful tree structure (left), or that with errorful word boundaries and a perfect tree structure (right).
Figure 1: Illustration of how STRUCT-IOU (§§ 4.1 and 4.2) evaluates textless speech constituency parsing. Best viewed in color, where nodes with the same color are aligned. Numbers in parentheses are the starting and ending times of the corresponding spans (in seconds).
Recent work (Lai et al., 2023; Tseng et al., 2023) has proposed a new task of textless speech constituency parsing. In contrast to earlier work that parses manually labeled (Charniak and Johnson, 2001, inter alia) or automatic (Kahn and Ostendorf, 2012, inter alia) speech transcriptions, these models construct constituency parse trees over automatically recognized spoken word boundaries, where each word is represented with a time range of the spoken utterance, without using any form of text. To evaluate these textless models, we need a metric that compares the predicted tree (over spoken word boundaries) with the manually labeled ground-truth tree (over written words) and faithfully reflects the parsing quality. Since the automatically recognized word boundaries may be imperfect, the metric should also reflect the changes
12320
in parsing quality due to word boundary errors. To the best of our knowledge, none of the existing metrics meets these requirements, as they are all designed to compare parse trees over discrete word sequences, instead of continuous time ranges.
Motivated by the need for textless speech parsing evaluation, in this paper, we introduce the structured average intersection-over-union ratio (STRUCT-IOU; Figure 1), a metric that compares two parse trees over time ranges. We relax the definition of segment trees (Bentley, 1977) to represent speech constituency parse trees, where each node is associated with an interval that represents the time range of the corresponding spoken word or constituent. To obtain the “ground-truth” speech parse trees, we use the forced alignment algorithm (McAuliffe et al., 2017), a supervised and highly accurate method that aligns written words to time ranges of the corresponding spoken utterance, to project the ground-truth text parses onto the time domain. STRUCT-IOU is calculated by aligning the same-label nodes in the predicted and ground-truth parse trees, following structured constraints that preserve parent-child relations. The calculation of STRUCT-IOU can be formulated as an optimization problem (§4.1) with a polynomial-time solution (§4.2) in terms of the number of tree nodes.
Although STRUCT-IOU is designed to evaluate speech parsing, it is also applicable to text parsing evaluation. We analyze STRUCT-IOU for both purposes: in speech parsing evaluation, STRUCTIOU robustly takes into account both the structure information and word boundaries; in text parsing evaluation, while maintaining a high correlation with the PARSEVAL F1 score, STRUCT-IOU shows a higher tolerance to potential syntactic ambiguity.

heading: 2 Related Work
text: Text constituency parsing and evaluation. In the past decades, there has been much effort in building and improving constituency parsing models (Collins and Koo, 2005; Charniak and Johnson, 2005; McClosky et al., 2006; Durrett and Klein, 2015; Cross and Huang, 2016; Dyer et al., 2016; Choe and Charniak, 2016; Stern et al., 2017; Kitaev and Klein, 2018, inter alia). PARSEVAL (Black et al., 1991) has been the standard evaluation metric for constituency parsing in most scenarios, which takes the ground truth and predicted trees and calculates the harmonic mean of precision and recall of labeled spans. For morphologically rich lan-
guages, TEDEVAL (Tsarfaty et al., 2012) extends PARSEVAL to accept multiple morphological analyses over sequences of words. A few alternative approaches have been pursued to address the potential mismatch in words and sentences (Calder, 1997; Jo et al., 2024). All these metrics are designed to evaluate parses over discrete word sequences, and cannot be easily extended to evaluate speech parses over continuous time ranges. Although our metric, STRUCT-IOU, is designed to evaluate speech constituency parsing, it can be easily extended for text parsing evaluation, reflecting a different aspect from existing metrics (§5.2).
Speech constituency parsing and its evaluation. Work on conversational speech parsing has focused on addressing the unique challenges posed by speech, including speech recognition errors (Kahn and Ostendorf, 2012; Marin and Ostendorf, 2014), unclear sentence boundaries (Kahn et al., 2004), disfluencies (Jamshid Lou and Johnson, 2020; Kahn et al., 2005; Lease and Johnson, 2006), as well as integrating prosodic features into the parsing systems (Tran et al., 2018; Tran and Ostendorf, 2021). On the evaluation side, the closest work to ours is SPARSEVAL (Roark et al., 2006), which extends PARSEVAL to account for speech recognition errors by allowing for word-level insertion, deletion, and substitution. In contrast, our metric STRUCT-IOU applies to the cases where no speech recognizer is applied or available.
Other structured evaluation metrics for parsing. There have been evaluation metrics of abstract meaning representations (AMRs; Cai and Knight, 2013), where two AMR graphs are matched by solving an NP-complete integer linear programming problem. While our work shares the spirit with theirs, we focus on the evaluation of speech constituency parsing over continuous word boundaries. There also exists a polynomial-time exact solution to our optimization problem.

heading: 3 Preliminaries
text: We use real-valued open intervals to represent speech spans for simplicity, although most of the following definitions and conclusions can be easily extended to closed intervals and half-open intervals. Proof of each corollary and proposition can be found in Appendix A.

heading: 3.1 Open Interval Operations
text: Definition 1. The length of a real-valued open interval I = (a, b), where a < b, is |I| = b− a. Definition 2. The intersection size of open intervals I1 and I2 is
I(I1, I2) = { 0 if I1 ∩ I2 = ∅ |I1 ∩ I2| otherwise.
Definition 3. The union size of open intervals I1 and I2 is U(I1, I2) = |I1|+ |I2| − I(I1, I2). Definition 4. The intersection over union (IOU) ratio between open intervals I1 and I2 is
IOU(I1, I2) = I(I1, I2) U(I1, I2) .
Throughout this paper, we will use IOU as the similarity metric between two intervals.

heading: 3.2 Relaxed Segment Trees
text: We relax the definition of a segment tree (Bentley, 1977) as follows to represent parse trees.
Definition 5. A node n of a relaxed segment tree is a triple n = ⟨In, Cn, ℓn⟩, where ℓn refers to the label of the node, and 1. In = (sn, en) is an open interval (i.e., segment)
associated with the node n, where sn < en; 2. Cn is a finite set of disjoint children nodes of
n: for any p, q ∈ Cn(p ̸= q), Ip ∩ Iq = ∅. Cn = ∅ if and only if n is a terminal node; 3. For a nonterminal node n, sn = minp∈Cn sp, and en = maxp∈Cn ep. Corollary 5.1. For nodes p,n, if p ∈ Cn, then Ip ⊆ In. Definition 6. Node p is an ancestor of node q if there exists a sequence of nodes n0, n1, . . . , nk(k ≥ 1) such that (i.) n0 = p, (ii.) nk = q, and (iii.) for any i ∈ [k],2 ni ∈ Cni−1 . Corollary 6.1. If node p is an ancestor of node q, then Ip ⊇ Iq.
2[k] = {1, 2, . . . , k}, where k ∈ N.
Definition 7. Node p is a descendant of node q if q is an ancestor of p.
Definition 8. A relaxed segment tree T = ⟨rT , NT ⟩ is a tuple, where 1. rT is the root node of T ; 2. NT = {rT } ∪ {n : n is a descendant of rT }
is a finite set of all nodes in T . Example 8.1. A constituency parse tree over spoken word time ranges (Figure 1a) can be represented by a relaxed segment tree.
Corollary 8.1. A relaxed segment tree can be uniquely characterized by its root node.
In the following content, we use T (n) to denote the relaxed segment tree rooted at n.
Proposition 9. For a relaxed segment tree T and p, q ∈ NT , p is neither an ancestor nor a descendant of q⇔ Ip ∩ Iq = ∅.

heading: 4 The STRUCT-IOU Metric

heading: 4.1 Problem Formulation
text: Given relaxed segment trees T1 and T2 with node sets NT1 = {n1,i} |NT1 | i=1 and NT2 = {n2,j} |NT2 | j=1 , we can align the trees by matching their same-label nodes. Let n1,i ↔ n2,j denote the matching between the nodes n1,i and n2,j .
Definition 10 (conflicted node matchings; Figure 2). The matchings n1,i ↔ n2,j and n1,k ↔ n2,ℓ are conflicted if any of the following conditions holds: 1. n1,i is an ancestor of n1,k, and n2,j is not an
ancestor of n2,ℓ; 2. n1,i is not an ancestor of n1,k, and n2,j is an
ancestor of n2,ℓ; 3. n1,i is a descendant of n1,k, and n2,j is not a
descendant of n2,ℓ; 4. n1,i is not a descendant of n1,k, and n2,j is a
descendant of n2,ℓ.
Intuitively, we would like the alignment to be consistent with the ancestor-descendant relationship between nodes.
The optimal (i.e., maximally IOU-weighted) structured alignment between T1 and T2 is given by the solution to the following problem: Problem 11 (maximally IOU-weighted alignment).

heading: A∗ = argmax
text: A
|NT1 |∑
i=1
|NT2 |∑
j=1
ai,j IOU ( In1,i , In2,j )
s.t. ∑
j ai,j ≤ 1(∀i ∈ [|NT1 |]), (1) ∑
i
ai,j ≤ 1(∀j ∈ [|NT2 |]), (2)
ai,j + ak,ℓ ≤ 1if n1,i ↔ n2,j and n1,k ↔ n2,ℓ are conflicted.
A ∈ {0, 1}|NT1 |×|NT2 | denotes an alignment matrix: ai,j = 1 indicates that the matching n1,i ↔ n2,j is selected, otherwise ai,j = 0. The last constraint of Problem 11 ensures that there are no conflicted matchings selected. Equations (1) and (2) imply one-to-one matching between nodes; that is, in a valid tree alignment, each node in T1 can be matched with at most one node in T2, and vice versa. The solution to Problem 11 gives the maximal possible sum of IOU over aligned node pairs. Definition 12. The structured average IOU (STRUCT-IOU) between T1 and T2 is given by
IOU(T1, T2)
= 1
|NT1 |+ |NT2 |
|NT1 |∑
i=1
|NT2 |∑
j=1
a∗i,j IOU ( In1,i , In2,j ) ,
where A∗ = {a∗i,j} is the solution to Problem 11.

heading: 4.2 Solution
text: We present a polynomial-time algorithm for the exact solution to Problem 11, by breaking it down into structured subproblems and solving them recursively with dynamic programming.
We define the subproblem as follows: given relaxed segment trees T1 and T2, we would like to find the maximum IOU weighted alignment of T1 and T2, where the roots of T1 and T2 are aligned. Without loss of generality, we assume that the root nodes of T1 and T2 are both indexed by 1. Formally, Problem 13 (maximum IOU weighted alignment, with root nodes aligned).
fT1,T2 = max A
|NT1 |∑
i=1
|NT2 |∑
j=1
ai,j IOU ( In1,i , In2,j )
s.t. a1,1 = 1;∑
j ai,j ≤ 1(∀i ∈ [|NT1 |]), ∑
i
ai,j ≤ 1(∀j ∈ [|NT2 |]),
ai,j + ak,ℓ ≤ 1if n1,i ↔ n2,j andn1,k ↔ n2,ℓ are conflicted,
where A ∈ {0, 1}|T1|×|T2| is the alignment matrix. While Problems 11 and 13 are not equivalent in principle, Problem 11 can be reduced to Problem 13 within O(1) time, by adding a dummy root node to each tree that associates with segments covering all the segments in both trees. We now present a polynomial-time solution to Problem 13. Definition 14. Given a node n of a relaxed segment tree, D = (n1,n2, . . . ,nk) is an ordered disjoint descendant sequence of n if 1. (ordered) for any i, j ∈ [k] and i < j, sni <
snj , where sni and snj are left endpoint of the associated intervals;
2. (disjoint) for any i, j ∈ [k] and i ̸= j, Ini ∩ Inj = ∅; 3. (descendant) for any i ∈ [k], ni is a descendant of n.
Corollary 14.1. In an ordered disjoint descendant sequence D = (n1,n2, . . . ,nk) of n, eni ≤ sni+1 for any i ∈ [k − 1].
The solution to Problem 13 is given by the following recursion:
fT1,T2 =IOU ( IrT1 , IrT2 ) +
max |D1|=|D2|
|D1|∑
i=1
fT (d1,i),T (d2,i), (3)
where rT1 and rT2 denote the root nodes of T1 and T2 respectively; | · | denotes the length of a sequence; D1 = (d1,1,d1,2, . . . ,d1,|D1|) and D2 = (d2,1,d2,2, . . . ,d2,|D2|) are same-length ordered disjoint descendant sequences of rT1 and rT2 respectively. Equation (3) can be computed within polynomial time, by solving a knapsack-style problem with dynamic programming. Specifically, let
g[T1, T2,e1, e2] =
max |De11 |=|D e2 2 |
|De11 |∑
j=1
fT (de11,j),T (d e2 2,j) ,
Algorithm 1 Polynomial time solution to Equation (3) Input: T1, T2 g[T1, T2, x, y]← 0,∀x, y g′[T1, T2, x, y] := maxx′<x,y′<y g[T1, T2, x′, y′] d1 ← the sequence of all descendants of rT1 , sorted in increasing order of right endpoint d2 ← the sequence of all descendants of rT2 , sorted in increasing order of right endpoint for i← 1 . . . , |d1| do
for j ← 1 . . . , |d2| do g[T1, T2, ed1,i , ed2,j ]← max(g[T1, T2, ed1,i , ed2,j ], fT (d1,i),T (d2,j) + g′[T1, T2, sd1,i , sd2,j ]) update g′ accordingly within O(1) time using bidimensional prefix sum
end for end for
Output: Equation (3) = g[T1, T2, erT1 , erT2 ]
where e1 and e2 are arbitrary scalars denoting the constraints of endpoints; De11 = (de11,1, . . . ,d
e1 1,|De11 | ) is an ordered disjoint descendant sequence of rT1 , where for any j ∈ [|De11 |], the right endpoint of the corresponding node ede11,j ≤ e1; similarly, D e2 2 = (de22,1,d e2 2,2, . . . ,d
e2 2,|De22 | ) is a disjoint descendant sequence of rT2 of which the right endpoint of each node does not exceed e2. Algorithm 1 computes g and Equation (3) within polynomial time, and therefore leads to a polynomial-time solution to Problem 13.
Complexity analysis. Suppose |T1| = n and |T2| = m. To compute fT1,T2 , all we need to compute is g[T ′1 , T ′2 , e′1, e′2] for all T ′1 , T ′2 , e′1 and e′2. Here, T ′1 and T ′2 enumerate over all subtrees of T1 and T2, respectively, and e′1 and e′2 enumerate over the endpoints of all nodes in both trees, respectively. The update process requires O(1) time for each T1, T2, e1, e2. The edge cases, i.e., g values of terminal nodes, can be directly computed in O(1) time, and therefore, the overall time complexity to solve Problem 13 is O(n2m2).

heading: 5 Experiments

heading: 5.1 Speech Constituency Parsing Evaluation
text: We use the NXT-Switchboard (NXT-SWBD; Calhoun et al., 2010) dataset to train and evaluate models, where the parser can access the forced alignment word boundaries in both training and testing stages. We train an off-the-shelf supervised constituency parsing model for speech transcriptions (Jamshid Lou and Johnson, 2020) on the training set of NXT-SWBD, do early-stopping using PARSEVAL F1 on the development set, and perform all
the analysis below on the development set. The model achieves F1 = 85.4 and STRUCT-IOU (averaged across sentences)3= 0.954 on the standard development set.

heading: 5.1.1 Comparison with PARSEVAL F1
text: Since the forced alignment word boundaries are accessible by the models, the PARSEVAL F1 metric can be directly calculated between the predicted speech constituency parse tree and the ground truth. We compare the values of STRUCT-IOU and PARSEVAL (Sekine and Collins (1997) implementation with default parameters) in the settings with forcedalignment word segmentation (Figure 3), and find a strong correlation between the two metrics.
3Unless otherwise specified, all STRUCT-IOU scores reported in the paper are computed by averaging across STRUCTIOU scores of individual sentences. We compare and discuss sentence-level and corpus-level STRUCT-IOU in §5.1.3

heading: 5.1.2 Analysis: STRUCT-IOU with Perturbed Word Boundaries
text: In textless speech parsing (Lai et al., 2023; Tseng et al., 2023), the word boundaries are unknown, and the boundaries predicted by the parser are usually imperfect. As a controlled simulation to such settings, we perturb the forced alignment word boundaries of the predicted parse tree (Figure 4), and calculate the STRUCT-IOU score between the perturbed parse tree and the ground truth over the original forced alignment word boundaries. Specifically, we suppose the word boundaries of a sentence with n words are B = b0, b1, . . . , bn,4 and consider the following types of perturbation with a hyperparameter δ ∈ [0, 1] controlling the perturbation level: • Noise-δ. We start with B(0) = B, and update
the boundaries iteratively as follows. For each i ∈ [n − 1], we randomly draw a number ri from the uniform distribution U(−δ, δ), and let b (i) i = b (i−1) i + |ri|∗ ( b (i−1) i+sgn(ri) − b(i−1)i )
, where sgn(·) : R→ {1,−1} denotes the sign function
sgn(x) = { 1 if x ≥ 0; −1 if x < 0.
For all j ̸= i and j ∈ [n], we let b(i)j remain the same as b(i−1)j . Finally, we take B(n−1) as the perturbed word boundaries for the predicted tree. • Insert-δ. We randomly draw a number ri from the uniform distribution for each boundary index i ∈ [n]. If ri < δ, we insert a word boundary at the position b′i, randomly drawn from the uniform distribution U(bi−1, bi), breaking the ith spoken word into two (i.e., [bi−1, b′i] and [b ′ i, bi]). • Delete-δ. Similarly to the insertion-based perturbation, we randomly draw a number ri from the uniform distribution U(0, 1) for each boundary index i ∈ [n − 1], and delete the boundary bi if ri < δ. Since such boundary deletion may break the predicted tree structure, we use the base model to re-predict the parse tree with the new word boundaries, where words concatenated by space are taken as the textual input (Jamshid Lou and Johnson, 2020).
A larger δ means a higher level of perturbation is applied, and we therefore expect a lower STRUCTIOU score; δ = 0 means no perturbation is applied, and the STRUCT-IOU score is the same as that
4We assume no silence between spoken words; if any interword silence exists, we remove it.
for the predicted parse trees with forced alignment word boundaries.
For each δ ∈ {0.1, 0.2, . . . , 1.0}, starting from the base model (for deletion-based perturbation) or its predicted parse trees (for noise and insertionbased perturbation), we run the perturbation 5 times and report both the mean and the standard deviation of the STRUCT-IOU result after perturbation.
Results and discussion. We present how the STRUCT-IOU value changes with respect to δ for different types of perturbation (Figure 5). The standard deviation is nearly invisible in the figure, showing that our metric is stable under a specific setting. For all three types of perturbation, as desired, a larger δ leads to a lower STRUCT-IOU score. Among the perturbation types, STRUCTIOU is the most sensitive to deletion, and the least sensitive to noise-based perturbation. Although the results are not comparable across perturbation types in the most rigorous sense, this reflects the fact that STRUCT-IOU, to some extent, is more sensitive to structural change of the trees than simple word boundary changes.
Although both word boundary insertion and deletion change the predicted tree structures, the former has less impact on the STRUCT-IOU scores. This also aligns with our expectation: boundary insertion only splits some of the spoken words into two and keeps the longer constituents; however, deletion may change significantly the tree structure, especially when it happens at the boundary of two
long constituents.

heading: 5.1.3 Corpus-Level vs. Sentence-Level Metric
text: Note that 39.7% utterances in the NXT-SWBD development set contain only one spoken word, and the STRUCT-IOU score of such sentences is always high—the metric degenerates to the IOU score between two intervals. Averaging the STRUCTIOU scores across all sentence pairs in the dataset may therefore overly emphasize these short utterances. To address this, we introduce the corpuslevel STRUCT-IOU score as an alternative, where Definition 12 is modified as follows:
Definition 15. The corpus-level STRUCT-IOU between two sets of parsed trees D1 = {T1,k} and D2 = {T2,k} is given by
IOU(D1,D2)
= ∑|D1| k=1 (|T1,k|+ |T2,k|) IOU(T1,k, T2,k)∑|D1|
k′=1 |T1,k′ |+ |T2,k′ | ,
where |D1| = |D2|, and a pair of T1,k and T2,k denotes the parse trees of the kth sentence in the corpus respectively.
We compare the corpus-level and sentence-level STRUCT-IOU scores (Figure 6). As desired, the corpus-level STRUCT-IOU score has lower absolute values than the sentence-level one, and the difference is more significant when longer sentences are considered. A similar phenomenon has been found in text constituency parsing (Kim et al., 2019) as well, where corpus-level PARSEVAL F1 scores are lower than sentence-level ones.

heading: 5.2 Evaluation of English Text Constituency Parsing
text: We extend our experiment to the evaluation of text constituency parsing. In this part, we suppose every written word corresponds to a unit-length segment— analogously, this can be considered as speech parsing with evenly distributed word boundaries, for both predicted and ground-truth trees.

heading: 5.2.1 Correlation with PARSEVAL F1 Scores on the Penn Treebank
text: We use the Penn Treebank (PTB; Marcus et al., 1993) dataset to train and evaluate Benepar (Kitaev and Klein, 2018), a state-of-the-art text constituency parsing model, doing early-stopping using labeled PARSEVAL F1 on the development set. The base model achieves PARSEVAL F1 = 94.4 and STRUCT-IOU (averaged across sentences) = 0.962 on the standard development set.
We compare the STRUCT-IOU scores with the PARSEVAL F1 scores on the development set (Figure 7). As in the speech parsing experiment, we find a strong correlation between the two metrics, showing that STRUCT-IOU is consistent with the existing metric in the text parsing domain.

heading: 5.2.2 STRUCT-IOU vs. PARSEVAL F1 on Syntactically Ambiguous Sentences
text: We consider a special setting of parsing syntactically ambiguous sentences, where the syntactically plausible parse tree of a sentence may not be unique (see examples in Figure 8). We simplify the case shown in Figure 8 using synthetic sentences with syntactic ambiguity with the template N (P
N){n}, where P denotes a preposition and N denotes a noun, and n determines how many times the P N pattern is repeated. For N (P N){2}, the two potential parse trees are shown in Figure 9.
We compare PARSEVAL and STRUCT-IOU in the following scenarios, choosing a random syntactically plausible parse tree as the ground truth: • Ground truth vs. random parse trees, where
the random parse trees are constructed by recursively combining random consecutive words (or word groups) into a binary tree. We construct 100 random parse trees and report the average. • Ground truth vs. syntactically plausible parse trees, where we report the lowest possible score between the ground truth and other syntactically plausible trees.
As shown in Table 1, the lowest possible PARSEVAL F1 score between the ground truth and another syntactically plausible tree is significantly lower than the score achieved by meaningless random trees; however, STRUCT-IOU consistently assigns higher scores to the syntactically plausible parses, showing a higher tolerance to syntactic ambiguity.

heading: 5.3 Evaluation of Hebrew Text Constituency Parsing
text: We extend our experiment to the evaluation of text constituency parsing on Hebrew, a morphologically rich language that allows different tokenizations of the same sentence. In this subsection, we suppose every written character (instead of a word for English; §5.2) corresponds to a unitlength segment. We evaluate STRUCT-IOU by running a pre-trained Hebrew constituency parsing
model (benepar_he2; Kitaev et al., 2019) on the SPMRL 2013 Hebrew development set (Seddah et al., 2013). Similarly to the English text parsing evaluation result (§5.2), we obtain a high Spearman rank correlation coefficient of 0.823 (over 10- sentence buckets) between STRUCT-IOU (0.959 averaged across sentences) and PARSEVAL F1 scores measured by EVALB-SPMRL (93.3).
In addition, we demonstrate that STRUCT-IOU naturally provides a metric that supports misaligned morphological analysis. The default tokenization in the SPMRL dataset does not extract the plural morphemes !תו and !Mי; therefore, simply extracting the plural morphemes forms another acceptable tokenization strategy (see Figure 10 for an example). We break the nouns ending with these two morphemes and feed the new tokenization to
the benepar_he2 model.5 The prediction with our new tokenization receives a STRUCT-IOU of 0.907 against the ground-truth—as desired, it is lower than 0.959 with the ground-truth tokenization. However, the STRUCT-IOU score remains high, reflecting the facts that (1) the manipulation introduces misalignment between parses, and (2) the Benepar model is fairly robust to such mismatch on tokenization (see footnote 5). In contrast to TEDEVAL (Tsarfaty et al., 2012), which treats all mismatched nodes as errors with the same penalization in the final metric, STRUCT-IOU offers an alternative approach for evaluating parsing quality under misaligned morphological analyses, assigning partial credit to aligned same-label nodes with IOU > 0.

heading: 6 Conclusion and Discussion
text: In this paper, we present STRUCT-IOU, the first metric that computes the similarity between two parse trees over continuous spoken word boundaries. STRUCT-IOU enables the evaluation of textless speech parsing (Lai et al., 2023; Tseng et al., 2023), where no text or speech recognizer is used or available to parse spoken utterances.
In the canonical text and speech parsing settings, STRUCT-IOU complements the existing evaluation metrics (Black et al., 1991; Roark et al., 2006; Tsarfaty et al., 2012). Even for the evaluation of English constituency parsing, STRUCT-IOU shows a higher tolerance to potential syntactic ambiguity under certain scenarios, providing an alternative interpretation of the parsing quality.
5The benepar_he2 model is not trained on this tokenization; however, we expect the model to work reasonably well, since it uses XLM-R (Conneau et al., 2020) as the word embeddings, which provides syntactic information of the new tokenization.
Faithful evaluation of parsing quality is crucial for developing both speech and text parsing models. In supervised parsing, it has been common sense that higher evaluation metric scores (i.e., PARSEVAL F1) imply better models. However, the misalignment between linguistically annotated ground truths and model predictions, especially unsupervised parsing model predictions, does not necessarily indicate poor parsing quality of the models (Shi et al., 2020)—instead, the models may have learned different but equally valid structures. Conversely, annotations made by linguistic experts (such as the Penn Treebank) may exhibit discrepancies when compared to the responses of native speakers who lack formal linguistic training. We suggest that future work investigate what properties of the parses are emphasized by each evaluation metric, and consider multi-dimensional evaluation metrics (Kasai et al., 2022, inter alia).

heading: Acknowledgement
text: We thank Yudong Li, Jiayuan Mao, and the anonymous reviewers for their valuable suggestions. This work is supported in part by a Google Ph.D. Fellowship to FS.
Limitations
STRUCT-IOU is designed to evaluate constituency parse trees over continuous spoken word boundaries, and is not directly applicable to evaluate other types of parses, such as dependency parse trees; however, it may be extended to evaluate other types of parse trees by modifying the alignment constraints. We leave the extension of STRUCTIOU to other types of parses as future work. We do not foresee any risk associated with the use of STRUCT-IOU beyond the inherent minimal risks encountered in computer science research.

heading: A Proof of Corollaries and Propositions
text: We present the proof of the corollaries mentioned in the main content as follows. Corollary 5.1 For nodes p,n, if p ∈ Cn, then Ip ⊆ In.
Proof. According to the definition of open intervals and Definition 5 (3),
an ≤ ap < bp ≤ bn ⇒Ip = (ap, bp) ⊆ (an, bn) = In.
Corollary 6.1 If node p is an ancestor of node q, then Ip ⊇ Iq.
Proof. According to Definition 6, there exists a sequence of nodes n0,n1, . . . ,nk(k ≥ 1) such that (1) n0 = p, (2) nk = q and (3) for any i ∈ [k], ni ∈ Cni−1 .
Corollary 5.1 implies that for any i ∈ [k], Ini−1 ⊇ Ini ⇒ In0 ⊇ Ink ⇒ Ip ⊇ Iq.
Corollary 8.1 A relaxed segment tree can be uniquely characterized by its root node.
Proof. (⇒) Definition 8 implies that each relaxed segment tree has one root node. (⇐) Given a specific node n, we have the unique setN = {n}∪{n′ : n′ is a descendant of n}, and therefore extract the set of all nodes in the relaxed segment tree rooted at n.
Proposition 9 For a relaxed segment tree T and p, q ∈ NT , p is neither an ancestor nor a descendant of q⇔ Ip ∩ Iq = ∅.
Proof. (⇒) Let z denote the least common ancestor of p and q. There exists p′, q′ ∈ Cz(p′ ̸= q′) such that Ip′ ⊇ Ip and Iq′ ⊇ Iq; therefore
Ip ∩ Iq ⊆ Ip′ ∩ Iq′ Definition 5 (2) = ∅ ⇒ Ip ∩ Iq = ∅.
(⇐) If Ip ∩ Iq = ∅, according to Definition 5 (3) and Definition 6, p is not an ancestor of q and vice versa.
Corollary 14.1 Given an ordered disjoint descendant sequence S = (n1,n2, . . . ,nk) of n, for any i ∈ [k − 1], bni ≤ ani+1 .
Proof. If there exists i ∈ [k − 1] such that bni > ani+1 , then
Ini ∩ Ini+1 =(ani , bni) ∩ (ani+1 , bni+1) = { x : max(ani , ani+1) < x < min(bni , bni+1) } = { x : ani+1 < x < min(bni , bni+1) }
(Definition 14 (1)).
Since bni+1 > ani+1 (definition of open intervals),
ani+1 < min(bni , bni+1)⇒ Ini ∩ Ini+1 ̸= ∅.
This conflicts with Definition 14 (2).

heading: Future Work
text: We leave the extension of STRUCTIOU to other types of parses as future work. We do not foresee any risk associated with the use of STRUCT-IOU beyond the inherent minimal risks encountered in computer science research