Unsupervised Parsing by Searching for Frequent Word Sequences among Sentences with Equivalent Predicate-Argument Structures (2024)

Abstract: Unsupervised constituency parsing focuses on identifying word sequences that form a syntactic unit (i.e., constituents) in target sentences. Linguists identify the constituent by evaluating a set of Predicate-Argument Structure (PAS) equivalent sentences where we find the constituent appears more frequently than non-constituents (i.e., the constituent corresponds to a frequent word sequence within the sentence set). However, such frequency information is unavailable in previous parsing methods that identify the constituent by observing sentences with diverse PAS. In this study, we empirically show that constituents correspond to frequent word sequences in the PAS-equivalent sentence set. We propose a frequency-based parser span-overlap that (1) computes the span-overlap score as the word sequence’s frequency in the PASequivalent sentence set and (2) identifies the constituent structure by finding a constituent tree with the maximum span-overlap score. The parser achieves state-of-the-art level parsing accuracy, outperforming existing unsupervised parsers in eight out of ten languages. Additionally, we discover a multilingual phenomenon: participant-denoting constituents tend to have higher span-overlap scores than equal-length event-denoting constituents, meaning that the former tend to appear more frequently in the PAS-equivalent sentence set than the latter. The phenomenon indicates a statistical difference between the two constituent types, laying the foundation for future labeled unsupervised parsing research.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 3760–3772 August 11-16, 2024 ©2024 Association for Computational Linguistics
Unsupervised constituency parsing focuses on identifying word sequences that form a syntactic unit (i.e., constituents) in target sentences. Linguists identify the constituent by evaluating a set of Predicate-Argument Structure (PAS) equivalent sentences where we find the constituent appears more frequently than non-constituents (i.e., the constituent corresponds to a frequent word sequence within the sentence set). However, such frequency information is unavailable in previous parsing methods that identify the constituent by observing sentences with diverse PAS. In this study, we empirically show that constituents correspond to frequent word sequences in the PAS-equivalent sentence set. We propose a frequency-based parser span-overlap that (1) computes the span-overlap score as the word sequence’s frequency in the PASequivalent sentence set and (2) identifies the constituent structure by finding a constituent tree with the maximum span-overlap score. The parser achieves state-of-the-art level parsing accuracy, outperforming existing unsupervised parsers in eight out of ten languages. Additionally, we discover a multilingual phenomenon: participant-denoting constituents tend to have higher span-overlap scores than equal-length event-denoting constituents, meaning that the former tend to appear more frequently in the PAS-equivalent sentence set than the latter. The phenomenon indicates a statistical difference between the two constituent types, laying the foundation for future labeled unsupervised parsing research.

heading: 1 Introduction
text: Unsupervised constituency parsing focuses on identifying the constituent structure for target sentences. The constituent structure consists of a set of
We release the code and data at https://github.com/ junjiechen-chris/ACL24-SpanOverlap
syntactically self-contained word sequences, commonly known as constituents. In addition to being syntactic units, those constituents encode semantic units in the sentence’s semantic structure (Carnie, 2007; Fasold and Connor-Linton, 2014). Among all possible word sequences that can arise in natural language, the constituent is one of the few word sequences that can describe the semantic unit. As a result, we would have a higher chance of repeatedly observing the constituent than random word sequences given the semantic unit. In other words, we can observe the constituent as a frequent word sequence among semantically equivalent sentences. We refer to the frequent word sequence as a word sequence pattern.
In this paper, we empirically show a hypothesis: constituents correspond to frequent word sequences in the set of sentences with equivalent Predicate-Argument Structures (PAS). PAS ab-
3760
stracts the sentence semantics as an event with three elements: (1) predicates of the event; (2) participants of the event (i.e., arguments); (3) the semantic relationship between the predicates and arguments (i.e., thematic relations) (Palmer et al., 2005) (Figure 1). We refer to two sentences as PredicateArgument Structure (PAS) equivalent when they have the same PAS elements. Controlling the PAS provides a good approximation to controlling the semantic structure. Figure 1 exemplifies our hypothesis with a target sentence, “John ate an apple this morning.” Word sequences like “an apple” are constituents in the target sentence, whereas word sequences like “John ate” are non-constituents. Compared to the non-constituent, the constituent appears more frequently in the target sentence’s PAS-equivalent sentence set, manifesting itself as a word sequence pattern.
However, such frequency information was not available to previous unsupervised parsing methods. Those previous methods focused on uncovering the constituent structure by either inducing a complex grammar (Kim et al., 2019; Yang et al., 2021) or evaluating whether a word sequence can withstand syntactic transformations (Cao et al., 2020; Wu et al., 2020; Li and Lu, 2023). Both methods operate upon sentences with diverse PAS, as shown in the upper half of Table 1. These sentences provide little information about the constituent structure, as evidenced by the low frequency of word sequences corresponding to constituents. Linguistic researchers perform constituency tests (e.g., movement) (Carnie, 2007) using handcrafted sentences with equivalent PAS, as shown in the lower half of Table 1. These PAS-equivalent sentences, as discussed above, provide significantly more information about the constituent structure, as evidenced by the high frequency of word sequences corresponding to constituents. The gap in data characteristics motivates our research on unsupervised parsing using PAS-equivalent sentences.
We apply the hypothesis to unsupervised parsing and propose a frequency-based method, spanoverlap, to exploit the frequency information. The span-overlap method (1) generates a PASequivalent sentence set using a Large Language Model (LLM), (2) computes the span-overlap score for word sequences as the sequence’s frequency in the sentence set, and (3) identifies the constituent structure by finding the constituent tree with maximum span-overlap scores. The span-overlap parser obtains higher parsing accuracy than state-of-theart unsupervised parsers in eight out of ten languages evaluated and by a large margin. The span-overlap score can effectively separate constituents from non-constituents, especially for long constituents. These results highlight the utility and applicability of the frequency information to unsupervised parsing. In addition to the hypothesis, we discover a multilingual phenomenon: participantdenoting constituents tend to have higher spanoverlap scores than equal-length event-denoting constituents, meaning that the former tend to appear more frequently in the PAS-equivalent sentence set than the latter. The phenomenon indicates a statistical difference between the two constituent types, laying the foundation for labeled unsupervised parsing research.

heading: 2 Background
text: Unsupervised constituency parsers seek to identify unlabeled constituents for any given sentence. A constituent in a sentence w := (w[1], ..., w[n]) is a consecutive word sequence wi,j := (w[i], ..., w[j]) that functions as an independent unit in the structure. The word sequence can be represented as a two-tuple (i, j), referred to as a span. We measure the parsing accuracy by the F1 score between the set of predicted spans and the set of gold spans (Klein and Manning, 2002).
Unsupervised parsers make predictions by (1) evaluating a span score for each word sequence and (2) finding a tree optimizing the span score. Grammar-based unsupervised parsers (Yang et al., 2021; Zhao and Titov, 2020) compute the span score as the posterior span probability pG((i, j)|w) with respect to a probabilistic grammar G (i.e., how likely the grammar selects (i, j) as a constituent). The grammar-based parsers predict constituents by maximizing the span score, seeking to maximize the probability of the prediction containing constituents. Language model-based parsers (Cao
et al., 2020; Wu et al., 2020; Li and Lu, 2023) compute the span score by whether the language model reacts to syntactic transformations involving the span. The language model-based parsers predict constituents by minimizing the span score, searching for constituents most independent of the remainder of the sentence.
The language model-based unsupervised parsing methods have attracted much attention lately due to the language model’s applicability in various syntactic (Hu et al., 2020; Futrell et al., 2019) and semantic tasks (Huang and Chang, 2023). Given a word sequence wi,j , the method evaluates the span score as the distance d between w (the original sentence) and f(w, (i, j)) (the transformed sentence designed to test the constituency of wi,j), as shown in Equation 1. Cao et al. (2020) measures the distance by the change in the acceptability between w and f(w, (i, j)) using an external acceptability model trained with synthetic data. Wu et al. (2020) and Li and Lu (2023) measure the difference by the distance in language model embeddings between w and f(w, (i, j)).
score(w, (i, j)) := d(w, f(w, (i, j))) (1)
Recent advances in Large Language Models (LLM) have led to a surge in the application of LLM-enhanced data to Natural Language Processing systems (Yang et al., 2023; Perez et al., 2023; Liu et al., 2022). Given an instruction (typically in the form of prompts), the LLM iteratively generates its answer based on the instruction and partially generated answers. Recent studies (Zhou et al., 2023; Chia et al., 2023) point out that state-of-theart LLMs can follow difficult instructions and generate the correct answer with over 90% accuracy. This excellent instruction-following capability enables us to sample PAS-equivalent sentences from the LLM.

heading: 3 Related Works
text: Grammar-based unsupervised parsers (Klein and Manning, 2002; Kim et al., 2019; Zhao and Titov, 2020) learn probabilistic grammar by maximizing the likelihood of sentences observed in a corpus. However, these sentences have diverse PAS to the degree that constituent word sequences can only be observed a limited number of times. The limited observation provides insufficient positive evidence on whether a word sequence is a constituent through the sequence’s frequency. Also, the grammar has to reject non-constituent but frequent word
sequences tied to certain expressions. For example, the word sequence “it is” is a non-constituent but frequent word sequence in sentences like “(it (is (a book)))”. The lack of positive evidence and the abundance of false-positive evidence make up the main challenges in unsupervised constituency parsing. The two issues likely contribute to the low performance of the grammar-based method in non-English languages (see Table 3 and Table 4). Our span-overlap method is immune to the two issues. Our method detects the constituent-related word sequence pattern, utilizing the positive evidence revealed in the PAS-equivalent sentences. The PAS-equivalent sentences express the same PAS in diverse ways, effectively suppressing the frequency of the expression-tied non-constituent word sequences (i.e., false-positive evidence). Our method achieves significantly higher parsing performance than the grammar-based methods in nonEnglish languages.
Previous language model-based methods conduct constituency tests by applying handcrafted templates to word sequences. However, different languages implement the syntactic transformation differently, requiring templates tailored to each language. To circumvent the problem, Cao et al. (2020) limit the scope of languages to English, which results in an English-only parser. Wu et al. (2020) and Li and Lu (2023) limit the transformation to movement and standalone tests only, which limits their parser’s performance. By leveraging LLMs for performing the transformation, our span-overlap method can cover a diverse set of transformations while being applicable to many languages. Furthermore, the previous method has to perform the constituency test for every word sequence, incurring O(n2) language model evaluations for sentences with n words. Our method only generates a few dozen acceptable PAS-equivalent sentences for each target sentence, incurring only O(1) evaluations of the LLM. The low complexity allows for more efficient language model-based parsing. For example, our method takes 14 minutes to parse the development set of Penn Treebank (Marcus, Mitchell P. et al., 1999) whereas Li and Lu (2023)’s method takes about an hour.
Researchers have long pointed out that language models capture shallow syntax, such as subjectverb agreements (Hu et al., 2020) and phrase segmentations (Luo et al., 2019; Downey et al., 2022). However, the shallow syntax covers only a small subset of information that hierarchical constituent
structures provide. For example, knowing “a beautiful New York City” as an NP does not provide any information about the NP’s internal structure (i.e., “(a (beautiful ((New York) City)))”). Language model-based methods utilize the language model’s ability to identify shallow syntax for reconstructing the hierarchical constituent structure. Our span-overlap method represents an improvement over previous language model-based methods, as discussed in the above paragraph. As we will see in Section 5, both our span-overlap parser and the baseline language model-based parser outperform the GPT-3.5 parser in all languages. This result indicates that LLM alone cannot recover the full constituent structure, prompting further research on language model-based unsupervised parsing.

heading: 4 Method
text: This section introduces the proposed span-overlap method (Figure 2). Given a target sentence w, the span-overlap method performs parsing in four steps: (1) Generating a set of sentences s = {s1, s2, ...} where all sk are PAS-equivalent to w; (2) Normalizing words in w and sk ∈ s; (3) Mea-
suring the frequency of all word sequences of w within the sentence set s. We denote each word sequence that extends from the i-th word to the j-th word as wi,j . We refer to the frequency of wi,j within s as the span-overlap score for the span (i, j); (4) Performing maximum tree decoding over the span-overlap score.
Firstly, we generate a PAS-equivalent sentence set s for w using an instruction-following LLM (e.g., GPT-3.5). We instruct the LLM with six types of PAS-preserving instructions listed in Table 2. The movement instruction contains four sub-instructions common in most languages (topicalization, preposition, postposition, and extraposition). The instruction models that constituents can be moved around as an independent unit. The question conversion instruction asks the LLM to convert w to a yes-no question or a wh-question. The tense conversion instruction asks the LLM to convert w to another tense. The instruction models the closer relationship between the verb and object via adding or removing auxiliaries between the verb and subject. The clefting instruction asks the LLM to construct a cleft sentence based on w, modeling subject and object extractions in embedded clauses. The passivization asks the LLM to convert w from an active voice to a passive voice. The language-specific instruction contains three sub-instructions: scrambling, ba-construction, and heavy NP shift. We present the instruction configuration in Appendix A.2.
Secondly, we normalize words in w and sk ∈ s to reduce incorrect word sequence mismatches due to unmatching inflectional suffixes or misaligned tokenization settings. We perform stemming and contraction normalization in this step. The stemming removes the inflectional suffixes in a word (e.g., working → work). The contraction normalization expands one contracted word to two separate words (e.g., didn’t → did n’t), aligning with the treebank’s tokenization.
Thirdly, we compute the span-overlap score as
the frequency of the normalized wi,j within the normalized sentence set s (Equation 2). The spanoverlap score is an exact matching score, requiring wi,j to be a substring of sentences sk ∈ s. We also present results in appendix A.1 for a span-overlap variant using fuzzy matching scores.
score(w, (i, j)) :=
∑ sk∈s 1(w i,j is a substring in sk)
|s| (2)
Lastly, we decode the constituent structure by finding a tree with maximum span-overlap scores. We apply the Viterbi algorithm (Cohen et al., 1971) for this purpose.

heading: 5 Experiment

heading: 5.1 Experimental Settings
text: We conduct experiments in 10 languages, namely English (Penn Treebank (PTB)) (Marcus, Mitchell P. et al., 1999), Chinese (Penn Chinese Treebank 5.1 (CTB)) (Palmer, Martha et al., 2005)1, German, French, Hungarian, Swedish, Polish, Basque, Hebrew, and Korean (SPMRL) (Seddah et al., 2013). We apply the same training/development/testing split as Li and Lu (2023). We use the development section for analysis and the testing section for comparison with existing unsupervised parsers. As per previous research (Shen et al., 2018; Kim et al., 2019), we evaluate the parsing accuracy using the unlabeled sentence-F1 (SF1) score on sentences of length ≥ 2. We drop punctuation (e.g., comma and period) and trivial spans (i.e., spans containing a single word and the sentence level span) when computing the SF1 score.
We generate PAS-equivalent sentences using an instruction-following LLM GPT-3.5-turbo-0125. We apply English instructions for all languages but add a specific directive asking the LLM to generate native samples for non-English languages. We manually examined 100 samples in English and Chinese and found that most samples had PAS identical to or similar to the target sentence.2 We apply the snowball-stemmer3 for English, French, German, Basque, Hungarian, and Swedish, and the stempel-stemmer4 for Polish. We skip the stemming process for Chinese, Korean, and Hebrew. Chinese words do not take inflections. Hebrew
1The PTB and CTB datasets are available under the LDC User Agreement for Non-Members license.
2More details in Appendix A.3 3Available as a part of the NLTK package. 4Available as a part of the egothor project.
and Korean do not have a readily available stemmer. We normalize contractions only in English and French because of the common use of contractions in these languages. While not implementing the full normalization may harm the parsing accuracy in many languages, the span-overlap parser achieves state-of-the-art level performance without the full normalization in those languages.
We analyze the span-overlap score for two types of constituents: participant-denoting and eventdenoting constituents. The span-overlap score measures the word sequence frequency within the PASequivalent sentence set. The objective is to identify how the language model behaves when generating constituents of different characteristics. We refer to a constituent as participant-denoting when it denotes a participant in the PAS. Likewise, we refer to a constituent as event-denoting when it denotes an event in the PAS. For example, Noun Phrases (NP) and Prepositional Phrases (PP) are common participant-denoting constituents, whereas Verb Phrases (VP) and Inflectional Phrases (IP) are common event-denoting constituents. We compare the span-overlap score for phrase types with more than twenty occurrences. We normalize the phrase type annotation for three languages (French, Polish, and Basque) whose annotation does not align with the PTB’s annotation. In French, we merge the Srel, Sint, and Ssub phrases into S, and the VPpart and VPinf into VP. In Polish, we treat the zdanie phrase as S, the fwe as VP, and the fno as NP (Woliński et al., 2018). In Basque, we treat the SN phrase as NP and the SP phrase as PP (de Ilarraza Sánchez et al., 2008).

heading: 5.2 Comparison with the State-of-the-Art
text: Table 3 and Table 4 showcase the effectiveness and applicability of the frequency information revealed in the PAS-equivalent sentence set. The two tables compare the span-overlap parser with three PCFG-based parsers (N-PCFG, C-PCFG (Kim et al., 2019), and TN-PCFG (Yang et al., 2021)) and the latest language model-based parser (MLMIP (Li and Lu, 2023)). We also include a GPT-3.5 parser that predicts PTB style parse tree in its serialized form (e.g., “(S (NP (DT The) (NN value)) (VP (VBD was) (VP (VBN disclosed))))”). We drop trivial and duplicate spans when computing the unlabelled SF1 score for the GPT-3.5 parser.
The span-overlap parser represents a significant improvement over previous unsupervised parsers, outperforming the best baseline parser in eight out
of ten languages. The span-overlap parser outperforms the MLM-IP parser by 3.99 SF1, 10.9 SF1, and 4.32 mean SF1 scores in the English PTB, the Chinese CTB, and the multilingual SPMRL dataset, respectively. It respectively outperforms the best grammar-based parser (TN-PCFG) by 9.5 SF1 and 9.31 mean SF1 scores in the Chinese CTB and the multilingual SPMRL dataset but underperforms the TN-PCFG parser by 4.7 SF1 score in the English PTB dataset. The high parsing accuracy establishes the span-overlap parser as a simple but competitive multilingual unsupervised parser.
The strong parsing performance is not solely due to the application of the GPT-3.5 model. As shown in the table, the GPT-3.5 parser is the worstperforming baseline parser in the PTB, CTB, and SPMRL dataset, achieving 36.2 SF1 in the PTB, 5.1 SF1 in the CTB, and 23.2 mean SF1 score in the SPMRL dataset. The low parsing accuracy indicates that the GPT-3.5 model can only recover a small portion of the constituent structure. In contrast, the proposed span-overlap parser uncovers significantly more constituent structures, outperforming the GPT-3.5 parser by 16.7 SF1 in the PTB, by 43.6 SF1 in the CTB, and by 26.86 mean SF1 score in the SPMRL dataset.

heading: 5.3 Participant-denoting Constituents Tend to Have Higher Span-overlap Scores than Equal-length Event-denoting Constituents
text: In this section, we demonstrate a statistical difference between equal-length participant-denoting
and event-denoting constituents through the spanoverlap score. We focus on four participantsdenoting constituents (i.e., Noun Phrases (NP), Prepositional Phrases (PP), Adpositional Phrases (AP), and Quantifier Phrases (QP)) and four eventdenoting constituents (i.e., Declarative Clauses (S), Verb Phrases (VP), Inflectional Phrases (IP), and Complementizer Phrases (CP)). We also include the span-overlap score for random word sequences with the same sequence length statistics as gold constituents (i.e., For each random sequence, there is a gold constituent with the same sequence length).
We begin by observing the difference between the two constituent types regardless of the constituent’s sequence length. Figure 3 shows the mean span-overlap score along with the score distribution for NP, PP, QP, S, and VP in the English PTB dataset. The AP, IP, and CP are not annotated in the dataset and, therefore, cannot be shown in the figure. The three participant-denoting constituents (i.e., NP, PP, and QP) have higher mean scores than the two event-denoting constituents (i.e., S and VP). The distribution figure shows that most participant-denoting constituents have high scores, whereas most event-denoting constituents do not. We can characterize this feature using the skewness value of the score distribution (Russo, 2020). A lower skewness value indicates more constituents with high scores than those with low scores. As shown in the figure, the participant-denoting constituents have a negative skewness value, whereas the event-denoting constituents have a positive skewness value. This skewness value indicates that participant-denoting constituents tend to appear more frequently in the PAS-equivalent sentence set than event-denoting constituents. The comparison establishes the use of the mean score and skewness value as a metric to reflect the statistical difference between the two constituent types.
Figure 4 demonstrates the statistical difference between equal-length participant-denoting and event-denoting constituents in English and Chinese. Comparing constituents of different lengths, we see
that short constituents tend to have higher scores than long constituents. The correlation between span-overlap scores and sequence lengths prompts whether the statistical difference is due to the length difference. This is unlikely the case. As shown in the figure, participant-denoting constituents have higher mean span-overlap scores and lower skewness values than equal-length event-denoting constituents over all sequence lengths. In other words, participant-denoting constituents tend to appear more frequently in the PAS-equivalent sentences than equal-length event-denoting constituents. We include the mean score and skewness value for the other eight languages in Figure 6, which shows the same trend as in Chinese and English. These results highlight a multilingual phenomenon: participantdenoting constituents tend to have higher span-
overlap scores than equal-length event-denoting constituents. The phenomenon indicates a statistical difference between the two constituent types, shedding light on future unsupervised parsing research with phrase labels.

heading: 5.4 Span-overlap Score Distinguishes between Constituents and Non-constituents
text: This experiment provides a glimpse of the discriminating power of the span-overlap score. We compare the span-overlap score with random scores in terms of whether the score can identify the constituent in a set of mutually incompatible word sequences. The random score is uniformly sampled from [0, 1], the same range as the span-overlap score. We measure the proportion of gold constituents to which the score assigns a strictly higher value than all incompatible non-constituents. We refer to a non-constituent as incompatible with a gold constituent when the two sequences partially overlap (i.e., they cannot co-exist in the same tree).
Figure 5 plots the proportion by word sequence lengths. The span-overlap score distinguishes between constituents and non-constituents, identifying more constituents than the random score for all sequence lengths and in all languages. In the European languages, the span-overlap score reliably identifies around 50% of constituents, regardless of the constituents’ length. In contrast, the random score identifies around 40% of short constituents but only around 10% of long constituents. The comparison highlights the effectiveness of the spanoverlap score in identifying constituents, especially
for long constituents. However, Chinese and Koreans are exceptions. While the span-overlap score still identifies more constituents than the random score, the proportion drops for long constituents. This is because no long word sequences appear in both the target sentence and the generated PASequivalent sentences. The problem might be related to the sample quality problem discussed in Section 7.

heading: 5.5 Ablation of PAS-Preserving Instructions
text: Table 5 showcases the benefit of aggregating multiple PAS-preserving instructions. The table compares the span-overlap parser with only one instruction type and with all six instruction types. Even with a single instruction type, the span-overlap parser outperforms the random, right-branching, and GPT-3.5 parsers. This result suggests that the span-overlap method uncovers non-trivial constituent structures. The best parser with a single instruction type is always the one with the movement instruction. The phenomenon suggests a fundamental role of movable constituents in identifying constituent structures. Combining all instruc-
tion types sees significant improvements in parsing accuracy. The average improvement is 3.09 SF1 score between the parser with all instruction types and the best parser with a single instruction type. This result indicates the benefit and necessity of aggregating multiple instructions in generating the PAS-equivalent sentence set.

heading: 6 Conclusion
text: In this paper, we approached the unsupervised constituency parsing problem by exploiting word sequence frequency information revealed in a set of PAS-equivalent sentences. Firstly, we empirically verified a hypothesis: constituents correspond to frequent word sequences in the PASequivalent sentence set. We proposed a frequencybased method span-overlap that computes the spanoverlap score as the word sequence’s frequency in the PAS-equivalent sentence set. The span-overlap parser outperformed state-of-the-art parsers in eight out of ten languages evaluated and significantly outperformed the GPT-3.5 parser in all languages. The span-overlap score can effectively separate constituents from non-constituents, especially for long
constituents. These results highlight the effectiveness and applicability of the frequency information to unsupervised parsing. Additionally, our study uncovers a multilingual phenomenon: participantdenoting constituents tend to have higher spanoverlap scores than equal-length event-denoting constituents, meaning that the former tend to appear more frequently in the PAS-equivalent sentence set than the latter. The phenomenon indicates a statistical difference between the two constituent types, laying the foundation for future labeled unsupervised parsing research. Finally, our ablation analysis shows a significant improvement by aggregating multiple PAS-preserving transformations, indicating the necessity of the aggregation in comprehensively exposing the constituent structure.

heading: 7 Limitations
text: The span-overlap method employs a diverse but non-exhaustive set of instructions to generate PASequivalent sentences. We did not cover PASpreserving transformations that are unique to some languages. For example, Korean and Chinese have a null subject transformation that can remove the subject from sentences. The span-overlap method may miss out on word sequence patterns that those transformations cover. However, this paper aims to demonstrate the utility of word sequence frequency information and verify that constituents correspond to frequent word sequences. We will leave the incorporation of more complex language-specific transformations to future work.
Additionally, the span-overlap method faces challenges with sample quality, particularly in non-English languages. Previous research (Rosól et al., 2023; Seghier, 2023) has pointed out that the GPT-3.5 model performs significantly better in English than in other languages. Similarly, we observed that English samples are of better quality than Chinese samples. We refer the readers to Appendix A.3 for more details. We also observed that using GPT-4 instead of GPT-3.5 can significantly improve the parsing accuracy in a preliminary experiment. These results suggest that the sample quality currently limits the span-overlap parser. That being said, the span-overlap parser’s performance indicates the utility of the word sequence frequency estimated with even low-quality samples. We expect the sample quality issue to be mitigated in the future with the rapid development of LLM technologies.

heading: A Appendix
text: A.1 Span-overlap Scoring with Fuzzy Matching
The application of exact matching (i.e., the span only matches identical substrings in PASequivalent sentences) rather than fuzzy matching (i.e., the span matches similar substrings) requires further discussions. In a preliminary experiment, we found that the fuzzy matching variants of the span-overlap parser underperform the span-overlap parser utilizing exact matching. We experimented with two fuzzy matching variants, one based on the BLEU score (Papineni et al., 2002) and the other based on the METEOR score (Banerjee and Lavie, 2005). The two fuzzy matching scores allow the matching between two textually different word sequences that share the same meaning. For example, “give John a book” and “give a book to John”.
score(w, (i, j)) :=
∑ sk∈s maxp f(w i,j, sp,p+j−ik )
|s| (3)
To introduce the fuzzy matching variants, we generalize Equation 2 as Equation 3. When the scoring function f is an indicator function, Equation 3 is the span-overlap score we introduced in the main text. In the BLEU and METEOR scorebased variants, we employ the BLEU and METEOR scores for f , respectively. Table 7 compares the parsing accuracy of the two fuzzy matching variants against the exact matching variant on a 10- word subset of the development set of the PTB and the SPMRL German dataset. The 10-word subset contains sentences of length ≤ 10, as per previous research (Klein and Manning, 2002). As shown in the table, the fuzzy matching variants fall behind the exact matching variant in parsing accuracy by a large margin. Upon further investigation, we found that fuzzy matching scores not only match “give John a book” with grammatical strings (e.g., “give a book to John”), but also match with ungrammatical strings (e.g., “give John a book in”). The matching with ungrammatical strings might introduce unnecessary noise to the span-overlap model, which is detrimental to the parsing performance.
A.2 PAS-preserving Instructions Table 6 shows the specific instruction we used in our experiment. To generate samples for nonEnglish languages, we append an additional directive, “The generated sentences must be in <language>.”, to the above instructions. For example, we add “The generated sentences must be in German.” to generate German sentences.
A.3 Examining Sample Qualities We manually examined 100 random pairs of (target sentence, generated PAS-equivalent sentence) in English and Chinese. We rate the PAS-equivalence in the range of 0-10 (i.e., whether two sentences express the same PAS information). We rate 10 if the two sentences express the same PAS information, 5 if half of the information from the target sentence is missing in the generated sentence, and 0 if the two sentences are completely irrelevant. We present some rating examples in Table 8.
We can make two observations from Figure 7: (1) Most generated sentences have a PAS that is reasonably similar to the PAS of the original sentence; (2) English samples tend to have higher PASequivalence scores than Chinese samples. These observations, on the one hand, confirm the plausibility of generating PAS-equivalent sentences with the GPT-3.5 model. On the other hand, the observa-
tions indicate a limiting factor for the span-overlap method: the sample quality.

heading: Future Work
text: We will leave the incorporation of more complex language-specific transformations to future work. Additionally, the span-overlap method faces challenges with sample quality, particularly in non-English languages. Previous research (Rosól et al. , 2023; Seghier, 2023) has pointed out that the GPT-3. 5 model performs significantly better in English than in other languages. Similarly, we observed that English samples are of better quality than Chinese samples. We refer the readers to Appendix A. 3 for more details. We also observed that using GPT-4 instead of GPT-3. 5 can significantly improve the parsing accuracy in a preliminary experiment. These results suggest that the sample quality currently limits the span-overlap parser. That being said, the span-overlap parser’s performance indicates the utility of the word sequence frequency estimated with even low-quality samples. We expect the sample quality issue to be mitigated in the future with the rapid development of LLM technologies