LLM-enhanced Self-training for Cross-domain Constituency Parsing
Authors: jianling li
Meishan Zhang
Peiming Guo
Min Zhang
Yue Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: constituency parsing

Abstract: Self-training has proven to be an effective approach for cross-domain tasks, and in this study, we explore its application to cross-domain constituency parsing. Traditional self-training methods rely on limited and potentially low-quality raw corpora. To overcome this limitation, we propose enhancing self-training with the large language model (LLM) to generate domain-specific raw corpora iteratively. For the constituency parsing, we introduce grammar rules that guide the LLM in generating raw corpora and establish criteria for selecting pseudo instances. Our experimental results demonstrate that self-training for constituency parsing, equipped with an LLM, outperforms traditional methods regardless of the LLM's performance. Moreover, the combination of grammar rules and confidence criteria for pseudo-data selection yields the highest performance in the cross-domain constituency parsing.

LLM-enhanced Self-training for Cross-domain Constituency Parsing
Jianling Li
Meishan Zhang² Peiming Guo Min Zhang²
Yue Zhang
School of New Media and Communication, Tianjin University, China Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen), China
Sccooo of Engineering, Westlake University, China
stanlingl@tju.edee cn, mason. zmsêgmail. com, guopeiming. suppeiming.gpm@@gaill.. gpm@gmail. com zhangmin2021 2wt2222222hhtc.. edu. cn, yue. www.zzainnnaa.oooo orw.wiiss..o.oooo .
Abstract
and Reichart, 2019; Ramponi and Plank, 2020; Ye et al., 2020; Wang al., 2021). This method uti- lizes a source domain model to automatically label a large-scale raw corpus from the target domain dur- ing each iteration. High-confidence pseudo data is then selected as additional training data to improve target domain performance. However, the quality and quantity of raw corpus cannot always be guar- anteed for low-resource domains (Steedman et al., 2003; Qiu al., 2014; Peng al., 2021), which limits the use self-training approaches. Tradi- tional methods struggle to construct fine-grained sentences that facilitate knowledge transfer. The Large Language Model (LLM), with its powerful generative capabilities, can serve potential so- lution to the challenge of the raw corpus quantity and quality for the target domain (as shown in Fig- ure 1). It's important note that our experiments revealed that LLMs exhibit limited performance for constituency parsing.
Self-training has proven to be an effective ap- proach for cross-domain tasks, and in this study, we explore its application to cross dossin constituency parsing. Traditional self-training methods rely on limited and potentially low- quality raw corpora. To overcome this limita- tion, we propose enhancing self self-training with the large language model (LLM) generate domain-specific raw corpora iteratively. For the constituency parsing, we introduce grammar rules that guide the LLM in generating raw pora and establish criteria for selecting pseudo instances. Our experimental results demon- strate that self-training for constituency parsing, equipped with an LLM, outperforms traditional methods regardless of the L.LM's performance. Moreover, the combination of grammar rules and confidence criteria for pscudo-data selec- tion yields the highest performance in the cross- domain constituency parsing".
1 Introduction
To tackle the challenges of LLMs' flexibility and hallucination problems (Bang et al., 2023; Man- akul et al., 2023) in generating sentences, we em- ploy grammar rules as instructions for LLMs to generate target domain sentences. Grammar rules have proven effective in cross-domain data gener- ation (Wang al., 2023) and are closely related to constituency parsing (Yang al., 2022). By in- corporating grammar rules, we aim to generate a large volume of high-quality raw sentences highly relevant to the cross-domain constituency parsing task. Furthermore, we dynamically embed LLMs into the iterative process of self-training, enhancing their adaptability and flexibility.
Constituency parsing, fundamental task in natural language processing (NLP), has achieved remark- able progress on in-domain benchmarks (Liu and Zhang, 2017; Gaddy et al., 2018; Kitaev and Klein, 2018; Kitaev et al., 2019; Cui et al., 2022), indicat- ing the growing competence of parsers in capturing the underlying syntactic structures. However, open- domain constituency parsing is notably challenging (Fried et al., 2019; Yang et al., 2022). In diverse, open-domain scenarios, constituency parsing faces complexities beyond the well-defined task. Ad- dressing these challenges is crucial for its broader real-world NLP applications.
To address the issue of domain shift, self- training-based unsupervised domain adaptation has emerged as promising approach (Yu et al., 2015; Sachan and Xing, 2018 8; He et al., 2019; Rotman
In each iteration of self-training, we 1) use LLM with grammar rules extracted from currently avail- able training instances generate target domain raw corpus; 2) train a constituency parser with this data; 3) parse the raw corpus using the trained parser; and 4) select high-quality pseudo data from the parsed trees as additional training instances for
*Corresponding author:
'We have made our code publicly available at https:// gi github. cow/ aniinnnnnnni http://nnnn//LLMMSS.osssssarrri

---

Genernte by the grammars: S-AP
to annotate the treebank (Mareus et al., 1993; Xue et al., 2005; Seddah et al., 2013). In contrast, unsupervised or weakly supervised open-domain parsing remains a challenging research task (Yang et al., 2022). There are limited studies on cross-domain constituency parsing (McClosky et al., 2010; Fried et al., 2019).
Terms were desclosed.
Parse the constituency lree for the sentenne:: were desclosed
(VP were n't (VP disclosedi
Figure 1: the capabilities of LLMs. Despite encounter- ing parsing challenges such as missing elements, LLMs can generate reliable sentences using grammar rules.
Self-training Researchers (Wang et al., 2015; Yang et al., 2022) have attempted to directly trans- fer source domain parsers to target domains. How- ever, such direct transfer approaches are insuffi- cient for target domains with large gaps compared to the source domain. Self-training (Yarowsky, 1995; McClosky et al., 2006; Yu et al., 2015; Ram- poni and Plank, 2020; Guo et al., 2022) is a simple and early bootstrapping approach for domain adap- tation, applied to tasks such as classification Dong and Schäfer (2011) 1); Ye et al. (2020), sequence la- beling Wang et al. (2021, 2020), dependency pars- ing Yu et al. (2015); Rotman and Reichart (2019); Guo et al. (2022), sequence generation He et al. (2019), and QA Sachan and Xing (2018) and on. However, self-training has not been applied cross-domain constituency parsing.
the next iteration. For data selection, we employ multiple criteria such as token-based, confidence- based, and grammar-rule- rul-naarr-ul--bsedd selection.
Experimental results demonstrate the superior constituency parsing performance of our LLM- enhanced self-training approach. The raw corpus generated by LLM proves quite effective for the cross-domain self self-training tasks, and combining grammar-rule rultnarr-ule-baedd and confidence-based selec- tion of pscudo data performs best. This criterion considers both structural information and ensures reliability. Additionally, open-source LLMs can effectively replace closed-source L.Mss, achieving comparable performance in our methods. Through- out the self-training iterations, updated grammar rules and selected pseudo data progressively move closer to the target domain.
LLMs' Parsing LLMs have been successfully applied various NLP tasks (Brown al., 2020; He al., 2023; Mysore et al., 2023; Zhang et al., 2023; Pangakis et al., 2023; Ashok and Lipton, 2023). While LLMs exhibit limitations in struc- tured extraction tasks (Qin et al.. 2023), such as cross-domain constituency parsing, their genera- tive capabilities can be leveraged to provide raw corpus for self-training constituency parsing.
To the best of our knowledge, we are the first to dynamically incorporate LLMs into the self- training for constituency parsing. Our model uti- lize LLMs to create fine-grained raw corpus that gradually adapts from the source the tar- get domain. We also introduce an optimal pseudo- data selection criterion, combining grammar-rule- based and confidence-bart based criteria. This selec- tion effectively contributes to our LLMMenhanced self-training method for cross-domain constituency parsing. Furthermore, we observed that LLMs un- derperform in few-shot settings compared to both our baselines and our proposed method.
To address the issue of hallucinations in LLMs. we propose employing grammar rules to regulate the structure of generated sentences. We also provide a small set of domain-specific sentences as style references and length limits to further guide the LLM generation process. Prior research Yang et al. (2022) has shown close relationship between grammar rules and constituency parser performance, while Wang et al. (2023) success- fully used grammar rules to generate cross-domain data. Our work uniquely integrates LLMs into teif-training iterations, generating fine-grained raw corpus based on the grammar rules, making the process more flexible and controlled.
2 Related Work
Cross-domain Constituency Parsing Con- stituency parsing, a long-standing traditional NLP task, has evolved over the years (Collins, 1997; McClosky al., 2006; Zhu et al., 2013; Dyer et al., 2016; Stern et al., 2017: Gaddy al., 2018). Supervised constituency parsing (Liu and Zhang, 2017; Kitaev and Klein, 2018; Kitaev et al., 2019; Cui et al., 2022) has achieved remarkable performance, but it requires massive manual efforts
Pseudo-data Selection Strategy When apply- ing the self-training method to different tasks, re- searchers typically develop task-specific criteria

---

for pseudo-data selection, taking into account the unique characteristies of each task. For example, Dong and Schäfer (2011) use an ensemble learn- ing model for reliable newly labeled data selec- tion in classification. Wang et al. (2020) show that sampling strategies may vary across datasets, and Jiang et al. (2021) propose using out-of-vocabular numbers measure source-target domain distance in Chinese word segmentation and POS tagging. this work, we propose adopting grammar rules as selection criterion and combining it with confidence-basee selection. This method ensures the pseudo-dale contains rich structural information and exhibits high reliability, ultimately improving cross-domain constituency parsing performance.
L.M generated
Corpus
Pacudo Treebank
Source
Constituency Parser
Treebank
NEVP NP PPNE
Selection Strrogy
Raw
Corpus
Figure 2: Vanilla and I..M-enhanced self-training frameworks for cross-domain constituency parsing. The former is represented by the lower gray dashed line loop, while the latter is depicted by the upper blue solid loop.
The vanilla self self-traiinngg-bee based cross- coos-domain con- stituency parsing is an iterative process aimed training target parser.
3 Method
3.1 Baseline Model
Specifically, in each iteration of the vanilla ap- proach, three main steps are conducted: 1) Training the parser: We train the Berkeley Neural Parser ing the source domain constituency treebank. 2) Parsing raw corpus: We apply the trained model to parse the raw text from the target domain, gen- erating parse trees that serve as candidate pseudo trees for the next step. 3) Selecting pseudo-data: We select high-confidence pseudo trees to serve additional training instances, which are then used to enhance the model performance the target domain. By repeating these steps iteratively, the self-training method adapts the parser to the tar- get domain, leveraging both the source annotated treebank and high-quality pseudo trees generated throughout the process.
We employ the Berkeley Neural Parser (Kitaev and Klein, 2018) as the foundation of our method. This parser is a chart-based approach that adopts self- attentive encoder and chart- based decoder, utiliz- ing pre-trained embeddings as input to enhance the parsing process.
Due the integration of pre-trained language models, the Berkeley Neural Parser inherently pos- sesses cross-domain constituency parsing capabil- itics. This enables the parser to be trained on the source domain and then directly applied to parse trees for the target domain. We use this direct model transfer baseline for comparison. Additionally, we compare the cross-domain pars- ing performance of the smaller model (e.g.. Berke- ley Neural Parser) with that of the LLMs con- stituency parsing. Specifically, we provide the Large Language Model with few parse trees from the source domain as prompts and ask it to generate parse trees for the target domain. This comparison further helps us understand the strengths and weak- nesses of both large and small models when applied constituency parsing tasks.
3.3 LLM-enhanced Self-training
To improve the quality and quantity of raw corpus used vanilla self-training, we propose to inte- grate LLM into the self-training iteration shown in Figure 2. We dynamically embed the LLM crucial component our iterative self-training process. each iteration, we utilize the LLM to generate raw corpus for the target domain, based on the updated treebank from the previous step. Fol- lowing the detailed LLM-enhanced self-training constituency parsing algorithm 1, our method re- quires annotated treebank from the source do- main, as well small number of sentences from the target domain. The Grammar Rules (GRs) are extracted from the treebank and play crucial role in guiding the LLMs generation of raw corpus for target domain.
3.2 Vanilla Self Training
In this section, we introduce a vanilla self-training method for cross- cross-domain constituency parsing, which has been investigated in other tasks. Please note that we refer the standard self-training method vanilla self-training. The primary goal of self-training is to generate high-quality training instances for the target domain, subsequently using these instances to train the target domain model.

---

Algorithm 1: LLM-enhanced Self-training
process and improving the parser's performance on the target domain throughout the iterations.
Input:
Source Constituency Treebank $
Treebank Grammar Rules GRs
3.4 Instanecs Selection Criteria
Small Target Domain Raw Corpus R
Output: Target Constituency Parser M
To align with the goal of the LLM-generated raw corpus, which aims to incorporate as much struc- tural information possible by grammar rules improve constituency parser performance, we pro- pose grammar-rule based selection criterion for pscudo-data. Unlike previous sclf-training selee- tion criteria that focus solely on the task, this crite- rion considers both the task and the characteristics of the L.M--eeerrted corpus, ensuring that the se- lected pscudo-data is appropriate for cross-domain parsing using self-training.
Pseudo Target Treebank 1)
D
foreach in [1, .... until convergence do
LLM Generating: R GPT(GRs, R)
3
Parser Training: M (rain(S D)
Domain Parsing: D = parse( (M,
Trees Selection: = topK(D)
7 Treebank Update: S = SUD
GRs Extraction: GRs = extract(S)
end
In particular, LLMs generate sentences that closely resemble the target domain using grammar rules. Our selection criterion, on the other hand, employs grammar rules to choose pseudo-data that is as relevant as possible to the source domain, ducing potential transfer failures due to large biases and gaps. The feasibility of grammar-rule-base selection criteria is also supported by Yang et al. (2022) and Wang al. (2023).
We divide the LLM-enhanced self-training con- stituency parsing into six detailed steps on each iteration: 1) LLM Generating: We first leverage the Large Language Model to produce a raw corpus R for the target domain, based on GRs extracted from the currently available treebank and few sample sentences (R) from the target domain. 2) Parser Training: Next, we train a constituency parser using the source treebank S and the selected pscudo trees D for the target domain. During the initial step, the pscudo treebank isempty (D {}) and the parser is trained solely on the source do- main data. 3) Domain Parsing: We apply the trained parser to parse the generated raw corpus R, resulting in a set candidate parse trees D. 4) Trees Selection: From the generated parse trees D. we select a subset of high-quality parse trees to form the pseudo treebank D.. The selection criteria detailed in subsection 3.4. 5) Treebank Update:
However, directly measuring the distribution dis- parity between training set and candidate in- stance be challenging. We provide high-level inspection of how to evaluate the distance between large set and an individual instance. Given the source set, denoted as S, and the candidate instance, denoted as E (candidate set), we describe the distance computation between c and S using equa- tion (1):
D(c, (SS
(1)
instances topk arg min D(c,
(2)
cEC
We update the source treebank S by adding the selected pscudo treebank D to it, effectively in- creasing the diversity of training data for the target domain. 6) GRs Extraction: We extract grammar rules G Rss from the updated treebank S, which will guide the LLM to generate more Informative raw corpus for the target domain in the next iteration. The LLM-enhanced self-training process is itera- tively continued until convergence. The final out- put of the algorithm is trained target constituency parser M and a pseudo treebank D for the target do- main. This approach leverages the Large Language Model generated raw corpus, which replaces the vanilla ready-made text, enhancing the adaptation
We then pick the topK candidates closest to the source domain set additional training instances in the self-training process. This approach ensures that the most relevant instances are selected, en- hancing the model's gradual adaptation to the target domain.
The distance computation can be performed at either the token level the grammar rule level by adjusting the represent token distribution or grammar rule distribution, respectively. The grammar rules we use include both terminal and non non-terminal rules. Our instance selection process involves three levels of criteria: token, confidence,

---

and grammar rule. We also combine the two best- performing criteria, namely confidence-based se- lection and grammar-rule-based selection, result- ing in a more effective criterion for identifying high-quality instances for adaptation to the target domain.
LLM Prompt
As a language assistant, you excal at creating sen- tences of specific length while adherating to N gram- mar rules provided above. Please consider the M ex- amples, generate one sentence of L1 L2 words: GRs: S-PP VP. PP-IN NP. VP-VP NP. Snts: 1. History brings inspiration peace. 2. is the jack of all trades and master of none ...
Token-based criteria prioritize trees with word distributions closer to the source treebank (us- ing the JS distance in equation (2)).
Figure 3: LLM prompt example for generating a sen- tence by the grammar rules and domain sentences. Note that the blue tokens and dashed lines not part of the actual prompt, only shown for illustration.
Conf-based criteria focus on the scores of la- beled phrases provided by the model to iden- tify high-confidence pseudo trees.
opt to extract 5 target domain sentences based on preliminary experiments. Since the length of gen- erated sentences is closely related to the number of grammar rules N, we use another normal dis- tribution, denoted as N = =N(N, 3) to sample two values, L1 and L2, which define the limits for the length of the generated sentence.
GRs-based criteria favor instances with simi- lar structures by calculating the JS distance be- tween the candidate tree and the source gram- mar rules set.
GRsConf-based criteria select high- confidence instances among candidates with high scores the grammaa rule, considering both structural information and data reliability.
An illustration of the LLMs prompt example is presented Figure 3, and we utilize gpt-3.5- turbo with the temperature set to 0 for the LLMs generation process.
3.5
LLM prompt
To generate sentences that encompass comprehen- sive structural information and closely resemble the target domain sentence style, we introduce a LLM prompt that integrates grammar rules and target domain examples. During the generation, need prepare the following parameter: 1) N grammar rules extracted from the treebank, 2) M sampled sentences from the target domain, and 3) length constraints L1 ~ L2 for the generated sentence to ensure they are neither too short nor too long. Through preliminary experiments, we have found direct correlation between the number of grammar rules and the length of LLM generated sentences. Therefore, N by sampling from the distribution of treebank sentence lengths from which we extract the gram- mar rules. Note that the grammar rules are di- rectly extracted from the constituent tree, where the parent node corresponds to the left hand of the grammar rule, and all child nodes correspond to the right tail side. For instance, if the treebank is the source domain data PTB, we introduce Gaus- sian distribution for the averge length, denoted as N N (avg_len, 6) to obtain N grammar rules. The number of target domain sentences be extracted is customizable, but due to resource con- straints and minimal performance differences, we
Experiments
4.1 Data
We use the PTB as our source domain (newswire) and the Multi-domain Constituent TreeBank (MCTB) as the target domain (Yang et al., 2022). covering Dialogue, Forum, Law, Literature, and Re- view. For validation, we utilize PTB. dedv treebank in our cross-domain parsing. For each domain, the vanilla self-training process utilizes the raw corpus of 100k sentences collected from same source the test set, including Wizard (Dinan al.), Reddit (Völske et al., 2017), ECtHR (Stiansen and Voeten, 2019), Gutenberg and Amazon (He and McAulcy, 2016). This signifies that the source training data and the target test set sentences are homologous, thereby guarantecing the robustness of the vanilla (i.e., standard) self-training method as our baseline. To further enhance the quality the crawled raw corpus, we filter out sentences that are either too long (number of words 100) or too short (number of words < 3). Then, we sample 40k raw sentences for the Vanilla self-training method. The raw sentence length statistics for the selected 40k and the LLM-ggnerated sentences during the four
determine the value of
Phttss//www.gutcnerrggooo

---

Method
Criteria
Dialogue
Forum
Law
Literature
Review
Avg
gpt-3.5-turbo
70.70-aa.
71.56m
80.72-mw
72.83m
71.24mm
73.41 1m
Kitaev and Klein (2018)
85.92
86.00
91.71
85.04
83.51
86.44
Token
86.05
86.52
91.75
83.53
83.96
86.77
Conf
86.03
86.54
91.94
85.60
83,93
86.81
Vanilla ST
GRs
86.04
86.49
91.92
85.63
83.94
86.80
GRsConf
86.13
86.57
91.93
85.76
84.02
86.88
Token
86.01
86.52
91.69
85.55
83,92
86.74
Conf
86.26
86.67
92.00
85,90
84.07
86.98
LLM enhanced ST
GRs
86.18
86
91.97
86.00
84.09
87.01
GRsConf
86.71
87.10
92.30
86.17
84.34
87.32
Liu and Zhang (2017)*
85.56
86,33
91.50
84,96
83.89
86.45
Kitacv and Kicin (2018)*
86.30
87.04
92.06
86.26
84.34
86.20
LLM-enhanced ST*
GRsConf 87.59
87.55
93.29
87.54
85.58
88.31
Main results of Vanilla and LLM-enhhnced Self Self-training (ST) with four pseudo-date selection criteria: Token, Conf, GRs and GRsConf. & and denote the results based on large bert and referred from Yang et al. (2022), respectively. The scaling probability applied to gpt-3.5-turbo results corresponds to the proportion of correct outputs. The $ indicates statistical significance compared baselines with p 0.05 paired t-test.
Table I:
iterations are included in appendix A. To better understand the characteristics of our generated sen- tences, we also provide several typical examples for both crawled and LLM-generated raw sentences in appendix A.2.
and only the best methods were further experi- mented on bert-large-uncasedd The performance of the constituency parser on five target domains is reported in Table 1.
In the first stage of our experiment, we assessed the performance of gpt-3.5-turbo few-shot set- tings for constituency parsing five target do- mains. We provided the model with three gold- annotated parse trees paired with sentences from the PTB as demonstrations, and then had gpt-3.5- turbo generate bracketed trees for target domain sentences. However, due to issues such as miss- ing elements and mismatched brackets in LLM's output, more than half of the parse trees are un- available. For the five target domains under con- sideration, each comprising 1,000 test samples, the number of available outputs 424 (Dialogue), 212 (Forum), 276 (Law), (Literature), and 367 (Re- view), respectively. The LLM exhibits domain bias in the formatting errors of parse tree. It is impor- tant to highlight that the reported scores are likely higher than the actual performance, and the scores presented in the main table have been adjusted by multiplying the corresponding available probability. Furthermore, compared the other domains, gpt- 3.5-turbo demonstrates a significantly better perfor- mance in constituency parsing for Law domain, just looking at the correctly formatted parsing results.
4.2 Parameters
We employ the same parser for self-training methods as in Kitaev and Klein (2018)'s work. Dur- ing the self-training iteration, the raw corpus ini- tially segmented by Stanza (Qi et al., 2020) and subsequently tagged by the trained parser. The self- training process in all cases comprises iterations and involves selecting the topk = 2k pseudo-data to be integrated as additional training instances for the subsequent iteration. In the vanilla self-training we choose the topK high-quality instances from pool of 100k examples and remove the selected sentences from the raw corpus. contrast, for the LLM-enhanced method, we select the topK data from a pool of 10k examples, as LLMS generate 10k raw sentences for self-training in each iteration. For the LLM-eehanced constituency parser, we extract grammar rules from current available tree- bank and integrate them with gpt-3.5-turke for gen- erating raw corpora. All the parsers employ three distinct seeds, and the performance measured as the average Fl score.
4.3
Main Results
Secondly, we investigated direct model transfer for cross-domain constituency parsing, a strong baseline method compared with LLMs' parsing.
For convenience, the main comparative experi- ments were conducted using bert-base-uncased.

---

We trained the parser on the source PTB treebank and directly applied it the five target domains. From the results, we observed varying distances between the five target domains and the source domain, with the Law domain being closest and the Review domain being farthest in similarity. The difference in Fl scores between Law domain and Review domain is 91.71 83.51 8.2 points. On average, the performance of the model transfer method based on port-base-uncasee surpasses that of the Large Language Model's parsing, which is 86.44 - 73.41 13.03.
ment observed in the Literature domain, while the least improvement seen in the Law domain. worth noting that Yang al. (2022) used the di- vergence of grammar rules to measure the distance between different domain constituency parsing tree- banks. Among these, the Law domain closely resembles the source domain, exhibiting mini- mal improvement of 0.59. Moreover, our LLM- enhanced self-training approach is more effective for domain adaptation tasks with larger difference between the domains.
Additionally, we included two baseline models that employed bert-large-uncasee for transition- based and graph-based cross cross- domain constituency parsing. The results demonstrate that direct model transfer is relatively effective method. It is im- portant note that we cannot make direct com- parison with the pert-base-uncante results, as the experimental settings (including seed, batch size, and predict tags) are not entirely consistent.
In the third stage. we examined vanilla self- training using four different selection strategies. From the observation, we find that the optimal se- lection strategy is not the same for the five target do- mains. In the Dialogue and Literature domains, the selection based on GRsConf apparently obtained the best performance. We also noticed that the Forum and Review domains exhibit only slight variations across the four pseudo-data selection criteria. However, for the Law domain, employ- criteria is the best
Lastly, we conducted experiments of the LLM- enhanced self-training method with the best- performing selection strategy GRsConf under bert- large-uncased. The approach based on bert-large- uncased outperforms the with anaverage improvement of 0.99. The largest improvement is observed in the Literature domain, with score increase of 87.54 - 86.17 = 1.37. On the other hand, the smallest improvement is seen in the Forum domain, with a score increase from 87.55 87. 0.45. These results indicate that utilizing larger pre-trained language models can lead to better performance in the constituency parsing task across various domains.
ing only the confidence- choice to achieve self-training improvements. The token-based selection criteria do not demonstrate a significant advantage; they still improved the con- stituency parser by 0.33 compared to the model transfer. Looking the average performance, it be- comes evident that the selection strategy GRsConf is relatively superior compared to other approaches. This highlights the effectiveness of criteria com- bination, which not only considers the structural information but also ensures data reliability.
method
Subsequently, we explored LLM-enhanced self- training for constituency parsers, employing the four selection strategies. The superiority of L.MM- enhanced self-training consistency parsing over the vanilla approach is evident across all selection crite- ria, except for the Token-based selection, where the latter performs better. Furthermore, it is notable that the GRs-based method show a bit more en- hancements compared to the Conf-hased selection. This highlights that the effectiveness of selection criteria is significantly influenced by the quality of the raw corpus utilized in self-training. The positive results also demonstrate the efficacy of our incremental approach, which uses the LLM generate target domain sentences in each iter- ation. Compared to the basic model transfer, our LLM-enhanced method achieves an average im- provement of 0.88. The most significant improve-
5 Analysis
To conduct thorough analysis and gain deeper insights into our methods, we have chosen the Re- view domain for the detailed exploration. Due space constraints, we placed the comparison be- tween open-source and closed-source LLMs ap- proaches in the appendix A.3
5.1 The Instance Selection Strategy
We first investigate four distinct selection strate- gies for each iteration: loken-based, Conf-based, GRs-based, and GRsConf-basee The line chart in Figure is divided into two partitions. illustrat- ing the parser performance during the iterations for both Vanilla and LLM-enhanced self-training constituency parsing. The chart distinctly shows

---

GlteConf
0.25
SE cisstance to largel domaiu
86.0
F distance source
0118
JS distance
5377
AO0000 1.3
0415
2.09
83.
0.05
2
4
Trerasions for Vanilla ST
Deratione Sor LLM eablnde ST
Iterations for I.L.Menhanced ST
Figure 4: Constituency parsing performance by iter- ation: Vanilla (left) and LLM-enhanced (right) self- training (ST), the dashed line the right plot indicates the optimal performance of the Vanilla method.
Figure 5: Distances between Pseudo-data and source and target domains during iterative transfer of Review.
5.2 Pseudo-data from GRsConf
In the context of cross-domain constituency pars- ing based LLM-enhanced self- self-training, the key to performance improvement lies in whether the selected pseudo-data gradually moves closer to the target domain. The LLM generation process and the selection strategies guide the iterations from two opposite directions: the L.M--genrrted raw text progressively shifts towards the target domain, while the selection criteria aim to ensure that the pseudo-data remains close to the source domain. Consequently, analyze the best selection strat- egy for the Review domain, GRsConf, and examine the distribution of the selected pseudo-date during each iteration. Following the work Yang et al. (2022), we also use the JS divergence of GRs to measure the distance between the selected pseudo- data and both the source and target domains.
that for the Vanilla method, all strategies except for GRsConf exhibit an initial increase in performance followed by a decrease. This trend suggests that after a few iterations, the candidate data becomes increasingly feature-biased and less suitable for the domain transfer. In the Review domain, the best performance of Vanilla self-training is achieved using with GRsConf- pseudo-data.
In contrast, the LLMM-ehanced self-training demonstrates a consistent upward trend for all four selection strategies, indicating that the selected data of high quality and that the adaptation pro- cess is both gradual and progressive. This out- come highlights the feasibility and effectiveness of incorporating LLMS into the self-training iter- ation process, enabling more fine-grained trans- fer from the source domain to the target domain. It is also worth noting that the LLM-enhanced method, except for the token-based selection strat- egy, achieves performance that either similar to or better than that of the best Vanilla method. The I.MM-enhaced method's s best performance is achieved using GRsConf, further solidifying the no- tion that well-designed selection criterion, when combined with high-quality data, leads to more effective results during the adaptation process.
As depicted in the chart (Figure 5), the distance between the selected pseudo-data and the source domain increases, while the distance to the target domain gradually decreases. The trend reveals that domain transfer minimal in the first iteration, with more substantial adaptation occurring in the second and third iterations, and eventually stabiliz- ing in the fourth iteration. This distance evolution- ary trends suggests that the domain transfer pro- cess is both gradual and progressive, corroborating the effectiveness of the GRsConf selection strat- egy combined with LLMM-cnhanced self-training for cross-domain constituency parsing.
It is essential to note that our analysis only dis- plays results for four iterations. In our experi- ments, also tested up to six iterations. However, the results indicate that, for the Vanilla method, the performance decline becomes increasingly pro- nounced. And for the L.LM Me--ennhdced self-training, no further improvement in performance is observed beyond the fourth iteration.
5.3 Target Sentences Effect
To further investigate the impact of the number of target domain sentences on our M.eehanhae self- training, we conducted a comparative experiment in the Review domain using the pseudo-data selec-

---

tion method based GRsConf. We compared the parser performance, setting the number of target do- main sentences to 0, 5, 10, and 20 respectively. As shown Table 2, we can conclude that the quantity sentences does not significantly impact the final target domain parser. Moreover, when no target domain sentences were provided and only the tar- get domain name (Review) was given, the results showed decrease in performance. Further anal- ysis revealed that chatGPT's generated sentences, based on the domain name, significantly differed from the actual domain data.
Models
GRs
No GRs
LL.M-enhaneed ST
84.
83.15
Table 3: Results of using GRs VS using GRs.
teria with hrammar-rule-bene selection, progres- sively moving the training data closer to the target domain. Through experiments, we demonstrated that our method's domain transfer is effective, re- sulting in improved performance within the target domain. In conclusion, our LLM LLM-enbanced self- training approach offers a promising solution for cross-domain adaptation tasks.
5.4 GRs effect
Additionally, we set up the LLM generation process with 5 target domain sentences, omitting the intro- duction of grammar rules. According to the experi- mental results shown in Table 3, it is evident that the parser's performance without the grammar rules is inferior to that of the standard 4LM-enhanced self- telf-nraining approach. This demonstrates that con- straining LLM's generation with grammar rules is a reasonable choice.
A ckonwwedde
We sincerely thank the reviewers for their invalu- able feedback, which significantly improved the quality of this work. We are deeply grateful the National Natural Science Foundation of China (NSFC) for their funding support, under Grant Nos. (19.61976180 and Nos.62176180, which made this research possible.
Regarding grammar rules, there are other intrigu- ing findings to consider. For example, treebanks from different domains exhibit long-tail distribu- tion of grammar rules. While sharing consider- able number of grammar rules among themselves, each domain possesses a significant count of unique grammar rules, albeit in smaller proportions. Fur- thermore, our LLM-enhanced self-training method not only modifies the distribution of grammar rules within the training instances but also integrates pre- viously unseen grammar rules.
Limitations
because of the expensive cost, we do not employ the GPT-4.It's also intresting that target domain A self-training also improve target B domain con- stituency parsing performance, which plore in the next work. There are many detailed exploration for LLM-equipped self training in the raw corpus generation partition, e.g. the influence of the prompt write by different people. For time constraints, we just chose falcon-40-instruee and the smaller LLMs are also worth try.
will ex-
6 Conlusion
In this study, we introduced an innovative LLM- enhanced self-training method for cross-domain adaptation in constituency parsing. By harnessing the generation of LLMs and integrating them into the self-training process, we showed that our ap- proach considerably enhances constituency parsing performance across various domains. Our method effectively merges high-confidence selection cri-
References
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi. Alessandro Cappelli. Ruxandra Cojocaru. Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: open large language model with state-of-ther the-ar art performance.
Dhananjay Ashok and Zachary C Lipton. 2023. Prompt- Prompting for named entity recognition. arXiv preprint arXiv:2305. 15444.
Model
0
5
10
20
LLM-enhanced ST
83.33
84.12
84.09
84.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung. et 2023. A multi- task, multilingual, multimodal evaluation of chatgpt
Table Table Results for various numbers of target sentences. notes that when target sentence is pro- vided, we supply the LLM with the domain name.

---

on reasoning, hallucination, and interactivity. arXiv preprint arXhv:2302.04023.
Zexue He. Marco Tulio Ribeiro, and Fereshte Khani. 2023. Targeted data generation: Find- ing and fixing model weaknesses. arXiv preprint arXiv:2305.1 17804.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Peijie Jiang, Dingkun Long, Yueheng Sun, Meishan Zhang, Guangwei Xu, and Pengjun Xie. 2021. A fine-grained domain adaption model for joint word segmentation and pos tagging. In Proceedings of the 2021 Conference EMNLP, pages 3587-3598.
Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In 35th Annual Meet- ing of the ACL and 8th Conference of the EACL, pages 16-23.
Nikita Kitaev, Steven Cao. and Dan Klein. 2019. Multi- lingual constituency parsing with self-attention and pre-training. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3499-3505.
Leyang Cui, Sen Yang. and Yue Zhang. 2022. Inves- tigating non-local features for neural constituency parsing. In Proceedings of the 60th Anmual Meeting of ACL (Volume 1: Long Papers), pages 2065-20000000 2075.
Nikita Kitaev and Dan Klein. 2018. Constituency pars- ing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of ACL (Volume 1: Long Papers), pages 2676 6686.268
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan. Michael Auli, and Jason Weston. Wizard of wikipedia: chowledge-poweres conversational agents. In International Conference on Learning Representations.
Jiangming Liu and Yue Zhang. 2017. In-order transition-based constituent parsing. Transactions of the Association for Computational Linguistics, 5:413- 424.
Cailing Dong and Ulrich Schäfer. 2011. Ensemble-style self-training on citation classification. In Proceed- ings of 5th international joint conference on natural language processing, pages 623-631.
Potsawee Manakul, Adian Liusie, and Mark JF Gales, 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.08886
Chris Dyer. Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199-209, ACL.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building large annotated cor- pus english: The penn treebank. Computational Linguistics, 19(2):313-3)
David McClosky, Eugene Charniak, and Mark John- son. 2006. Reranking and self-training for parser adaptation. In Proceedings of the 21 st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 337-344.
Daniel Fried, Nikita Kitaev, and Dan Klein. 2019. Cross-domain generalization of neural constituency parsers. In Proceedings of the 57th Annual Meeting of pages 323-330.
David Gaddy, Mitchell Stern, and Dan Klein. 2018. What's going on in neural constituency parsers. An analysis. In Proc. of NAACL.
David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computationll Linguistics, pages 28- 36.
Peiming Guo, Shen Huang, Peijic Jiang, Yueheng Sun, Mcishan Zhang, and Min Zhang. 2022. Curriculum- style fine-grained adaption for unsupervised cross- lingual dependency transfer. IEEF/ACM Transac- tions on Audio, Speech, and Language Processing. 33222233..
Sheshera Mysore, Andrew McCallum, and Hamed Zamani. 2023. Large language model augmented narrative driven recommendations. arXiv preprint arXiv:2306.02250
Junxian He, Jiatao Gu, Jiajun Shen, and Mare Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. International Conference on Learning Representations.
Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023. Automated annotation with generative ai re- quires validation. arXiv preprint 2xXi:22306.00/77
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pages 507-517.
Qi Peng, Changmeng Zheng, Yi Cai, Tao Wang, Haoran Xie, and Qing Li. 2021. Unsupervised cross-domain named entity recognition using entity-aware adver- sarial training. Newral Networks, 138:68-77.

---

Peng Qi. Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meet- ing of ACL: System Demonstrations.
Bailin Wang. Zi Wang, Xuezhi Wang, Yuan Cao, Rif A Saurous, and Yoon Kim. 2023. Grammar prompting for -ooaainspecific language generation with large language models. arXiv preprint arXiv:2305. 192.4444.
Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Has- san Awadallah. 2021. 1. Meta self-training for few-shot neural sequence laheling. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Duta Mining.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? arXiv preprint mxXiv23302.06770
Xipeng Qiu, ChaoChao Huang, and Xuan-Jing Huang. 2014. Automatic corpus expansion for chinese word segmentation by exploiting the redundancy of web information. In Proceedings of COLING 2014, the 25th International Conference on Computational Lin- guistics:
Yaqing Wang. Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Has- san Awadallah. 2020. Adaptive self-training for few- shot neural sequence labeling, arXiv e-prints, pages arXiv-2010.
Technical Papers, pages 1154-1164.
Alan Ramponi and Barbara Plank. 2020. Neural un- supervised domain adaptation nlp-a survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6838 -8555.
Zhiguo Wang. Haitao Mi, and Nianwen Xue. 2015. Fea- ture optimization for constituent parsing via neural networks. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natu- ral Language Processing ( (Volume I: Long Papers), pages 1147.
Guy Rotman and Roi Reichart. 2019. Deep contex- tualized self-training for low resource dependency parsing. Transactions of the Association for Compu- tational Linguistics, 7:695-713.
Naiwen Xue, Fei Xia. Fu-Dong Chiou, and Marta Palmer. 2005. The penn chinese treehank: Phrase structure annotation of a large corpus. Natural lan- guage engineering, 1 1122)207--33.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- rional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 629-640.
Sen Yang, Leyang Cui, Ruoxi Ning, Di Wu, and Yue Zhang. 2022. Challenges to open-domain con- stituency parsing. In Findings of the A Association for Computational Linguisties: ACI. 2022, pages I 112- 127.
Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D Choi, Richárd Farkas, Jennifer Fos- ter. lakes Goenaga. Koldo Gojenola Galletebeitia, Yoav Goldberg, et al. 2013. Overview of the spmrl 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceed- ings of the Fourth Workshop on Statistical Parsing of Morphologicalll-Riia Languages, pages 146-182, Association for Computational Linguisties.
David Yarowsky. 1995. Unsupervised word sense dis- ambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196.
Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng. and Lidong Bing. 2020. Feature adaptation of pre-traine rre-trained language models across languages and domains for text classification. In Empirical Methods in Natural Language Processing.
Mark Steedman, Miles Osbore,, Anoop Sarkar, Stephen Clark, Rebecca Hwa, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Bootstrapping statistical parsers from small datasets. In 10th conference of the European chapter of the association for computational linguistics.
Juntao Yu, Mohab El-karef, and Bernd Bohnet. 2015. Domain adaptation for dependency parsing via self- training. In Proceedings of the 14th International Conference on Parsing Technologies, pages 1-10.
Mitchell Stern, Jacob Andreas, and Dan Klein, 2017. A minimal span- neural constituency parser. In Proceedings of the 55th Annual Meeting of ACL (Volume 1: Long Papers), pages 818-827.
Ruohong Zhang, Yau-Shian Wang, and Yiming Yang- 2023. Generation-driven contrastive self-training for zero-shot text classification with instruction-tunnd gpt. arXiv preprint arXiv:2304. 11822.
Øyvind Stiansen and Erik Voeten. 2019, ECIHR judg- ments.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. TI; dr: Mining reddit to learn au- tomatic summarization. In Proceedings of the Work- shop on New Frontiers in Summarization, pages 59-.. 63.
Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang. and Jingbo Zhu. 2013. Fast and accurate shift-reduce constituent parsing. In Proceedings of the 51 An- nual Meeting of ACL (Volume I: Long Papers), pages 434-443.

---

Category
Dialogue
Forum
Law
Literature
Review
use the Crawed Raw Corpus
Test
23.24
13.30
1. would not recommend 2. like that the fittings are
Raw Coepos (40K)
18.67
20.72
21.74
20.30
20.56
metal and seem to be very sturdy 3. Then he learns, she was trying to get to her child who is a babyseat in the back 4. Decent movie worth a rental and you are collector buy it 5.
LLM-irrl
23.98
37.08
39.39
37.02
22.91
1-M-t12 (Token)
23.98
37.02
39.51
$7.05
23.06
LLM 5r2 (Conf)
23.97
36.97
39.31
$7.07
22.95
11-6-i12 (GRs)
24.08
36.99
39.28
57.15
22.97
LLM-itz (GRsCoof)
23.82
36.82
38.98
36.82
23.07
LL.M-generated Raw Corpus
LLM 3ir3 (Token)
24.07
36.89
39.27
36.88
23.07
1. The turns of play until end of the game were intense
11M-itr (Cont)
36.80
38.94
36.67
22.92
LLM-itr3 (GRS))
24.01
36.95
39.21
57,08
22.98
instructions
(GRsConf)
product will be given in manual 3.
L.M-inr4 (Token)
23.95
35.78
39,26
$6,95
23.00
itr 2: 1. She major client, and removed the drive from the enclosure hand spelled " love you in sign language but she did n't notice
LLM strd (Conf)
23.96
36.83
38.96
36.81
22.90
how warm it was 2. My
LLM lidd (GRs)
24.03
36.87
39.06
37.12
22.99
LLM-itr4 (GRsConf)
23.91
36.85
38.90
57.01
23.02
itr 3: 1. The movie was comfortable but for corpo- ratism Mama recommended Telephone in February 2. I have a disc that has been dented during delivery. and now need to return it 3.
Table 4: Length statisties for the test set, crawled raw corpus (40k), and LLM-generated raw sentences for each selection strategy (10k per iteration).
1. Very old Spiderman under the cases and easy to place others on top 2. really prefer the normal ones not including those with too many features 3.
A Appendix
A.1 Length of Instances
Figure 6: Comparison of typical examples of crawled and iterative -eM-eeeerdtedd raw corpus.
The length statistics for the test set, crawled raw corpus(40k), and LLn-ggnerate raw sentences for each selection strategy (10k on each iteration) are reported at table 4. It is evident that, from the per- spective of sentence length, the raw corpus used for Vanilla self-training is closer to the target domain data compared the sentences generated by LLM. Interestingly, different selection criteria can affect the average length of the generated sentences. For example, when selecting data based on grammar rules (GRs), the generated sentences tend to be longer, which can be attributed to the fact that the GRs-based criteria prefer sentences contain more complex structures.
LLM
OpenSre
R
P
F
chatGPT
x
82.07
86.26
84.12
falcon
/
82.0
86.
84.05
Table Results of chatGPT and falcon. OpenSre de- notes whether the LLM is open source.
this idea, we use the powerful (Almazrouei et al., 2023) as an alternative open- source LLM in the Review domain based on the GRsConf selection strategy.
In comparison to API requests for chatGPT, the open-source LLM features much faster inference speeds. The experimental results, shown in Table 5, reveal that using chatGPT's prompt generate raw text with Falcon results in strong performance when applied to our iterative method. This finding indicates that incorporating open-source LLMs into our self seff-training approach is also feasible, and the open-source LLM can seamlessly embed into the process in completely closed-loop manner. This eliminates the need for manual intervention in the iteration process, is required with chatGPT, mak- ing the approach more efficient and cost-effective.
A.2 LLM Generated Sentences
We have selected several crawled raw sentences from domain Review GRsConf selection strategy, which are concise, readable, and more colloquial. In our improved self-training method, LLM gen- erates raw corpora for each iteration, as shown in Figure These sentences incorporate more diverse information and exhibit more complex structures; however, there may be instances where the seman- tics are not that smooth or coherent.
A.3 Open Source LLM Exploration
Obtaining raw corpus for each iteration using closed-source LLMs, such as chatGPT, can be time- consuming and costly. Therefore, we investigate the possibility of employing open-source Large Language Models to achieve similar effects. To test


Reviews:
[{"review_id": "aWS1YpuyGc", "reviewer": "Reviewer_8ZMA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments. \n\nQ0: Evaluation for Available Target Text Size  \nIn the pre-experiment, we simply tested performance using 5, 10, and 20 available target texts. There are no significant differences on these three settings. Considering resource limitations, we select 5 samples for efficiency. In the final version, we will conduct a more extensive analysis, delving into the influence of different target text quantities on LLM-generated text, including diversity. \n\nQ1: Evaluation for Grammar Rules Effect  \nIn the revised manuscript, we will dedicate a section to conducting experiments that exclude grammar rules from the prompt generation process and examine the effect of grammar rules in our method.\n\nQ2:  Improved Implementation Detail  \nWe will refine the implementation details in the camera-ready version. The LLM parameter information will be relocated from Section 4.3 to Section 3.5 for clearer presentation. We will improve the details in demonstrating the instance selection criteria and ensure its clarity in future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The investigation of using LLM to generate target domain text for domain adaptation in the self-training setting is an interesting problem, and the paper demonstrates some positive results despite the small improvement.", "reasons_to_reject": "- Since the proposed LLM-based approach aims to benefit from low-resource target domains, it is necessary to evaluate how the size of available target text affects performance. However, the paper only investigates one fixed amount of target text, leaving important questions unanswered. For example, does the amount of available target text affect the diversity of the generated text and impact the effectiveness of the proposed approach? Would this approach be more effective if you further reduce the amount of target text?\n- The paper emphasizes using grammar rules as prompts for LLM to generate target-domain text, but no experiments were conducted to evaluate the benefits it brings.\n- The writing could be improved to provide more implementation details. For instance, the hyperparameters used for constructing LLM prompts in section 3.5 are not detailed. The implementation details of the instance selection criteria are not provided, such as the confidence measure and how it is combined with the grammar-rule-based selection approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper explores the use of LLM to create target-domain text for improving constituency parsing in a cross-domain self-training setting. The approach involves an iterative process where an initial baseline constituency parser (berkely-neural-parser) is trained on a source domain. LLM is then prompted with grammar rules from the current set of source+pseudo trees and sample sentences from the target domain to generate new text in the target domain. A selection process with certain criteria is used to choose 'high-quality' samples as pseudo training data for the next round of parser training. The experiments conducted on the PTB->MCTB cross-domain setting demonstrate that the proposed LLM-based self-training approach achieves slightly better performance (0.1~0.2 improvement in F1 score) compared to a vanilla ST approach that has access to a small fixed set of target text. Some analysis shows that the LLM-based approach consistently improves during the iteration process, and the selected pseudo text gradually becomes more similar to the target domain.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uWac7Sk91J", "reviewer": "Reviewer_wYfy", "summary": "", "questions": "1. How exactly are the grammar rules extracted? I am not sure I fully grasp the method.\n2. In lines 385, 386, lenght refers to tokens? It should be specified.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the insightful comments.\n\nQ0: Weak Experimental Setup and Small Improvements  \nWe have already performed an statistical significance analysis where p<0.05 on three different random seed. The results support our claim that our LLM-enhanced self-training can outperform the strong baseline (vanilla self-traning). We will update the results in the revised manuscript.\n\nQ1: Grammar Rules Extraction  \nThe grammar rules are directly extracted from the constituent tree, where the parent node corresponds to the left hand of the grammar rule, and all child nodes correspond to the right tail side. In the camera-ready version, we will ensure that readers can fully understand the extraction method without ambiguity.\n\nQ2: Clarification on \"length\" in Lines 385-386  \nThank you for pointing out the need for clarity regarding the term \"length\" in Lines 385-386. In the revised version, we will specify that \"length\" refers to the number of tokens, providing readers with precise information to better understand our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Method and discussion of results are described in detail.\n2. Simple and elegath proposed method.\n3. Comparison of open- and closed-source language models.", "reasons_to_reject": "1. Weak experimental setups. Improvements are small and without a statistical significance analysis, it is difficult to extract conclusions from them.", "questions_to_authors": "1. How exactly are the grammar rules extracted? I am not sure I fully grasp the method.\n2. In lines 385, 386, lenght refers to tokens? It should be specified.", "paper_topic_and_main_contributions": "The authors propose a LLM-enhanced self-training for constituency parsing on different domains. Using causal LLMs, they propose an iterative method consisting in generating new training instances on a target domain using grammar-rules extracted from a source domain treebank and few samples on the target domain. Then they select the synthetic samples with higher confidence and update the source treebank with these silver data, and udpate the grammar rules with the updated treebank. \n\nOverall, the proposed approach is interesting and obtains better results than the baselines. However, experimental setup is a bit weak. Although they perform a detailed discussion on results, thee improvements are small and are not controlled for seed randomization or tested their statistical significance, so the claims they made are not really supported.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sgs05Ny7xi", "reviewer": "Reviewer_imit", "summary": "", "questions": "Lines 351-354 I did not quite understand how you sample the grammar rules from PTB. How do you derive the parameters?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. \n\nQ0: Not Superior Performance  \nit's important to highlight that self-training inherently serves as a strong baseline for cross-domain tasks. Additionally, we've conducted a statistical significance analysis that yielded results with p<0.05, reinforcing our claim that our LLM-enhanced self-training surpasses the strong baseline (vanilla self-training). We will revise and underscore this aspect in the revised manuscript.\n\nQ1: Noticeable Disadvantage for Different Grammar Rules  \nWe've examined the grammar rules distribution of the MCTB dataset compared with source PTB train set and found that different grammar rules exist in all the domains. However, our method can predict the grammar rules never seen in the training set and self-training transfers gradually grammar-rule distribution from the source domain to the target domain. We will provide a more comprehensive analysis of different grammar rules in the revised manuscript.\n\nQ2: 2K for Vanilla and 10k for Our Method  \nFor all methods, vanilla self-triaining and LLM-enhanced self-training, we added 2k pseudo trees from a pool of 10k pseudo data. We will clarify this setup further in the camera-ready version.\n\nQ3: Confusion on Standard Self-training Scheme  \nIn fact, what we referred to as \"vanilla self-training\" aligns with what you mentioned as the \"standard self-training scheme.\" For the vanilla self-training, unsupervised samples are from the same \"homologous\" resources not the other \"homologous\" resources. The sentences of MCTB are selected from the 5 resources (line 377~382) which also served as our homologous resources. We will make this aspect clearer in future version.\n\nQ4: Redundant Open-source LLM in 5.3  \nWe will relocate Section 5.3 along with Table 2 to the appendix. The comparison experiments involving open-source LLM are limited to one domain due to resource constraints, which couldn't be included directly in Table 1 encompassing five domains. This adjustment in space will provide room for more insightful analysis, as you suggested.\n\nQ5: Typos Grammar Style And Presentation Improvements  \nIn the revised version, we will improve all the writing and stylistic issues you've raised for improvement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, I really like the proposed method. It is quite simple and effective and marries together the timelessly self-training technique with the novel capabilities of LLMs. The paper also nicely combines the training of standard classifiers with pseudo-generated samples. In general, I would like to see more papers of this kind that find novel ways to extend standard techniques with LLMs.\n\nThe newly proposed selection methods are also refreshing but don't seem to contribute much to overall performance.\n\nFigures 4 and 5 hold informative analyses and highlight the strengths of the proposed method over baselines.", "reasons_to_reject": "The proposed approach assumes that the source and target samples obtain similar grammar rules, but it would fail on target domains that obtain different grammar rules, not to mention, sentences from other languages. This is a noticeable disadvantage.\n\nThe thing that bothers me the most, if I understand correctly, the authors did not add the same amount of new training samples at each iteration - for vanilla self-training they added 2K samples, while for their method they added 10K samples. This is not a fair comparison and severely harms the results. It isn't clear if the authors then select 2K samples out of the total of 10K generated samples or not. This affects both Table 1 and Figure 4.\n\nSecondly, the authors choose to test all models on the MCTB dataset which only consist of test samples. For supplying unsupervised samples for the vanilla self-training they turned to other \"homologous\" resources.\nHowever, it is very common, even for open-domains, to have a decent amount of available unlabeled samples. In most cases, self-training directly operates over unlabeled target samples (which do not appear in the held-out test set). I would like to see experiments on such a scenario based on a target dataset that was manually split to train and test samples. That way, the authors could test their method against the standard self-training scheme.\n\nOther than that I find section 5.3 and table 2 to be redundant. The authors could have used an open-source LLM in advance instead or in addition to the reported results on GPT3.5-turbo in Table 1. Or, they could have put it in the appendix. I believe the occupied space could have been used better for a more insightful analysis.", "questions_to_authors": "Lines 351-354 I did not quite understand how you sample the grammar rules from PTB. How do you derive the parameters?", "paper_topic_and_main_contributions": "The paper presents a novel way to support open-domain transfer for consistency parsing by combining the famous self-training technique with the help of LLMs. The authors propose to generate target domain examples using an LLM given a prompt that contains their grammar rules and raw examples of target sentences. The generated samples with their corresponding trees are then added to the training set and the parser is trained iteratively for 4 iterations. \n\nThe authors also propose new methods for self-training data selection using similarity of token-based distribution and similarity of grammar rules and combine them with the known confidence-based method.\n\nResults show competitive but not superior performance of their methods, which weakens the overall claim.\n\nThe authors also demonstrate their ability to seamlessly integrate their method with open-source LLMs such as Falcon-40B.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 250: the abbreviation of GRs is not yet defined.\n\nline 256: D_hat = \\empty_set.\n\nline 421 - chatGPT (overall it is very confusing that you use both chatGPT and GPT3.5-turbo for referring to the same model)\n\nTable 1 - please bold the highest numbers in each column to ease the reading.\n\nline 489 - The most...\n\n"}}]

Author rebuttal:

Meta-review: The paper presents an interesting approach to improve cross-domain parsing, with a clever way to use LLMs, producing small gains over reasonable baselines. The reviews found the use of LLMs to produce target domain text for self-training to be an appealing idea, and suggested several ways in which the paper could be strengthened, including the inclusion of results over multiple runs with random seeds, and clarification of details such as whether the size of available target texts affects performance, whether different data amounts are used in vanilla vs LLM-enhanced self-training, etc. The responses provided by the authors seem satisfactory.
Decision: Accept-Main