Large Language Models Are No Longer Shallow Parsers (2024)

Abstract: The development of large language models (LLMs) brings significant changes to the field of natural language processing (NLP), enabling remarkable performance in various high-level tasks, such as machine translation, questionanswering, dialogue generation, etc., under endto-end settings without requiring much training data. Meanwhile, fundamental NLP tasks, particularly syntactic parsing, are also essential for language study as well as evaluating the capability of LLMs for instruction understanding and usage. In this paper, we focus on analyzing and improving the capability of current state-of-the-art LLMs on a classic fundamental task, namely constituency parsing, which is the representative syntactic task in both linguistics and natural language processing. We observe that these LLMs are effective in shallow parsing but struggle with creating correct full parse trees. To improve the performance of LLMs on deep syntactic parsing, we propose a three-step approach that firstly prompts LLMs for chunking, then filters out low-quality chunks, and finally adds the remaining chunks to prompts to instruct LLMs for parsing, with later enhancement by chain-of-thought prompting. Experimental results on English and Chinese benchmark datasets demonstrate the effectiveness of our approach on improving LLMs’ performance on constituency parsing.1

heading: 1 Introduction
text: Recently, the development of large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023a,b; Taori et al., 2023; Chiang et al., 2023) significantly reshapes the field of natural language processing (NLP), where they achieve impressive performance on a variety of NLP applications such as question-answering and dialogue interaction. Nevertheless, in the era of LLM, fundamental NLP
†Corresponding author. 1Resources of this paper are available at https://github.
com/synlp/LLMPar.
tasks that involve linguistic analysis still play essential roles in the field of NLP (Mahowald et al., 2023; Blevins et al., 2023). For example, syntax parsing provides structural understanding of sentences and thus is valuable in evaluating the capability of LLMs to understand and use instructions (McCoy et al., 2023; Muñoz-Ortiz et al., 2023; Blevins et al., 2023).
In general, syntactic parsing is an essential NLP task that has been studied for decades (Collins, 1997; Glaysher and Moldovan, 2006; Cai et al., 2009; Liu and Zhang, 2017; Kitaev and Klein, 2018; Tian et al., 2020b, 2022; Gu et al., 2022) and has two widely used conventional variants with phrase-structure and dependency grammar, of which the former drives constituency parsing that provides deep structural analysis of sentences with informative phrase and function markups. Recent studies (Bai et al., 2023; Lin et al., 2023) reveal that LLMs are ineffective in such parsing task, and we also confirm the fact in Figure 1 with parse trees generated by GPT-4 (Achiam et al., 2023) under various prompts, where they all look very different from human-crafted gold-standard trees, indicating syntactic parsing is challenging to LLMs. Although some studies (Blevins et al., 2023; Bai et al., 2023; Li et al., 2023) design prompts to instruct LLMs for better parsing, it is still limited in performing one-step (i.e., end-to-end) parsing, which does not fully benefit from the advantages brought by the strengths of LLMs.
In this paper, we explore the effectiveness of directly using state-of-the-art LLMs for constituency parsing, and find that they generate parse trees that are shallow and with relatively low recalls on retrieving correct text spans, while in the meantime LLMs show relatively better performance in chunking, especially in identifying short chunks. Therefore, based on our analyses and motivated by existing studies that decompose complex tasks into several steps to improve LLM’s performance
7131
(Wei et al., 2022; Valmeekam et al., 2023; Zhao et al., 2023; Guan et al., 2023; Ranaldi and Zanzotto, 2023), we propose a three-step approach to improve LLMs for parsing. The approach firstly prompts LLMs to extract chunks in the text, which is considered partial parsing and is demonstrated to be an essential step for full parsing (Gu et al., 2022; Yang and Tu, 2023). Then, our approach filters out long chunks that are more likely to carry noise than the short ones. Finally, it uses the remaining chunks as conditions in the prompts to instruct LLM for parsing, allowing LLMs to directly use the chunks to generate the parse tree of a sentence. We validate our approach on several benchmark datasets, including English Penn Treebank 3 (PTB) (Marcus et al., 1993), Chinese Penn Treebank 5 (CTB5) (Xue et al., 2005), and Genia (Tateisi et al., 2005), where the results prove the effectiveness of our approach over existing LLMs used directly for constituency parsing.
The contributions of this paper are summarized as follows. First, we provide comprehensive analyses of existing state-of-the-art LLMs on constituency parsing. Second, we propose a threestep approach that involves chunking, filtering, and parsing, to improve LLMs for constituency parsing. Third, the experiment results and further analyses demonstrate the effectiveness of the proposed approach for constituency parsing.

heading: 2 Analysis of Current LLMs for Parsing
text: In order to understand the issues and improvement directions in existing LLMs for parsing, we analyze
the performance of LLMs, which can be divided into two groups. The first group includes GPT3.5 and GPT-4, which are accessible only through online API. The second group contains publicly available LLMs; namely, the 7B and 65B versions of LLaMA-2 for English (Touvron et al., 2023b), and the 7B and 13B versions of Alpaca for Chinese (Cui et al., 2023) (we denote these English and Chinese LLMs as LLaMA for the sake of convenience).2
We use these LLMs to parse the test sets of PTB, CTB5, and Genia, where PTB and CTB5 are general domain English and Chinese datasets, respectively, and Genia is an English medical dataset. The statistics of the datasets are reported in Table 1. To perform efficient evaluation, we randomly
2We obtain English “Llama2-chat-hf ” from https://huggingface.co/meta-llama and the Chinese “Chinese-Alpaca-2” from https://github.com/ ymcui/Chinese-LLaMA-Alpaca-2, respectively.
sample 2,000 sentences from Genia dataset and run experiments on it.
Generally, there are two approaches to linearizing a parse tree into a string to facilitate LLM processing (Vinyals et al., 2015; Liu and Zhang, 2017). The first is the bracket-based approach, which uses brackets to represent the structure of a parse tree, such as “(S (NP he) (VP jumps))”. The second is the transition-based approach that utilizes the sequence of transition actions to construct the parse tree, such as “SHIFT he; NP; SHIFT jumps; VP; REDUCE; S”. According to Bai et al. (2023), when working with LLMs, the bracket-based approach outperforms the transition-based one; therefore, we adopt the bracket-based representation in our study.
Motivated by Bai et al. (2023), we design prompts (see Table 2)3 to instruct LLMs to generate the parse trees of sentences under the five-shot setting. The demonstration examples are (sentence, parse tree) pairs and can be obtained by various methods such as random sampling from a treebank or automatic generation with rules. When LLMs produce invalid parse trees with mismatched left or right brackets, we add a post-processing step to ensure bracket pairing by adding the necessary left or right brackets to the parse tree until they are matched. Finally, LLM outputs are evaluated with the standard evaluation toolkit EVALB4.
3For Chinese datasets, we translate the English prompts into Chinese to instruct the LLMs.
4https://nlp.cs.nyu.edu/evalb/

heading: 2.1 Parsing Results
text: We report parsing performance by labeled precision, recall and F1 scores in Table 3. For each corpus, the top part (above the horizontal lines) are the results reported in previous studies, and the bottom part are ours when running the LLMs under the five-shot setting. It is shown that the performance of LLMs is much lower than the parsers (marked by "*") that are based on BERT (Devlin et al., 2019) and fine-tuned on the training set. In particular, publicly available LLaMA models with various sizes are unable to produce valid parse trees, confirming the observations in Bai et al. (2023). Moreover, both GPT-3.5 and GPT-4 tend to produce parse trees that are shallower than the gold-standard trees (illustrated in Figure 1) and their precision is higher than the recall.
In addition, we use Figure 2 to show the average depth of gold-standard (i.e., the blue curve) and GPT4-generated (i.e., the orange curve) parse trees with respect to sentence length. On average, the LLM-generated trees are shallower than the gold-
standard ones. Interestingly, while the depth of the gold standard parse tree increases roughly linearly with the increase of sentence length, that of the LLM-generated trees tends to plateau for long sentences, indicating that LLMs have difficulties in generating deep structures.
Overall, our analysis of parsing results show that LLMs are ineffective in producing high-quality full parse trees.

heading: 2.2 Chunking Results
text: Previous studies (Wei et al., 2022; Valmeekam et al., 2023; Zhao et al., 2023; Guan et al., 2023; Ranaldi and Zanzotto, 2023) have shown that decomposing complex tasks into several steps is able to improve the performance of LLMs. In this study, we want to test whether the same approach is applicable to parsing.
To decompose the parsing task, we first examine how well LLMs perform at chunking. Here, the term “chunk” refers to a text span covered by an internal node in a parse tree. Unlike the standard chunking task where the chunks in the output should not overlap, the goal of the chunking task in this study is to identify all the chunks in a sentence. Our chunking experiments use the same datasets, LLMs and five-shot setting as in our parsing experiments. The only difference is the prompt for parsing is replaced by the prompt in Table 4. For demonstration examples, we extract chunks with various lengths from treebanks.
The chunking results are in Table 5, which also includes the results of the supervised approach with the BERT-base encoder. Not surprisingly, LLMs achieve better performance on chunking than parsing (Table 5 vs. Table 3). Furthermore, as shown in Figure 3, LLMs are better at identifying shorter chunks than longer ones.5

heading: 3 Adapting LLMs for Parsing
text: Based on experimental results on parsing and chunking, we hypothesize that LLMs’ performance on parsing can be improved by decomposing the parsing task into two steps: first, LLMs are asked to chunk the input sentence; second, LLMs are asked to produce a parse tree based on a list of chunks. To alleviate the effect of error propagation in a multistep system like this, after the chunking step, we filter out low-quality chunks and send the remaining ones to the parsing step, as illustrated in Figure 4. As LLMs are better at identifying short chunks than the long ones, the filtering step will simply filter by the chunk length. We also propose chainof-thought (CoT) prompting (Wei et al., 2022) in
5Figure 3 shows the F-scores for chunks with length 16+ are higher than chunks with 11-15 words. This is due to the fact that our calculation includes chunks that cover the entire sentences; those chunks tend to have 16+ words and can be identified by LLMs pretty accurately.
the third step for further enhancement.

heading: 3.1 Three-step Parsing with LLMs
text: The details of our parsing procedure are illustrated as follows. In Step 1, we prompt LLMs (e.g., using the one presented in Table 4) to annotate chunks in various word-based lengths following the five-shot setting. In Step 2, we filter out chunks whose wordbased length is larger than a threshold. We set that threshold to be 5, based on the results from Figure 3. In Step 3, we utilize the chunks as knowledge to enhance the performance of natural language processing models, which is demonstrated to be effective in existing studies (Song et al., 2017; Zhou and Zhao, 2019; Zhang et al., 2019a,b; Yang et al., 2019; Tian et al., 2020a; Qin et al., 2021; Ouyang et al., 2022; Song, 2022). Specifically, we add the chunks to the prompt as conditions to instruct LLMs for parsing. In this step, we do not want LLMs to treat the given chunks as hard constraints when producing the parse trees; thus, we ask LLMs to refer to the chunks without having to use all of them. This way, LLMs are allowed to ignore
the errors that could exist in the provided chunks, alleviating the impact of error propagation.

heading: 3.2 CoT for Parsing
text: In the aforementioned third step, to provide more detailed instructions for LLMs to leverage chunks for parsing, we perform CoT prompting. Specifically, to provide CoT demonstration, we automatically obtain step-by-step instructions from goldstandard parse trees through the following process. For each internal node N in the parse tree, we find all its children nodes C1 · · ·CM and the corresponding text spans s1 · · · sM . Then we add “s1, s2, · · · , sM are combined into a new text span s” (where s is the concatenation of s1 · · · sM ) into the demonstration, which serves as a part of the prompt. This process instructs LLMs to leverage chunk information when processing each test instance. In practice, s1, s2, · · · , sM include single
words; we only add the case where s1, s2, · · · , sM appear in the filtered chunks from Step 2.
Let us use the example in Figure 4 to demonstrate the process. Here, the provided chunks are “(NP the Dow)”, “(NP 35 points)”, and “(NP the 35 points)”. In the gold-standard parse tree, the PP node that covers “down about 15 points” has two children, namely, the single word “down” and an NP phrase “about 35 points”. Since the phrase “about 35 points” appears in the chunk list, we combine “down” and “(NP about 35 points)” and get a new chunk “(PP down about 15 points)” in the CoT part. In contrast, for the VP node that covers “was down about 15 points”, we do not include it in CoT since one of its children, the phrase “down about 15 points” does not appear in the chunk list.

heading: 4 Analysis

heading: 4.1 Main Results
text: We evaluate our approach on the benchmark datasets with five-shot learning, and the results are in Table 6. For Step 3, we show the results with or without CoT. There are two main observations. First, CoT improves system performance across the board, which confirms the benefits of providing step-by-step guidance on how chunks should be used during the parsing step.
Second, compared with the results of directly using LLMs for parsing in Table 3, our approach, with or without CoT, achieves significant improvements on all datasets for all four LLMs. Particularly, the improvements on recall are remarkable, which indicates that our approach is able to prompt LLMs to correctly identify more chunks and include them in parse trees. A possible explanation for the improvement is that, by separating chunking from parsing, the LLMs in Step 1 are able to focus on the shallow parsing, which they are good at. Thus, this step allows more correct chunks to be identified. In the parsing step, LLMs pay more attention to combining these chunks to construct a valid parse tree. In addition, we alleviate the risk of the error propagation by filtering out long chunks and asking LLMs to treat the given chunks as soft constraints. All these factors may contribute to the better performance of the 3-step approach over the 1-step direct parsing approach.
Furthermore, as shown in Figure 2, on average, the parse trees generated by our approach are deeper than the ones generated by the 1-step approach and the curves for our approach are similar
to the ones based on gold-standard parse trees. This indicates that LLMs are no longer shallow parsers, when used appropriately.

heading: 4.2 Effect of Chunking
text: In Step 3, LLMs are given a chunk list and are instructed to refer to the list but do not have to use all the chunks in the list when producing the parse tree. We want to know how many chunks in the list are actually used by LLMs. Let A be the set of chunks that appear in the parse tree produced by
an LLM. Let B be the set of chunks provided to the LLM in Step 3’s prompt. Let B1 be the set of chunks in B that appear in the gold-standard parse tree for the sentence. We compute two metrics: the first one, UC (the percentage of "Used Chunks"), is the percentage of the provided chunks that are used by the LLM; that is, UC =| A ∩ B | / | B |. The second metric, UCC (percentage of "Used Correct Chunks"), is the percentage of the gold chunks that are used by the LLM; i.e., UCC =|
A ∩ B1 | / | B1 |. Table 7 show the results of two representative LLMs; namely, LLaMA and GPT-4 (one open-source and one closed-source LLM). The table indicates that in Step 3 LLMs use the majority of chunks provided to them, and the percentage of chunks used is even higher among the gold chunks in the provided chunk list.
What would happen if we require LLMs to use all the chunks provided to them? The results under the no-CoT setting are in Table 8, which are worse than the corresponding rows in Table 6. In other words, forcing LLMs to use all the chunks hurts the parsing performance; therefore, the provided chunk list should indeed be treated as a reference, not a hard constraint.
Finally, we analyze the effect of the chunk length threshold used in Step 2. Figure 5 shows the parsing performance with different threshold values. The figure shows that the system performance improves a lot when the threshold value increases from 1 all the way to 5; afterwards, parsing performance quickly plateaus as bigger chunks identified in Step 1 are more error-prone and thus offer limited help.

heading: 4.3 Case Study
text: As previously shown, our 3-step approach outperforms the baseline system where LLMs produce the parse tree in one step ("the GPT-4 baseline"). To better understand where the gain comes from, we look at parse trees produced by these approaches and two examples are in Figure 6. In the first example, in the gold-standard parse tree, the VP “called it simply a contrast in styles” has a deep syntactic structure, where as the GPT-4 baseline simply combines all the words together to construct a flat
VP. In our system, GPT-4 identifies chunks such as "(PP in styles)" and "NP simply a contrast in styles" in Step 1. When they are provided to GPT-4 in Step 3, GPT-4 is able to produce a parse tree with much deeper structure.
In the second example, the GPT-4 baseline fails to combine the word “ago” and the NP phrase “all of three months” to form an ADVP. In our approach, GPT-4 produces chunks such as “all of three months ago” in Step 1. When they are passed to GPT-4 in Step 3, GPT-4 is able to build the correct parse tree.

heading: 5 Related Work
text: Constituency parsing is a fundamental NLP task that is generally performed via chart- or transitionbased approaches (Collins, 1997; Sagae and Lavie, 2005; Glaysher and Moldovan, 2006; Song and Kit, 2009; Dyer et al., 2016; Liu and Zhang, 2017; Stern et al., 2017; Kitaev and Klein, 2018; Suzuki et al., 2018; Fried et al., 2019; Zhou and Zhao, 2019; Tian et al., 2020c; Yang and Deng, 2020; Zhang et al., 2020; Nguyen et al., 2021; Yang et al., 2022; Gu et al., 2022), where extra resources are leveraged to improve the parsing performance (Mrini et al., 2019; Maveli and Cohen, 2021; Yang et al., 2022; He and Choi, 2023; Shayegh et al., 2023).
Recently, LLMs have achieved remarkable success in many NLP tasks, which motivates researchers to apply them to parsing. Existing approaches that leverage LLMs for parsing generally follow the sequence-to-sequence settings (Vinyals et al., 2015; Suzuki et al., 2018), where the goal of LLMs is to produce the parse tree in the bracket format. The LLM-based approaches fine-tune LLMs on the training data or design prompts to instruct
LLMs to generate valid parse trees (Bai et al., 2023; Li et al., 2023). For example, Bai et al. (2023) directly train LLaMA (Touvron et al., 2023a) on the training set of PTB and show inferior performance than the conventional chart- and transitionbased approaches that leverage small pre-trained language models such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019). They also perform zero-shot and few-shot parsing but the results are unsatisfactory, which confirms the ineffectiveness of LLMs for parsing. There are studies that select appropriate training data (Li et al., 2023) to further improve constituency parsing with LLM.
In this study, we propose to break parsing into three steps: chunking, filtering chunks, and parsing with chunks, and demonstrate this 3-step approach improve system performance significantly.

heading: 6 Conclusion
text: In this paper, we apply LLMs to parsing and observe that the current state-of-the-art LLMs are shallow parsers in that they are effective in perform-
ing simple syntactic tasks such as chunking but are ineffective at conducting full constituency parsing due to their auto-regressive nature of text modeling approach. Given this, we propose to break parsing into three steps: chunking, filtering, and parsing with a chunk list. Experiments on English, Chinese, and domain benchmark datasets show that the 3- step approach enables LLMs to produce deeper and better parse trees and thus they are no longer shallow parsers.
This study demonstrates the benefits of decomposing a complex task (e.g., parsing) into subtasks and the importance of including additional mechanisms (e.g., filtering chunks in Step 2 and treating given chunks as soft constraints in Step 3) to alleviate the risk of the error propagation. We plan to apply this idea to other NLP tasks in the future.

heading: Acknowledgements
text: This work is supported by the National Key Research and Development Program of China under the grant (2023YFC3303800).

heading: Future Work
text: We plan to apply this idea to other NLP tasks in the future