Empirical Study of Zero-Shot NER with ChatGPT
Authors: Tingyu Xie
Qi Li
Jian Zhang
Yan Zhang
Zuozhu Liu
Hongwei Wang
Venue: EMNLP 2023 Main (2023)
Keywords: ChatGPT
Named Entity Recognition
Zero-Shot
Reasoning
Large Language Models
TL;DR: A systematic empirical study of ChatGPT's reasoning capabilities on zero-shot NER.

Abstract: Large language models (LLMs) exhibited powerful capability in various natural language processing tasks. This work focuses on exploring LLM performance on zero-shot information extraction, with a focus on the ChatGPT and named entity recognition (NER) task. Inspired by the remarkable reasoning capability of LLM on symbolic and arithmetic reasoning, we adapt the prevalent reasoning methods to NER and propose reasoning strategies tailored for NER. First, we explore a decomposed question-answering paradigm by breaking down the NER task into simpler subproblems by labels. Second, we propose syntactic augmentation to stimulate the model's intermediate thinking in two ways: syntactic prompting, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic information generated by a parsing tool. Besides, we adapt self-consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types. The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chinese and English datasets, and on both domain-specific and general-domain scenarios. In addition, we present a comprehensive analysis of the error types with suggestions for optimization directions. We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs.

Empirical Study of Zero-Shot NER with ChatGPT
Tingyu X Qi Li Jian Zhang"?, Yan Zhang Zuozhu Liu?, Hongwei Wang 1.2+
CCollege of Computer Science and Technology, Zhejiang University, China *ZJU-UIUC Institute, Zhejiang University, China National University of Singapore, Singapore
(tingyuxie, liqi177, 12221038) (11111088@jjj.uuuu. @zju.edu.cn, eleyanz www.:/@uuuee....... zuozhuliu@inn www.:iiiinnn..n....... hongweiwang wwwpaannng.u.e....nn
Abstract
by GPT3.5 and GPT4 (OpenAI, 2023), has ex- hibited strong dialogue capabilities. As closed model, ChatGPT sparked lot of work for its eval- uation and application on diverse tasks and aspects (Qin et al., 2023; Wei et al., 2023; Liang et al.. 2023).
Large language models (LLMs) exhibited pow- erful capability in various natural language pro- cessing tasks. This work focuses on explor- ing LLM performance on zero-shot informa- tion extraction, with a focus on the ChatGPT and named entity recognition (NER) task. In- spired by the remarkable reasoning capability L.M on symbolic and arithmetic reason- ing, we adapt the prevalent reasoning meth- ods to NER and propose reasoning strategies tailored for NER. First, we explore a decom- posed question- weateirnn paradigm by break- ing down the NER task into simpler subprob- lems by labels. Second, we propose syntactic augmentation to stimulate the model's interme- diate thinking in two ways: syntactic prompt- ing, which encourages the model to analyze the syntactic structure itself, and tool augmentation, which provides the model with the syntactic in- formation generated by parsing tool. Besides, we adapt self- consistency to NER by proposing a two-stage majority voting strategy, which first votes for the most consistent mentions, then the most consistent types. The proposed methods achieve remarkable improvements for zero-shot NER across seven benchmarks, including Chi- nese and English datasets, and on both domain- specific and general-domain scenarios. In ad- dition, we present a comprehensive analysis of the егrог types with suggestions for optimiza- tion directions, We also verify the effectiveness of the proposed methods on the few-shot setting and other LLMs.!
Information extraction (IE) is fundamental topic NLP, which aims to extract structured in- formation from unstructured text, including tasks such as named entity recognition (NER) (Yu et al., 2020), relation extraction (RE) (Baldini Soares et al., 2019), event extraction (EE) (Chen et al.. 2015), etc. Evaluating ChatGPT's performance on IE is important for understanding its capabilities in structured prediction and language understanding (Li al., 2023; Wei et al., 2023).
With recent techniques for eliciting complex multi-step reasoning (Wei et al., 2022; Wang et al., 2022b), LLMs have shown remarkable zero-shot reasoning ability in arithmetic and symbolic reason- ing (Kojima et al., 2022). However, the reasoning ability of LLM on IE remained unexplored. To mit- igate this gap, we present a systematic empirical study exploring the reasoning capability of L.LM on IE, with a focus on the ChatGPT and zero-shot NER task. By adapting the prevalent reasoning techniques (Zhou et al., 2022; Wei et al., 2022; Wang et al., 2022b) to NER, we propose the follow- ing strategies to stimulate the reasoning potential of LLM on NER:
1 Introduction
We break down the NER task into a se- ries of simpler subproblems by labels and perform (Deconposed-QAA) paradigm, where the model extracts entities of only one label a time.
Large language models (LLMs)) (OpenAI, 2022;
Thoppilan et al.. 2022; Chowdhery et al., 2022) have brought revolutions in natural language pro-
cessing (NLP) due to the remarkable zero-shot and few-shot generalization. One of the most well- known LLMs, ChatGPT (OpenAI, 2022) powered
We propose syntactic augmentation of two ways: syntactic prompting, which encour- ages the model to first analyze the syntactic structure of the input text itself, then recog-
*Corresponding authors. 'Code available at: https: htt.///gtttubccoo wwt.//ooo/me/eeeeeeeeeemm Prro-Shot-NER wwiihhhhhhGPPP

---

nize the named entities based on the syntactic structure; tool augmentation. which provides the syntactic information generated by a pars- ing tool to the model.
2023; Laskar al., 2023) IE tasks. Wei et al. (2023) propose two-stage chatting paradigm for IE. At stage one, asks ChatGPT to recognize the types of elements; at stage two, asks ChatGPT to extract the mentions corresponding to each type recognized at stage one. Han et al. (2023) presents an analysis of ChatGPTT's performance on IE tasks from four aspects: performance, evaluation criteria, robustness, and errors. Wang et al. (2023) apply in-context learning (ICL) to NER by inserting spe- cial tokens into the demonstrations retrieved from the training set. Wan et al. (2023) apply CoT to relation extraction (RE) and use ChatGPT to gen- erate intermediate rationales for demonstrations retrieved from the training set. Different from pre- vious works, we focus on exploring the ChatGPT abilities for zero-shot reasoning IE, with fo- cus on the NER task. We explore the prevalent reasoning methods with LLM, which exhibited re- markable performance on arithmetic and logical reasoning tasks. Most importantly, these methods are first adapted to the NER task based on the task characteristics.
We tailor the self-consistency (SC) (Wang et al., 2022b) for NER and propose two- stage majority voting strategy: after sam- pling multiple responses of the model, we first vote for the most consistent mentions, then the most consistent types.
The main contributions of this paper include:
We present a systematic empirical investiga- tion of zero-shot NER with LLM, with a spe- cific emphasis on ChatGPT as one of the most robust LLMs available.
We adapt prevalent reasoning methods to NER and propose four strategies tailored for NER: decomposed- QA, syntactic prompting, tool augmentation, and two-stage majority voting.
We evaluate our strategies across seven bench- marks. Experiment results reveal that the proposed strategies significantly facilitate zero-shot NER across domain-specific out- of-distribution and general-domain datasets, including Chinese and English scenarios.
Method
Adapting the prevalent reasoning techniques (Zhou et al., 2022; Wei et al.. 2022; Wang et al., 2022b) to NER, we propose four strategies to stimulate the reasoning capabilities of LLM on NER. Examples of the proposed methods are shown in Fig. 1.
2 Related Work
2.1 Reasoning with LLM
LLM has shown remarkable zero-shot reasoning ability, in the way of explicitly encouraging the LLM to generate intermediate rational for solving problem. On the one hand, recent works, in both the few-shot (Wei et al., 2022; Zhang et al., 2022; Wang et al., 2022a) and zero-shot (Kojima al., (CoT) from
3.1 Decomposed-QA
Inspired by least-to-mont prompting (Zhou et al., 2022), we improve zero-shot NER by decomposing the task into a set of simpler questions. Recogniz- ing entities of all labels at one time may be too challenging for ChatGPT (as the vanilla zero-shot method shown in (a) of Fig. 1), especially when the label size is large, or the data is from specific out-of-distributio domain. This motivates break down the NER task by labels. Given an input sentence, the whole process of recognizing entities a multi-turn dialogue paradigm. Each time, Chat- GPT is asked to recognize entities of a single label. After ChatGPT provides its response to the current question, we proceed to ask questions related to the next label, incorporating all the previous questions and answers part of the dialogue context. Once all questions pertaining each label have been addressed, we conclude the entire conversation.
2022) setting, elicit LLM and modify the answer by step-by-step. On the other hand, the problem decomposition, like least-to-most prompting (Zhou et al., 2022), re- duces complex problems to the sub-problems, and then solves these sub-problems sequentially; the SC strategy (Wang et al., 2022b) generates di- verse set of answers by sampling from LLM, and then marginalizes out the sampled answers de- termine the optimal answer. In this work, we focus on investigating the zero-shot reasoning ability of LLM on the NER task.
2.2 LLM IE
A few works study the performance of the pow- erful LLM ChatGPT (Li et al., 2023; Ma et al.,
We name this paradigm Decomposed QA. The

---

Input text Could I ony Blai be in line for a gold medal?
Gold label: (Toony Blair; "Personr)
Label set: [PPesoon. 'Organizationn "Lccatinn, Faciliyy; 'W eapon; VV eeticcae Geo-Politiicll Eutiyy]
(a) Vanilla
(b) Basic Decomposed-QAA
Answer (Tony Blair: Person')
Given entity label set: [Label
Given entily lobel sets (labed set)
given enity label sel, please mooggnie the named ontities the given text.
Answer (Tony? Person))
Based on the given entity label set, please
Answer & (Tony Blair" 'Prrson
recagnize the named mntilie in the given text.
Text (input lexi)
Text (imput text)
Question: het are the named entitiees
A nswer: (ChatGPT Teeiisprerespppeee response)
labeled as 'Person' in the text?
(l) T woo-stage majority voting:
Assw en: (ChatGPT response)
Stage one: Tony Blair
(questions of each label) ...
...
Stage two: ( (Tony Blair: Personl)
(c) Syntactic Prompting Given entity Jabel (label set) Based on the given entity label set, please recognize the named entities in the given text.
(d) T Tool ugmentation Given entity lobel set: (label set) Given the text and the correpooning Part-of-Speech lags, please recognize the named entities the given text.
(e) Tool A ugmentation + Syntactic Prompting
ently
(label sel
Giee the text and the corresponding I Part-oč-Spees Speech tags, pleese recognize the named entities in the given text Text (input text)
Text: (input text)
Text (irput text)
Port-off Speoch tags: Cosld/MD Tony(NN Blain/NN beVB in/IN line/NN foc/IN a/DT gold/NN medal/NN 2/PU
Part-of-Speech tags: CouldMD Tozy/NN
Question:
W are the ramed entitiess
Blair/NN be/VB inft line/NN for|TN a/D golá/NN medal/NN 7/PC
labeled as 'Peroon in the text?
Question: W hat are the ramed entities labeked as Person in the text?
A nswen: First, let's perform Parf-of-
Speech tagging. T'ban, we reco gnize named entities based Part-of Part-of-Speech Speech tags.
Question: hot are nam ed entities
rewer Let's infler named enti Lies sleg by slep test. based on the given Part-off of--o--Speech tags. (ChatGPT response)
labeled as 'Person in text?
[ChotGPT res ponse)
nswen. (ChatGPT T response)
(questions each labol)
(questions of each label)
(questions each label)
Figure Examples of proposed methods for zero-shot NER with ChatGPT. (a) Vanilla zero-shot method. (b) Basic decomposed-Q2 where the NER task is broken down into simpler subproblems. (c) Decomposed-QA with syntactic prompting. Texts in green are the proposed syntactic reasoning hint . (d) Decomposed-QA with tool augmentation. Texts in orange
(e) Decomposed-QA with tool augmentation and syntactic prompting. (f) SC with two-stage majority voting, where stage one votes for the mentions and stage two votes for types. use part-of-speech tags as an example syntactic information this figure. The detailed prompts are shown in Appendix H.
the content of syntactic information.
example is shown in (b) of Fig. 1.
Back
Front
We obtain the label order used in the multi-turn dialogue by asking ChatGPT. For each dataset, provide the task requirement and the label set to ChatGPT, then ask it to give a reasonable label or- der based on its understanding of the labels. For domain-specific datasets, PowerPlantFlat and Pow- erPlantNested, which will be introduced Section 4.1, we also use a manual label order provided by the domain experts. The label orders are shown in Appendix G.
[Tesk assicutios]
(Task isstruction) Aystactic reasoning hind, (ispat tess)
(input
Question: (question)
Questoo: (question)
Answer, /omnectic reasoning Blld.
Anower: (ChaitPT response)
(CharGPP responad)
Figure Two positions syntactic reasoning hint.
five kinds of syntactic information are utilized: word segmentation, noun phrases, Pfrt-of-Speech (POS) tags, constituency trees, and dependency trees. Word segmentation only for Chinese. We propose the following two ways of syntactic aug- mentation.
3.2 Syntactic Augmentation
Aiming guide the model think step by step while extracting information, we encourage Chat- GPT to first grasp the syntactic structure of the input text and then leverage this syntactic struc- ture to extract relevant information. Among them,
Syntactic Prompting. We encourage the model to analyze the syntactic structure itself by inserting the syntactic reasoning hint in the input instruction, as shown (c) of Fig. 1. We explore two positions

---

of syntactic reasoning hint, i.e., in the back or front of the instruction, as shown in Fig. 2.
technical reports, which are formed during nu- clear power plant operation and maintenance. Pow- erPlantFlat only contains flat cases, while Pow- erPlantNested contains nested entities. The two datasets are formed in the vertical industrial do- data
Tool Augmentation. We first obtain the syn- tactic information of the input text via parsing tool;2 Then, we feed the input text together with the syntactic information to ChatGPT, as shown in (d) of Fig. 1. We do not apply noun phrases in tool augmentation since we do not obtain parsing tool with reliable ability to extract noun phrases.
main, and thus serve as out-of-distribun for ChatGPT. The statisties of the two datasets are shown in Appendix A. For general-domain datasets, evaluate commonly used benchmarks, including two English datasets, ACE05, and ACE04, and three Chinese datasets, OntoNotes 4 OOnto 4), MSRA (Zhang et al., 2006) and Weibo NER (Peng and Dredze, 2015). For evaluation on more datasets, please refer to Appendix E.
We further explore the combination of tool aug- mentation and syntactic prompting. To enhance the utilization of syntactic information from the parsing tool, we insert syntactic reasoning hint. The example is shown in (e) of Fig.
Model. We mainly evaluate on (gpt-333 turbo) with official API. 6 For Decomposed-QA, we maintain dialogue paradigm for each test sample. For vanilla setting, we generate the response sepa- rately for each test sample.
3.3 Seff-Consistency with Two-Stage Majority Voting
Harnessing the power SC (Wang et al., 2022b), we sample multiple responses from the model and select the most acknowledged answers as the final prediction. We design two-stage majority voting for NER, as shown in (f) of Fig. 1. At stage one, for each candidate mention appeared in all responses, we consider it an entity if it appeared in more than half of the responses; otherwise, we discard this mention. At stage two, for each mention kept in stage one, we choose the entity label predicted by the majority of responses as the final predicted label.
We also evaluate on GPT-3 (text-davinei-003) (Ouyang et al., 2022) and Llama2 (Touvron et al., 2023) to verify the effectiveness of the proposed methods on other LLMs. We use the 13B chat model of Llama2. The results of these two LLMs are in Section 4.5.
Self-consistency. We set the temperature to 0.7 and 0 for settings with and without SC, respectively. For cost saving, we conduct majority voting of 5 re- sponses in our main experiments. We first conduct both question-level and sample-level consistency on each dataset; then, we choose the way of higher performance for the rest of the experiments on the corresponding dataset.
We explore two levels of SC for decomposed- QA: question-level and sample-level. For question- level, we sample multiple responses for the current question and conduct majority voting; then, we fill the voted answer into the dialogue context for all subsequent questions. For sample-leve,, we run the whole dialogue multiple times independently and obtain the answer of each run, then conduct majority voting on these answers.
Data sampling. For syntactic augmentation, we evaluate the entire test sets of seven datasets. For SC and combinations of techniques, for cost ing, we evaluate on partial datasets and randomly sampled subsets of test sets: We evaluate on the two Nomain--peciffe datasets, PowerPlantFlat and PowerPlantNested, with entire test sets, and two general-domain datasets, Ontonotes and ACE05, by randomly sampling 300 samples from the test set three times and reporting the average results.
4 Experiment
4.1 Setup
evaluatee ChatGPT performance on both domain-specific and gemeral-domain datasets. For domain-specific datasets, we present two Chi- nese NER datasets of the electric power domain, PowerPlantFlat (PPF) and PowerPlantNested (PPN). The two datasets are collected from the
Datasets.
$ "catalog. Idc. upenn. wwwwn.eeeeeenn.... edu/LDc20066e6 *catalog. 1dc. upenn. wwtn.edddLLLLe5100000
'catalog, 1dc. upenn. wdw./Dc22e1/1ee
'The results of ChatGPT are obtained during May and June 2023 with official API.
WW use Hanlp (He and Choi, 2021) to generate syntactic information, since we found it performs well on hoth Chinese and English in our preliminary experiments.
"https: ///hgginggaacceeee /ttp///innaaacccco/eeetttaaa co/neta- so/metta-laama/
Llama 2- 1--3--baae chat-hf

---

Method
PPF
PPN
Weibo
MSRA
Onto.
ACE05
ACE04
Vanilla
27.85
20.43
30.09
45.51
33.74
28.12
20.09
Decomposed-QA
36.57
30.14
34.04
48.60
37,45
34.37
22.19
Word segmentation
36.16
30.38
32.72
47.52
37.47
Noun phrases
37.46
30.02
33,93
46.05
38.31
33.22
20.99
Front
POS tag
36.89
30.60
32.68
46.87
36.82
34.31
21.74
Constituency uree
36.21
29.88
31.85
46.02
36.52
33,22
20.86
Dependency tree
36.33
29.82
33,49
45.61
35.90
34,21
21.04
Syn.
Word segmentation
34.89
25.87
32.43
48.74
37.48
Noun phrases
32.59
24.32
28.71
46,84
38.27
29.36
21.74
Back
POS tag
36.18
26.11
33.51
44.40
36.82
28.84
23.88
Constituency tree
35.71
23.93
30.46
45.84
39.00
21.37
18.81
Dependency tree
31.05
21.02
27.61
44.87
38.52
25.57
21.04
Word segmentation
39.77
33.81
36,30
53.67
39.20
POS tag
38.11
30.97
35.14
51.99
37.61
34.33
22.41
Tool.
Constituency tree
36.51
30.25
32.00
48.32
38.40
32.96
22.15
Dependency tree
39.50
32.12
36.16
48.82
38.05
33.38
22.37
SOTA (fully-suppl supervised)
68.54
70.41
72.77
96.72
84.47
90.90
90.30
Table Overall performance. We report the values. Vanilla vanilla zero shot method without any techniques; Syn. for syntactic prompting; Tool. for tool augmentation. We use the same abbreviations in the rest of this paper when necessary. Syntactic augmentation is all conducted under the decomposed-O0 setting. Numbers in bold are the best results in the corresponding categories; Numbers underlined are the best results among all methods in the zero-shot scenario. The proposed decomposed-QU and syntactic augmentation achieve significant improvements for zero-shot NER on both Chinese and English datasets on both domain-specific and general-domas scenarios.
SOTA of fully-supervised methods. For Pow- erPlantFlat and PowerPlantNested, we use Global- Pointer (Su et al., 2022) since it performs well both flat and nested cases. For other benchmarks, we refer to corresponding papers: Weibo (Wang et al., 2021), MSRA (Li et al., 2020), Ontonotes (Li et al., 2020), ACE05 (Zhong and Chen, 2021), ACE04 (Zhong and Chen, 2021).
(with an average 9.22% Fl gain) than on general- domain datasets (with average 3.82% FI gain). This is presumably because out-of-distribution data are more challenging for ChatGPT, Decomposing makes ChatGPT acquire better understanding the out-of-distribution data. (3) We also explore the effect of reasoning techniques under the vanilla set- ting, and the results are in Table 9 of Appendix C. We found that the vanilla setting fails to stimulate the potential of reasoning techniques. Contrarily, decomposed QA stimulates the potential of syntac- tic augmentation.
4.2 Overall Performance
Table summarizes the performances of decomposed-QA and syntactic augmentation. For the two domain-specific datasets, we use the manual label orders they show better performance in preliminary experiments. For cost saving, we explore SC and combinations of reasoning techniques on selected datasets and sampled test sets, which are detailed Section 4.3
4.2.2 Effect of Syntactic Augmentation
As shown in Table 1, we draw the following conclu- sions: (1) Syntactic prompting alone brings limited benefits. This is presumably because conducting syntactic analysis without any other augmentation is challenging for ChatGPT. (2) Tool augmenta- tion exhibits consistent improvements across six datasets, showing that syntactic information helps ChatGPT better understand the input text. (3) Tool augmentation achieves more improvements on Chi- nese than English datasets. This may be due to the fact that Chinese is harder than English for Chat- GPT to handle, and syntactic information provides clue on how to understand the Chinese input bet- ter. (4) Different kinds of syntactic information exhibit various performances. On Chinese datasets, word segmentation shows the best performance.
4.2.1 Effect of Decomposed-QA
From Table 1, we have the following observa- tions: (1) Compared with the vanilla method, decomposed achieves sigmificant improve- ments across all benchmarks, including both Chi- nese and English scenarios, and both domain- specific and general-domain scenarios. This demonstrates that decomposing by labels makes the NER task much more manageable for Chat- GPT. (2) Decomposed- QA exhibits more signif- icant improvements on domain-specific datasets

---

Method
PPF
PPN
Onto.
ACE05
Vanilla
27.85
20.43
35.16 (1.57)
29.45 (0.69)
SC
28.85
20.72
35.79 (1.36)
29.37 (1.35
Decomposed-Q2
36.57
30.14
38.79 (1.66)
35.57 (0.83)
question-level
33
32.15
39,57 (1.50)
31.98 (0.31)
SC
sample-level
26.98
31.92
39.15 (0.76)
34.38 (0.85)
Word segmentation
38.16
30.38
37.67 (1.22)
Noun phrases
37.46
30.02
38.83 (1.24)
34.63 (0.78)
Front
POS tag
36.89
30,60
37.94 (1,49)
34.28 (0.45)
Constituency trce
36.21
29.88
38.43 (0.84)
34.47 (0.77)
Dependency tree
36.33
29.82
36.85 (1.16)
35.77 (0.45)
Syn.
Word segmentation
34.89
25.87
39. (1.52)
Noun phrases
32.59
24.32
39.52 (0.82)
29.78 (0.64)
Back
POS tag
36.18
26.11
37.00 (2.41)
29.72 (2.06)
on_conj
35.71
23,93
40.53 (2.54)
22.23 (0.40)
Dependency tree
31.05
21.02
39.06 (2.88)
26.65 (0.78)
Word segmentation
38.64
32.32
(1.13)
Noun phrases
38.16
32.11
40.34 (1.30)
32.35 (1.18)
Front
POS tag
38.06
31.75
38.71 (1.91)
33.02 (1.11)
Constituency
37.24
31.60
38.99 (1.52)
32.00 (0.42)
Dependency
37.65
31.30
37.17 (2.21)
34.59 (0.14)
Syn. + SC
Word segmentation
38.43
30.81
40.23 (2.59)
Noun phrases
38.73
29.19
39.79 (2.24)
34,92 (0.72)
Back
POS tag
38.48
30.77
40.27 (1.37)
34.40 (1.93)
Constituency tree
38.02
31.31
39.84 (1.90)
33.95 (0.90)
Dependency tree
37.24
31,20
40.15 (1.94)
34.42 (0.37)
Word segmentation
39.77
33.81
40.78 (2.58)
POS tag
38.11
30.97
(2.82)
35.35 (0.34)
Tool.
Constituency
36.51
30.25
(3.59))
34.54 (2.26)
Dependency
39.50
32.12
(3.04)
34.34 (0.52)
Word segmentation
39.63
33.97
41.84 (2.63)
POS tag
37.92
31.72
38.96 (4.21)
33.42 (0.64)
+
Constituency tree
36.59
28.35
(3.98)
34.60 (0.21)
Dependency tree
40.86
33.59
(2.61)
30.69 (0.97)
Word segmentation
39.67
32.97
41.09 (3.19)
POS tag
38.85
31.82
39.69 (3.98)
36.78 (1.36)
Front
Constituency tree
36.02
30.65
39.44 (2.92)
33.51 (3.04)
Dependency tree
37.16
32.06
38.83 (3.29)
34.09 (0.78)
Tool. Syn.
Word segmentation
36.24
31.46
39.68 (1.15)
POS tag
34.71
26.51
36.62 (1.05)
35.70 (1.17)
Back
Constituency tree
33.76
29.53
39.67 (1.55)
29.64 (2.95)
Dependency tree
33.18
27.73
36.85 (0.43)
29.19 (2.17)
Word segmentation
40.31
34.85
42.46 (2.20)
POS tag
38.21
30.89
40.86 (2.48)
33.19 (1.39)
Front
Constituency tree
35.76
29.00
41.36 (3.58)
33,42 (2.35)
Dependency tree
39.97
33.23
40.49 (3.49)
30.29 (0.71)
Tool. Syn,
Word segmentation
40.83
30.78
41.40 (2.81)
POS tag
38.00
30.64
38.58 (2.77)
30.28 (2.21)
Back
Constituency tree
36.26
26.36
40.53 (3.38)
29.78 (1.64)
Dependency tree
41.97
32.73
40.19 (2.13)
29.87 (0.17)
SOTA (fulll-supervised)
68.54
70.41
84.47
90,90
Table 2: Performance of SC and combinations of reasoning techniques. We report the F1 values. Numbers in parentheses are the standard deviations. Numbers bold are the best results in the corresponding categories; Numbers underlined the best results among all methods in the zero-shot scenario. SC with two-stage majority voting and combinations of reasoning techniques brings further improvements.
On English datasets, POS tags boost the most. This is presumably because simpler syntactic informa- tion easier for ChatGPT to understand. Com- plex syntactic information, such as dependency tree, though informative, can hard to understand, thereby, exhibiting unstable performance.
4.3 Effect of Self-Consistency and Combinations of Reasoning Techniques
Table summarizes the performance SC and the combinations of reasoning techniques. We visual- ize the results on PowerPlantFlat and Ontonotes 4

---

Without With
Without SC
With SC
FL-care
30
25-
20
Syn.
Syn.
Vanilla
Syn.
Syn.
(a) Ontonotes
(b) PowerPlantFlat
Figure 3: Performance of combinations of reasoning techniques. For methods involving syntactic augmentation, we plot the average results over all kinds of syntactic information. The vertical lines on top part some bars represent the performances range over all kinds of syntactic information. With SC of two-stage majority voting, the combinations of reasoning techniques further improve the performances.
Tool.
QA
Tool.
Error Types
Vanilla
QA
TS-SC
Syn.
Tool. Syn.
43.5
Syn.
Tool. + Sym.
Type
OOD rypes
42.4
Wrong types
141
150
156
40.4
41.4
Boundary
Cotain gold.
54
35
39.9
40.3
Cotained by gold.
27
24
T 39.2
Overlap with gold.
38.1
Completely-O
220
176
38.3
37.1
Omitted mentions
23
41
43
37.8
20 # sampled responses
135 10
23 sampled responses
20
OOD mentions
36
10
Total
585
530
444
(a) Ontonotes 4
(b) PowerPlantFlat
Table 3: Numbers of error types Ontonotes 4. "QA" for decomposed-QA. "TS-SC" for combinations of tool augmentation, syntactic prompting. and SC. Numbers in bold denote the best results, i.e., the least emrors. The proposed methods significantly reduce the total amount of error.
Figure 4: Increasing sampled responses generally im- proves performance under SC with two-stage majority voting.
in Fig. for better analysis.
From the table and the figure, we have the fol- lowing observations and conclusions: (1) SC shows consistent improvements almost all methods. As long the syntactic information is involved, SC can always boost performance. This may be due to the fact that syntactic information is helpful but hard to understand or analyze. Thus, syntactic in- formation gives ChatGPT the potential to perform better but also higher possibility of making mis- takes. SC can filter out errors, thereby, leveraging the advantages, and eliminating the disadvantages of syntactic information. (2) Syntactic prompting fails to boost tool augmentation and even hurts the performance. However, when equipped with SC, syntactic prompting improves tool augmentation. This may be due to the complexity of information provided by the combination of tool augmentation and syntactic prompting. The complex information leads the model to think and explore more, and of course, it is also accompanied by more possibilities for errors. This makes SC an effective means of fil- tering out errors here. (3) SC improves more when syntactic reasoning hints are put on the back than on the front. This is presumably because the closer
the reasoning hint is to the answer, the more it can stimulate the model's thinking. Hence, putting the reasoning hints the back encourages the model to generate more diverse answers, which provides better search spaces for majority voting.
We explore the effect of increasing sampled re- sponses SC, which are shown in Fig. 4. We sample up to 30 responses for cost saving. As seen in the figure, sampling higher number of re- sponses improves the performance. We conjecture that combining diverse syntactic information may further benefit SC on NER.
4.4 Error Analysis
4.4.1 Error Types
We take Ontonotes 4 for error analysis. Table 3 summarizes the statisties of error types. Fig. 6 visualize the percentages of error types. Below is the introduction of error types:
Type. 0OD types: predicted entity types not the given label set; Wrong types: predicted types

---

Input Text: The player who temporurily ranks second is German athlete Bao
Input Text:
Despire her natural and effortless movements, her dillicully cannot be compared to that of Luo Li, who only
Lizzo, with score 355.02 points, slightly lower than Lanwei.
Gold Label: ("Geraan": "eorman"""Gooo-"o Goo-Poliiical Eutity", "Lanwei": "Person".
scored 9.875 points and wen silver medal.
"Bao Lizzo": "Person")
Gold Lahel: ("Luo Li:;; "Person")
Vanilla Ans: ("Gemman athlete Bao Lizzo": "Person", "Lanwei": "Person")
Vanilla Ams: ("Luo Li": "Persoa", "Silver medal": "Location")
TS-SC Ans: ("Bao- Lizzo": "Person": "Person", "Lanwei": "Person",
TS-SC Ans: ("Lao Li": "Person")
"Germaan:: "Geo- Political Entity))
Corrected Error Types: Completely-O)
Corrected Error Types: Contain gold mention
Possible Reason: Better understanding the input text due syntactic information.
Possible Reason: Benefits from synlactic information, ie.. word segmentation here. Involved word segmentation: ("German". "atilete", "Bao Lizeo" "]
Input Text: Suddenly, there was a heavy crashing sound, followed by exclamation of surprise. La Xiugin, the leader of the research collection team. rished into the seene inspect.
Input Text: Xinhua News Agency, Tianjin, December 15th (Reporter L. Jianchung, intern Ge Suhong).
Gold Label: ("Li Jianchang": "Person", "Tiajjn" "Geo- Political Entily". "Xinhua News Agency:: "Organization',
Gold Label: ("reseurch collection team": "Organization", "Lu Xioqin": "Person")
Vanilla Ans: ("La Xiugia": : "Persoa") ") (Omiited ("reseurch collection team":
"Ge Suhong": "Person")
"Organization") ))
Vanilla Ans: ("Xinhua News Agency": "Organization",
TS-SC Ans: ("Lu Xiugin": "Person", "colleecion team". "Organization";
"Taanjnn:: "Location", "Li Jianchang"" "Peison". "Ge
Suhong": "Person")
TS-SC Aus: ("Li Jianchang": :
"Person", "Ge Suhong":
Increased Error Types: Contained by gold mention
"Person". "Xinhua News Agency": "Organization")
Possible Reason: Misguided by syntactsc information. Involved word ("rescardl'' "collection team") J. However, without the word segmentation, the model could completely overlook this entity.
(Omitted ("Tanjin": "Geo coliticll Entity"))
segmeniation:
Increased Error Types: Omitted mention
Optimization Direction: Provide more accurate and comprehensive syntactic information. E.g. Using more powerful parsing lool and combinations different syntnctic information.
Possible Reason: Inudequate understanding of entity types. Optimization Direction: Provide entity type information.
Case study of error correction and error increase with the proposed methods. We translate the original Chinese text into English in the demonstrations for readability. The upper two cases errors corrected, and the lower two are errors increased, Texts in blue are involved entities in the error cases. Our method shows effectiveness on error corrections. With the suggested optimization strategies, the error increased might be eliminated.
Figure 5:
the incomplete annotation or that ChatGPT would guess entities based on its prior common knowl- edge. The latter may be due to the inadequate understanding of entity types. As seen in Table 3, decomposed-QA reduces the total error numbers by 9.4%; The combination of Tool augmentation, Syntactic prompting and SC (TS-SC) reduces the error numbers by 24.1% 1,, showing remarkable ca- pability in error corrections.
Error Types
Wrong typas
00D types
11.0%
Cotained by gold.
Cotain gold.
0.05%
Overlap with gold.
Completely -O OOD mentions
56.90%
Omitted mention
Figure 6: Percentage of different error types Ontonotes 4 under the vanilla method.
4.4.2
Case Study of Error Correction and Error Increase
incorrect but in the given label set.
As seen in Table 3, TS-SC reduces errors mainly in types of contain gold. and completely-O, and increases errors mainly in types of contained by gold. and omitted mentions. Thus, we conduct case study on these four types, which are shown in Fig. 5. TS-SC corrects errors of contain gold. and completely-O presumably by providing syntactic information and making the model better under- stand the input text. Meanwhile, TS-SC increases errors of contain gold. and omitted mentions pre- sumably because of the misguiding of syntactic information and inadequate understanding of entity types, respectively. For the former, providing more accurate and comprehensive syntactic information might be a solution; for the latter, providing type information might be a direction of optimization.
Boundary. Contain gold.: predicted mentions containing gold mentions; Contained by gold.: pre- dicted mentions contained by gold mentions; Over- lap with gold.: predicted mentions not in the two above situations but overlap with gold mentions.
Completely-O: predicted mentions that do not have any of the three above boundary situations with any gold mentions.
0OD mentions: predicted mentions that do not appear in the input text.
As shown in Fig. 6, the majority error types are complete-O and wrong types, which account for over 80% of all errors. The former may be due to

---

Dataset
Method
O-shot
3-shot
5-shot
10-shot
Vanilla
35.16 (1.57)
38.67 (3.57)
44.51 (5.78)
52.45 (4.13)
Standard CoT
34.34 (6.61)
41.13 (6.31)
41.90 (2.43)
Ontonotes 4
Tool. w. word segmentation (Ours)
40.78 (2.58)
42.48 (3.34)
47.16 (5.42)
54.40 (2.68)
Syn. w. word segmentation (Ours)
37.94 (1.49)
43,89 (3.67)
50.70 (7.26)
56.71 (3.70)
Vanilla
27.85
35.81 (2.94)
37.44 (3.88)
41.13 (4.89)
Standard CoT
30,63 (6.45)
33.95 (3.59)
38.02 (1.03)
PowerPlandFlat
Tool. w. word segmentation (Ours)
32.41
39.43 (1.91)
41.12 (4.35)
42.05 (4,74)
Syn. word segrenttttio (Ours)
28.09
37.84 (2.59)
39.72 (2.79)
42.52 (3.71)
Table 4: Results under few-shot setting, where the number of shots is the number of texts. We randomly sample three sets of demonstrations and take the averages. Results for Ontonotes are averaged over three sets of randomly sampled 300 samples from the test set. We report F1 values. Numbers in parentheses are standard deviations. Numbers in bold are the best results. Our methods also achieve significant improvements in few-shot scenarios.
Dataset
ACE0S
BC5CDR
Model
GPT-3.5
GPT-3
Llama2
GPT-3.5
GPT-3
Llama2
Vanilla
29.45
14.03
61.28
29.49
26.12
Decomposed-QA
35.57
23.88
15.53
65.45
38.73
28.30
Syn. w. dependency tree
26.65
27.93
16.98
59.69
41.62
34.46
Tool. w. dependency tree
34.34
27.59
17.31
62.79
43.69
39.94
Tool. + Syn. w. dependency tree
29.19
18.38
26.99
57.28
16.38
39.57
Table 5: Performance on GPT-3 (text-davinci-003) and Llama2 13B chat model. Results are averaged over three sets of randomly sampled 300 samples from the test set. We report the values. Our proposed strategies show consistent improvements various LLMs.
4.5 More analysis
We also evaluate our methods on GPT-3 frxt-davvnci-O03: (Ouyang et al., 2022) and Llama2 (Touvron et al., 2023). Since Llama2 still has poor support for Chinese yet, evaluate two English datasets, one general-domain dataset, ACE05, and one biomedical dataset, BC5CDR (Li et al., 2016). The results are shown in Table 5, which the dependency tree used for demonstra- tion. The complete results are in Appendix F. The main results of BC5CDR are in Appendix E. Table 5 shows that our methods exhibit consistent im- provements across different LLMs, including the close-sourced ChatGPT model series and typical open-soureed model Llama.
Other LLMs.
Few-shot setting. We evaluate the proposed syn- tactic augmentation under few-shot setting. Han et al. (2023) investigate standard CoT on NER by generating intermediate rationales with ChatGPT. We take different perspective: we encourage the model explore syntactic information as their in- termediate thinking steps. Detailed adaptations of our methods to the few-shot setting are explained in Appendix D. For the decomposed-O0 -A and SC, we leave them to future work due to the cost budget.
We compared our methods the vanilla method and standard COT. We use CharGPT generate rationales standard COT, following (Han et al., 2023). Here, we use one general domain dataset, Ontonotes 4, and one domain-specific dataset, Pow- erPlantFlat, for demonstrations. The results are shown in Table 4, in which the word segmentation is used for demonstration. The results of various syntactic information are in Appendix D.
Conclusion
We present an empirical study of zero-shot NER with ChatGPT, with four proposed strategies to sim- ulate the reasoning potential of ChatGPT on NER. Inspired by the powerful reasoning capabilities of LLM on logical and arithmetic reasoning tasks, the proposed strategies involve task decomposition, syntactic augmentation, and tailored SC. We verify the effectiveness of our methods Chinese and English scenarios, and on both domain-specific and general-domain datasets. We provide an analysis of the error types with suggested solutions. Be- sides, we verify the effectiveness of the proposed methods on the fhw-shot setting and other LLMs.
As observed in Table 4, the standard CoT does not bring improvements, even hurt the performance. This is presumably because standard CoT is very sensitive to the rationales constructed, which is also mentioned in (Han al., 2023). However, our strategies have achieved significant improvements. This shows that the proposed methods are effective not only in the zero-shot scenario but also in the few-shot setting.

---

Ridong Han, Tao Peng. Chaohao Yang, Benyou Wang, 6 Limitations
David Dohan, Shivami Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
For cost saving, we focus the investigation of each individual syntactic information and have not explored the combinations of different kinds of syn- tactic information. Also, we have not investigated manual label orders on general-domain datasets for the same reason. We leave them to future work.
Gamal Crichton, Sampo Pyysalo, Billy Chiu, and Anna Korhonen. 2017. neural network multi-task learn- ing approach to biomedical named entity recognition. BMC bioinformatics, 18(1):1-14.
Acknowledgments
We would like to thank the anonymous review- ers for their insightful comments and constructive suggestions. This research is supported by the Na- tional Key Research and Development Program of China (Grant No. 1707803) and Zhejiang Provincial Natural Science Foundation of China (LDT23F02023F02).
Leon Derezynski, Eric Nichols, Marieke Van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Lu Liu, and Xiang Wan. 2023. information extrac- tion solved by chatgpt? an analysis of performance, evaluation criteria, robustness and emors. arXiv preprint arXiv:2305. 14450.
References
Michael Bada, Miriam Eckert, Donald Evans, Kristin Garcia, Krista Shipley, Dmitry Sitnikov, William A Baumgartner, K Bretonnel Cohen, Karin Verspoor, Judith A Blake, et al. 2012. Concept annotation in the craft corpus. BMC bioinformatics, 13(1):1-20.
Han He and Jinho D. Choi. 2021. The stem cell hy- pothesis: Dilemma behind qulli-task learning with transformer encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5555-5577 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks; Distributional similarity for relation learning. Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy. Association for Computational Linguistics.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan. Shafiq Joty, and Jimmy Xiangji Huang. 2023. A systematic study and comprehensive evaluation of chatgpt on bench- mark datasets. arXiv preprint arXiv:2305. 18486.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi- pooling convolutional neural networks. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 167-176, Beijing, China. Association for Computational Lin- guistics.
Bo Li, Gexiang Fang. Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt's information extraction capabilities: An as- sessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.1 1633.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi,
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sci- aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative cdr task corpus: resource for chemical disease relation extraction. Database, 2016
Xiaoya Li, Xiaofei Sun, Yuxian Meng. Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice loss for data- imbalanced nip tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 466-476, Online. Association for Computational Linguistics.

---

Yuanyuan Liang, Jianing Wang, Hamlun Zhu, Lei Wang. Weining Qian, and Yunshi Lan. 2023. Prompting large language models with chain-of-though for few- shot knowledge base question generation.
Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran. Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. 2022. Lamda: Language models for dialog appli- cations.
Mingyu Derek Ma. Xiaoxuan Wang, Po-Nien Kung, P PJeffrry Brantingham, Nanyun Peng. and Wei Wang. 2023. Star: Boosting low-resource event extraction by structure-to-text data generation with large lan- guage models. arXiv preprint arXiv:2305.15000
OpenAI. 2022. Introducing chatgpt.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2XXi:23377.0288
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc.
Zhen Wan, Fei Cheng. Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation ex- traction using large language models. arXiv preprint orXiv:2305.02105
Boshi Wang, Sewon Min. Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for Chinese social media with jointly trained embeddings. In Proceedings of the 2015 Con- ference Empirical Methods in Natural Language Processing, pages 548-554, Lisbon, Portugal, Asso- ciation for Computational Linguistics.
Shuhe Wang. Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304. 10428.
Sampo Pyysalo, Tomoko Ohta, Rafal Rak, Dan Sul- livan, Chunhong Mao, Chunxia Wang, Bruno So- bral, Jun'ichi Tsujii, and Sophia Ananiadou. 2012. Overview of the id, epi and rel tasks of bionlp shared task 2011. In BMC bioinformatics, volume 13, pages 1 1-26. Springer.
Xinyu Wang. Yong Jiang, Nguyen Bach, Tao Wang. Zhonggiang Huang, Fei Huang, and Kewei Tu. 2021. Improving named entity recognition by external con- text retrieving and cooperative learning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1sh International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 1800-1812, Online. Association for Computational Linguisties.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver?
Brik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independdrr named entity recognition. arXiv preprint cs/0306050.
Xuezhi Wang, Jason Wei, Dale Schuurmans. Quoe Le, Fdd Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. arXiv preprimt arXiv: /1171.
Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun, Wanwei Huang, Bo Wen, and Yunfeng Liu.
2022. Global pointer: Novel efficient saan-based approach for named entity recognition.
Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903..
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng. Amin Ghafouri, Marcelo Menegali. Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint 2302 10205.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020.
Named entity recognition as dependency parsing. In

---

Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
A Statisties of PowerPlant Datasets
Table 6 shows the overall statistics of the Power- Plant datasets, and Table 8 displays the classwise statistics.
Suxiang Zhang, Ying Qin, Juan Wen, and Xiaojie Wang. 2006. Word segmentation and named entity recog- nition for SIGHAN bakeoff3. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Pro- cessing, pages 158-161, Sydney, Australia. Associa- tion for Computational Linguistics.
Dataset
Split
#Sentences
#Entities
Train
3087
4379
Flat
Test
401
540
Train
3047
6924
Zhuosheng Zhang. Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
Nested
Test
492
1109
Table 6: Statistics of PowerPlant datasets.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 50-61, Online. Association for Computational Linguistics.
B Statistics of Errors on PowerPlantFlat
Table 7 summarizes the statisties of error types on PowerPlantFlat under vanilla, decomposed-Qan and TS-SC methods. Fig. shows the percent- ages of error types on PowerPlantFlar under vanilla method
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet. Quoe Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing large language models. arXiv preprint arXiv:2205.106::
Error Types Type
Vanilla
QA
TS-SC
OOD types
0
0
0
Wrong types
90
66
65
Boundary
Cotain gold.
153
167
110
Cotained by gold.
31
51
76
Overlap with gold.
8
Completely-O
406
279
Omitted mentions
65
37
56
OOD mentions
32
14
Total
800
767
603
Table Numbers of different error types on Power- PlantFlat. "QA" refers to decomposed-OA. -AQ, "TS-SC" refers to the combination of tool augmentation, syntac- tic prompting, and SC. Numbers in bold denote the best results on PowerP PlantFlat, i.c., the least errors.
B.00%
0.00%
90%
Ertrr Types
Wrong types
000
gold.
Overisp with gold.
Complenely -b 000 mentions
mention
Figure 7: Percentage of different enor types Power- PlantFlat under the vanilla setting.

---

C Performance Under Vanilla Setting
Meanwhile, conjecture that this is also due the fact that CoNLL-2003 and WNUT-17 contain more numbers of shorter texts, on which the rea- soning techniques are difficult to leverage their ad- vantages. However, the proposed methods achieve significant improvements in biomedical domain datasets BC5CDR, BioNLP11, and CRAFT. This demonstrates that the proposed methods can also improve zero-shot NER of other challenging do- mains besides the electric power domain. Plus the five datasets evaluated in Table 1, we evaluate on twelve benchmarks in total and achieve remarkable improvements on ten datasets among them.
We also investigate the effect of our proposed rea- soning techniques the standard setting. The re- sults are shown in Table 9. From the table, we can conclude that the potential the syntactic informa- tion cannot be fully exploited under the standard setting. On the contrary, the proposed decomposed- QA paradigm effectively utilizes the syntactic in- formation, as shown in Tabel 2 and Figure 3.
Under standard setting, the reasoning techniques bring limited benefits for general-domain datasets, sometimes even hurting the performance. However, these techniques exhibit improvements on domain- specific datasets, i.e., out-of-distribution datasets. This presumably because out-of-dixtribution data is much more challenging than general-domain data for ChatGPT. The reasoning techniques lead the model to have a better understanding of the out-of-distribution data.
F Evaluation on Other LLMs
The complete results on GPT-3 and Llama2 are shown in Table 12. These results show that our methods exhibit consistent improvements across different LLMs.
the smallest LLM evalu- ated, Llama2 13B, our proposed strategies still achieve remarkable performance improvements, with 19.72% and 17.51% F1 improvements on ACE05 and BC5CDR, respectively. This reveals that methods have wide applicability various sizes of LLMs, which is beneficial for low-resource scenarios such as when only smaller LLMs are af- fordable.
D Syntactic Augmentation Under Few-shot Setting
The following are the adaptations of our proposed syntactic augmentation strategies to the few-shot setting. (1) Syntactic prompting: For the test ple, we ask the model first perform syntactic analysis and then recognize entities. For demonstra- tions, we use parsing tools to generate intermediate syntactic parsing results. (2) Tool augmentation: We provide both the text and syntactic information for the demonstrations and the test sample.
G Label Order
Table 13 displays label orders used in our main periments and the corresponding results under ba- sic decomposed-QAa On the power plant datasets, manual label orders provided by domain experts achieve significantly better results. This demon- strates that when dealing with domain-specific datasets with ChatGPT, one may turn to domain knowledge to boost performance.
Table 10 shows the experiment results under the 3-shot setting.
E Evaluation on More Datasets
We additionally evaluate the proposed meth- ods more datasets, including general-domain datasets, CoNLL-2003 (Sang and De Meulder, al., 2017), and domain-specific datasets (i.e., biomedical domain), BC5CDR (Li et al., 2016), BioNIPP1 (Pyysalo et al., 2012), and CRAFT (Bada et al., 2012; Crich- ton et al., 2017).
Table 14 displays the label orders of additional datasets.
2003), WNUT-17(Derezynss
Table 15 shows the instructions for asking Chat- GPT to provide label orders of PowerPlantFlat and ACE05 datasets.
H Prompts
The results are shown in Table 11. We found that the proposed reasoning techniques cannot guaran- tee performance improvements on CoNLL-2003 and WNUT-17 and even hurt the performance. This is presumably because the label logic of these two datasets not suitable for decompusition, and the syntactic information generated for them is noisy.
We show all of our prompts with Ontonotes 4 and ACE05 examples. The prompts are in Table 16,
17, 18 and 19.

---

Flat
Nested
Label
Chinese Label
Train
Test
Train
Test
System name
K कM K
132
10
143
21
System identity
357
49
1654
270
Device name
2 Z M
1239
159
1191
199
Device identity
1517
185
1462
243
Component name
#
763
97
771
108
Location name
200
24
197
30
Person
A
171
184
24
Reactor Status
X PE
88
13
Power Plant Event
#
1234
201
Table 8: Classwise statistics of PowerPlant datasets.
Method
PPF
PPN
Weibo
MSRA
Ontonotes
ACE04
ACE05
Vanilla
27.85
20.43
30.09
45.51
33.74
20.09
28.12
Cil--onssitenyy
28.85
20.72
31.02
19.97
28.21
Syntactic Prompting
Word segmentation
28.09
20.37
28.
41.72
30.82
Noun phrases
28.94
21.81
28.89
41.5
30.89
18.77
26.21
Front
POS tags
30.12
22.47
27.23
41.20
30.59
19.54
28.27
Constituency Tree
26.38
20.47
28.23
40.62
30.61
19.75
28.49
Dependency Tree
27.21
20.7
28.51
40.44
30.77
19.68
28.46
Word segmentation
27.37
20.58
20.10
42,92
32.03
Noun phrases
31.65
21.36
17.09
42.60
31.62
19.69
26.05
Back
POS tags Constituency Tree
28.24
17.70
19.38
42.34
31.86
20.60
25.65
30.31
20.53
17.74
42.52
31.88
20.42
23.98
Dependency Tree
26.08
17.47
14.09
42.68
31.64
20.33
26.31
Tool augmentation
Word segmentation
32.28
26.57
25.37
29.75
POS tags
28.13
24.17
24.86
37.29
29.95
19.14
28.14
Front
Constituency Tree
23.62
20.97
22.98
26.1
16.97
27.65
Dependency Tree
26.08
17.47
14.09
31.64
20.33
26.31
Word segmentation
28.57
26.81
21.88
34.19
26.77
POS tags
22.04
202.5
24.69
34.77
27.84
18.16
25.17
Back
Constituency Tree
22.46
21.82
20.79
30.81
24.62
16.09
23.60
Dependency Tree
21.36
20.25
25.18
32.73
26.88
16.13
21.67
SOTA (fully-superviiee)
68.54
70.41
72.77
96.72
84.47
90.3
90.9
Table 9: Performance reasoning techniques under the vanilla setting (without decomposition). In this table. "vanilla" specifically refers to the zero-shot method without any techniques. We report the Fl values on entire test sets. We spare the SC on MSRA and Ontonotes 4 for cost saving.

---

Method
Ontonotes
PowerPlantFlat
Vanilla
38.71 (3.34)
35.81 (2.94)
Decomposed- QA
43.30 (1.84)
43.75 (3.06)
Word segmentation
40.12 (3.22)
37.33 (2.70)
POS tag
44.11 (3.52)
36.42 (0.26)
Syn.
Constituency tree
38.81 (2.38)
33.66 (2.50)
Dependency tree
35.05 (1.41)
36.16 (1.43)
Word segmentation
41.28 (3.65)
38.7 (2.27)
POS tag
44,93 (4.02)
36.19 (0,65)
Syn. + SC
Constituency tree
41.89 (3.50)
36.05 (2.65)
Dependency tree
37.98 (2.56)
39,06 (0.78)
Word segmentation
42.33
39,43 (1.91)
POS tag
42.51 (2.44)
37.04 (0.72)
Tool.
Constituency
38.51 (4.17)
35.14 (2.84)
Dependency
36.12
33.54 (1.44)
Word segmentation
40.49 (12.05)
41.09 (2.71)
Tool. SC
tag Constituency tree
43.28 (2.69)
38.66 (2.05)
40.12 (4.31)
35.54 (2.66)
Dependency tree
(2.50)
35.79 (1.87)
Word segmentation
43.11
39.69 (2.61)
POS tag
42.44
36.35 (1.81)
Tool. Syn.
Constituency tree
38.31
34.45 (2.53)
Dependency tree
35.21 (2.26)
34.1 (1.06)
Word segmentation
42.18 (2.29)
41.43 (1.78)
POS tag Constituency tree
41.53
37.95 (2.03)
Tool. Syn. SC
41.57 (4.54)
35.32 (3.82)
Dependency tree
40.33
35.85 (2.14)
Table 10: Performance of syntactic augmentation under 3-shot setting. We randomly sample three of demon- strations and report the means and standard deviations of F1 values. Numbers in parentheses are standard deviations. Numbers bold are best results in each category. The proposed syntactic augmentation exhibits significant improvements in the few-setting.
Dataset
CoNLL-2003
WNUT-17
BC5CDR
BioNLP11 51.29 (2.48)
CRAFT
Vanilla
69.42 (0.91)
46.61 (2.97)
61.28(3.11
21.66
Decomposed-O2
59.67 (0.36)
42.39 (1.99)
65.45 (0.89)
55.3 (0.54)
23.99 (2.65)
Syn. (Front)
Noun phrases
57.13 (0.41)
(2.42)
64.41 (1.74)
(1.89)
22.92 (3.04)
POS tag
(0.98)
39.74 (1.98)
66.24 (2.40)
53.97 (0.99)
23,59 (2.01)
Constituency tree
56.36 (1.07)
39.66 (1.51)
64.7 (0.80)
53.98 (1.24)
23.84 (1.94)
Dependency tree
54.27 (1.54)
38.36 (0.37)
65.56 (1.20)
54.37 (0.83)
23.51 (2.06)
Syn. (Back)
Noun phrases
58.89 (1.39)
36.47 (1.10)
60.44 (0.57)
(1.27)
21.71 (1.89)
POS tag
56.12 (1.54)
38.12 (0.55)
58. 19 (1.55)
54.41 (2.62)
22.69 (3.76)
Constitenccy tree
55.66 (2.17)
37.75 (2.04)
47.81 (1.99)
43.58 (1.44)
23.86 (1.30)
Dependency tree
58.36 (1.11)
37.49 (1.56)
59.69 (0.76)
55.23 (0.77)
22.84 (1.59)
Tool.
POS tag
62.79 (2.54)
43.81 (2.15)
(1.44)
52.38 (0.35)
24.54 (3.87)
Constituency
60.96 (0.34)
44.3 (1.02)
65.02 (3.00)
51.44 (0.81)
23.88 (2.24)
Dependency
59.23 (3.13)
41.6 (1.30)
62.79 (2.62)
42.71 (0.73)
24.86 (2.45)
Tool. Syn. (Front)
POS tag
63,46 (1.02)
43.9
64.24 (1.83)
49.87 (0.85)
25.05 (2.12)
Constituency tree
59.59 (0.99)
44.68 (2.59)
65.43 (2.51)
50.6 (1.39)
24.5 (2.76)
Dependency tree
57.93 (2.05)
40.96 (3.57)
59,93 (1.88)
40.18 (1.63)
24.46 (2,50)
Tool. Syn, (Back)
POS tag
58.08 (0.27)
39.32 (4.42)
60.84 (3.47)
47.9 (1.88)
13.71 (1.90)
Constituency tree
58.95 (2.19)
37.91 (5,46)
54,57 (1.86)
42.27 (2.78)
20.64 (1.85)
Dependency tree
54.68 (1.27)
36.7 (3.07)
57.28 (1.37)
45.62 (3.46)
24.05 (4.05)
Table 11: Performance on additional datasets. Results are averaged over three sets randomly sampled 300 samples from the test set. We report the means and standard deviations of Fl values. Numbers in parentheses standard deviations. Numbers in bold are best results in each category.

---

Dataset
ACE05
BC5CDR
Model
GPT-3.5
GPT-3
Llama2
GPT-3.5
GPT-3
Llama2
Vanilla
29.45 (0.69)
14.03 (0.94)
9.07 (1.33)
61,28 (3.11)
29.49 (3.12)
26.12 (2.94)
Decomposed-QA
35.57 (0.83)
23.88 (2.21)
15.53 (1.53)
65.45(0.88)
38.73 (2.58)
28.30 (0.73)
Syn. (Front)
Noun phrases
(0.78)
21.74 (2.16)
17.30
64.41 (1.74)
39.38 (3.01)
36, (1.24)
POS tag
34.28 (0.45)
21.98 (2.90)
23.86 (6.79)
66.24 (2.40)
45.31 (1.34)
34.65 (2.35)
Constituency tree
34.47 (0.77)
22.38 (2.09)
21.76 (0.31)
(0.80)
42.09 (1.51)
36,92 (0.37)
Dependency tree
35.77 (0.45)
22.84 (2.81)
25.91 (1.20)
65.56 (1.20)
43.19 (0.93)
33.11 (0.31)
Syn. (Back)
Noun phrases
29.78 (0.64)
24.45 (2.29)
15.73 (1.93)
60.44 (0.57)
35.17 (1.88)
33.75 (1.26)
POS tag
29.72 (2.06)
30.73 (2.90)
16.51 (1.82)
58. 19.19 (1.55)
45.17 (3.00)
34.62 (3.02)
Constituency tree
22.23 (0.40)
27.08 (2.82)
16.45 (1.74)
47.81 (1.99)
39.72 (1.96)
35.17 (0.88)
Dependency tree
26.65 (0.78)
(2.73)
16.98 (1.23)
59.69 (0.76)
41.62 (2.30)
34.46 (1.81)
Tool.
POS tag
35.35 (0.34)
24.74 (0.88)
18.00 (1.02)
66.40 (1.44)
47.04 (2.39)
40.45 (1.11)
Constituency
34.54 (2.26)
26.84 (2.46)
17.36 (0.65)
65.02 (3.00)
52.77 (2.73)
38.95 (0.85)
Dependency tree
34.34 (0.52)
27.59 (2.05)
17.31 (2.14)
62.79 (2.62)
43.69 (2.33)
39.94 (1.05)
Tool. + Syn. (Front)
POS tag
36.78 (1.36)
26.21 (1.88
17.94 (1.28)
(1.83
47.70 (2.46)
33.84 (1.63)
Constituency tree
33.51 (3.04)
29.93 (2.03)
18.11 (1.42)
65.43 (2.51)
54.12 (3.00)
30.23 (1.65)
Dependency tree
34.09 (0.78)
30.62 (1.79)
15.75 (2.38)
59.93 (1.88)
43.26 (2.11)
38.16 (4.50)
Tool. + Syn. (Back)
POS tag
35.70 (1.17)
22.08 (1.32)
24.50 (2.09)
60.84 (3.47)
17.72 (1.87)
43.63 (2.61)
Constituency tree
29.64 (2.95)
15.06 (0.23)
23.87 (1.18)
54.57 (1.86)
11.44 (1.72)
36.48 (1.91)
Dependency tree
29.19 (2.17)
18.38 (1.10)
26.99 (0.49)
57.28 (1.37)
16.38 (1.27)
39.57 (2.04)
Table 12: Complete results various LLMss. We g-t-3.5-turbo for GPT-3.5, text-davinci-003 for GPT-3, and 13B chat model for Llama2. For cost saving, sample 300 samples from the test set three times, and report the average results Fl values. Numbers parentheses are the standard deviations. Numbers in bold are the best results in the corresponding categories.

---

Dataset
Label order
Order generation
F1
vanilla
27.85
manual
36.57
PowerPlantFlat
"
ChatGPT
30.52
2 ",, 11
vanilla
20.43
I" ( "
manual
30.14
PowerPlan[Nested
"1, "1 "] ("
ChatGPT
20.16
vanilla
30.09
Weibo
[ TTAX41111844411100088888818111,1
ChatGPT
34.04
vanilla
45.51
MSRA
[[' ""
ChatGPT
48.60
vanilla
33.74
Ontonotes 4
II' 1
L
ChatGPT
37.45
vanilla
20.09
ACE04
[["Person"]J "."rOrranizzaiion"""""" "I.aatiiiii"""LLLaataaiaa "'"Coattin""), ["Facility"], ("Weapon" "eehccelll ["Geo
ChatGPT
22.19
-Political Entity"]]
vanilla
28.12
ACE05
[["Person"]. "rr."Orraazzaainn"...... L""'cattoo"""
("Facility"), ("Weapon") ".("eeeicll" ("Geo -Political Entity"]]
ChatGPT
34.37
Table 13: Label orders with corresponding performances. The results from the entire test set. "vanilla" refers the standard setting without any techniques.

---

Dataset
Label order
Order generation
F1
vanilla
69.42
CoNLL-2003
[["Location"], ["Organization"], (" ("Person"), ]|Miscellancous"]
ChatGPT
59.67
vanilla
46.61
WNUT-17
[["Person"], ["Location"], ("Corporation"), ["Product"], ["Creative work"J, ["Group"]]
ChatGPT
42.39
vanilla
61.28
BCSCDR
[["Chemical"], [ ["Disease"]]
ChatGPT
65.45
vanilla
51.29
BioNLPI1
[[' [['roteein]] J, O'OrganimmJ, ['Chemical'], ('Regulon-operonn] ]]
ChatGPT
55.30
vanilla
21.66
CRAFT
[['Simple_mt "Simplc_chemical"" ccemicall],, ['Gene_ or_gene _ei__neeeee_____e_e jann___rouut" Ceellllar_componee J. ['Complex']]
ChatGPT
23.99
Table 14: Label orders of additional datasets and corresponding performances. Results are averaged over three sets randomly sampled 300 samples from the test set.
Dataset
Prompts
PowerPlantFlat
"Given entity label set: l'Person', 'Orannization', 'Location', 'Facility' 'Weapon', 'Vehicle. 'Geo-Political Entity') \n We need recognize the corresponding named entities based different labels. What is the reasonable label order?"
ACE05
Table 15: Instructions for asking ChatGPT to provide label orders.

---

Syntactic prompting
si # syntactic reasoning hint (front) | # [n
THA |]
T 0- \ (syntactic reasoning hint
(back))
X
# 11 W A T 11 \
{syntactic reasoning hint
(back))
"4 11 31 1
11 n (syntacticreasooning reasoning
hint (back)}
1
HU 1(*
: "
(syntactic
reasoning hint (back))
Syntactic reasoning hint (front)
Word segmentation
$
Noun phrases
$
3.. 00 $
POS tagging
Constituency parsing
T 3 #
Dependency parsing
Syntactic reasoning hint (back)
t 2 $$
Word segmentation
E 1220113888... $..
Noun phrases
POS tagging
A ALAN HAl $
Constituency parsing
E M 3
Dependency parsing
Table 16: Syntactic prompting on Ontonotes 4.

---

Tool augmentation + syntactic prompting
: ,
instructiion (involving
syntactic [ (syntactic reasoning hint (nx M (synacciic information from tool
M TTAT X )1
M 1 [] \n (syntactic reasoning hint
(back))
(questions of each label)... ...
[aak instruction (involving syntactic tool))
Word segmentation
$
POS tagging
$
Constituency parsing
S 4
Dependency parsing
4
(Syntactic information from tool)
Word segmentation
POS tagging
E/R DNENNN WNN EINN TE/ ÆUNR A
Wb/VV',n
Constituency parsing
: (TOP) (IP) (NP (NP (NR +)) (NP (NN H (NN )) (NN E)) (VP (PP (P AE) (NP (NR ))) (VP (VV A
= n
Dependency parsing
$
P ". I' "nsu'"'', " 'prep''..
"E',' "o " "root" [J
(syntactic reasoning hint (front))
Word segmentation
POS tagging
Constituency parsing
Dependency parsing
{syntactic reasoning hint (back))
Word segmentation
POS tagging
Constituency parsing
Dependency parsing
Table 17: Tool augmentation w. / wo. syntactic prompting on Ontonotes 4. If using syntactic prompting, fill in (syntactic reasoning hint); If not, discard |syntactic reasoning hint).

---

Syntactic prompting
Given entity label set: ['Person'. 'Organization', 'Location', 'Facility', 'Weapon', 'Vehicle', 'Geo-Political Entity Nn task instruction (involving syntactic tool) ) (syntactic reasoning hint (front)) \nText: Could Tony Blair be in line for gold medal 2/n (syntactic information from tool
Question: What are the named entities labeled as 'Person' in the text? Provide the answer in the following JSON format: [[ (('Entity Name': 'Entity Label') }}. If there is no corresponding entity, return the following empty list: D. AnAnswer: [syntactic reasoning hint (back))
(questions of each label)
Syntactic reasoning hint (front)
Noun phrases
First, you should recognize the noun phrases. Then, you should recognize named entities based on the noun phrases.
POS tagging
First, you should perform Part-of-Speech tagging. Then, you should recognize named entities based on the Part-of-Speech tags.
Constituency
First, you should perform constituency parsing, Then, you should recognize named entities based on the constituency tree.
parsing
Dependency parsing
First, you should perform dependency parsing. Then, you should recognize named entities based on the dependency tree.
Syntactic reasoning hint (back)
Noun phrases
First, let's recognize the noun phrases. Then, recognize named entities based on the noun phrases.
POS tagging
First, let's perform Part-of-Speech tagging. Then, we recognize named entities based on the Part-of-Speech tags.
Constituency
First, let's perform constimency parsing. Then, we recognize named entities based on the constituency tree.
parsing
Dependency
First, let's perform dependency parsing. Then, we recognize named entities based on the dependency tree.
parsing
Table 18: Syntactic prompting on ACBOS.

---

Tool augmentation syntactic prompting
Given entity label set: ["Person', 'Organization', 'Location', 'Facility', 'Weapon', 'Vehicle', 'Geo-Political Entity" /nn task task instruction (involving syntactic tool)) (syntactic reasoning hint (front)) TeTxx Could Tony Blair be in line for gold medal?'n coddal?in(syntaacii (syntactic information from tool)",
Question: What are the named entities labeled as "Person' in the text? Provide the answer in the following JSON format: l('Entity Name": 'Entity Label") J. If there no corresponding entity, return the following empty list: [] AnAnswer: (syntactic reasoning hint (back))
(questions of each label) ...
(TTss instruction (involving syntactic tool))
POS tagging
Given the text and the corresponding Part-of-Speech tags, please recognize the named entities in the given text.
Constituency
Given the text and the corresponding constituency tree, please recognize the named entities in the given text.
parsing
Dependency
Given the text and the corresponding dependency tree, please recognize the named entities in the given text.
parsing
(Syntactic information from tool]
POS tagging
Part-of-Speech tags: Could/JJ Tony/NN Blair/NN be/NN in/P line/NN for/P a/CD gold/NN medal/NN 9/PUAn
Constituency
Constituency tree: (TOPln (NP'n (NPin (NP (ADJP (JJ Could)) (NP (NN Tony) (NN Blair) (NP (NN be) (PP (P in) (NP (NN line)))) (PP (P for) (NP (QP (CD a)) (NP (NN gold) (NN mn
parsing
Dependency
Dependency tree: [['Could', 'be', 'amod'], ['Tony'. 'be', 'nn']. ['Blair', 'be', 'nn'], ('be', "?" 'root'], ['in', 'be', 'prep'], ['line', 'in', 'pobj'], ['for', 'be', 'prep'], l'a', 'medal', 'nummod"], ], I'gold', 'medal', 'nn'l, ('medal', 'for', 'pobj'], ['?', 'be", 'punct']] JJ\n
parsing
(syntactic reasoning hint (front))
POS tagging
Please infer named entities step by step from the text based on the given Part-of- Speech tags.
Constituency
Please infer named entities step by step from the text based on the given constituency tree.
parsing
Dependency
Please infer named entities step by step from the text based on the given dependency tree.
parsing
(syntactic reasoning hint (back))
POS tagging
Let's infer named entities step by step from the text based on the given Part-of- Spreehh tags.
Constituency
Let's infer named entities step by step from the text based on the given constituency tree.
parsing
Dependency
Let's infer named entities step by step from the text based the given dependency tree.
parsing
Table 19: Tool augmentation w. wo. syntactic promptings on ACEOS. If using syntactic prompting, fill in (syntactic reasoning hint]; If not, discard (syntactic reasoning hint).


Reviews:
[{"review_id": "dQO0fzGFxb", "reviewer": "Reviewer_5Z6K", "summary": "", "questions": "Question A:\nThe authors mention chain-of-thought (CoT) prompting but do not compare against standard CoT techniques. Could the authors include experiments on conventional CoT methods to better analyze the improvements from the reasoning-based approaches?\n\nQuestion B:\nFor a more comprehensive assessment, could the authors evaluate their methods on additional widely-used NER datasets like CoNLL-2003 and WNUT-17? This would provide more context on the generalization capability.\n\nQuestion C:\nThe authors used ChatGPT in the experiments. How do the authors think their techniques will transfer to other LLMs? Could the authors provide any preliminary analysis on another large model to demonstrate wider applicability?\n\nQuestion D:\nThe results show significant improvements over the vanilla method, but there is still a large gap compared to supervised state-of-the-art. What are some potential ways to further enhance the reasoning and reduce this gap?\n\nQuestion E:\nThe authors identified some optimization directions based on the error analysis. Could the authors elaborate on their next steps to address these limitations and refine the reasoning techniques?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and feedback. We hope these could address your concerns.\n\n**Question A: Not including standard chain-of-thought (CoT).**\n\nWe conduct preliminary experiments to evaluate the proposed strategies under few-shot settings. Here, one general domain dataset, Ontonotes 4, and one domain-specific dataset, PowerPlantFlat, are employed for demonstrations. \n\nWork [1] investigates few-shot CoT on NER by generating intermediate rationales with ChatGPT. We take a different perspective: we encourage the model to explore syntactic information as their intermediate thinking. The following are the adaptations of our proposed zero-shot strategies to the few-shot setting.\n* Decomposed-QA: We provide the complete dialogue of each demonstration and let the model complete the dialogue of the test sample.\n* Syntactic prompting: For the test sample, we ask the model to first perform syntactic analysis and then recognize entities. For demonstrations, we use parsing tools to generate intermediate syntactic parsing results. \n* Tool augmentation: We provide both the text and the syntactic information for the demonstrations and test sample.\n* Self-consistency with two-stage majority voting: the method is the same as the zero-shot setting.\n\nThe table below shows the experiment results under the **3-shot** setting. We randomly sample three sets of demonstrations and report the means and standard deviations. \n\n|                            | Ontonotes 4  | PowerPlantFlat  |\n|----------------------------|---------------|-----------------|\n| Vanilla                    | 38.71 (3.34)  | 35.81 (2.94)    |\n| Decomposed-QA              | **43.30** (1.84)   | **43.75**(3.06)    |\n| Syn.(front)                |               |                 |\n| Word segmentation          | 40.12 (3.22)  | **37.33** (2.70)    |\n| POS tag                    | **44.11** (3.52)  | 36.42 (0.26)    |\n| Constituency tree          | 38.81 (2.38)  | 33.66 (2.50)    |\n| Dependency tree            | 35.05 (1.41)  | 36.16 (1.43)    |\n| Syn. + Consistency         |               |                 |\n| Word segmentation          | 41.28 (3.65)  | 38.7 (2.27)     |\n| POS tag                    | **44.93** (4.02)  | 36.19 (0.65)    |\n| Constituency tree          | 41.89 (3.50)  | 36.05 (2.65)    |\n| Dependency tree            | 37.98 (2.56)  | **39.06** (0.78)    |\n| Tool.                      |               |                 |\n| Word segmentation          | 42.33 (3.14)  | **39.43** (1.91)    |\n| POS tag                    | **42.51** (2.44)  | 37.04 (0.72)    |\n| Constituency tree          | 38.51 (4.17)  | 35.14 (2.84)    |\n| Dependency tree            | 36.12 (1.93)  | 33.54 (1.44)    |\n| Tool. + Consistency        |               |                 |\n| Word segmentation          | 40.49 (12.05) | **41.09** (2.71)    |\n| POS tag                    | **43.28** (2.69)  | 38.66 (2.05)    |\n| Constituency tree          | 40.12 (4.31)  | 35.54 (2.66)    |\n| Dependency tree            | 38.39 (2.50)  | 35.79 (1.87)    |\n| Tool. + Syn.               |               |                 |\n| Word segmentation          | **43.11** (2.52)  | **39.69** (2.61)    |\n| POS tag                    | 42.44 (2.67)  | 36.35 (1.81)    |\n| Constituency tree          | 38.31 (3.30)  | 34.45 (2.53)    |\n| Dependency tree            | 35.21 (2.26)  | 34.1 (1.06)     |\n| Tool. + Syn. + Consistency |               |                 |\n| Word segmentation          | **42.18** (2.29)  | **41.43** (1.78)    |\n| POS tag                    | 41.53 (3.25)  | 37.95 (2.03)    |\n| Constituency tree          | 41.57 (4.54)  | 35.32 (3.82)    |\n| Dependency tree            | 40.33 (4.87)  | 35.85 (2.14)    |\n\nFor each dataset, we choose the two most effective syntactic augmentation methods and evaluate them on different numbers of demonstrations. The results are shown below. Due to time constraints, for combinations of decomposed-QA, syntactic augmentation, and self-consistency under few-shot settings, we leave them as our future work.\n\n| # Demonstrations                 | 0            | 3            | 5            | 10            |\n|---------------------------|--------------|--------------|--------------|---------------|\n| Ontonotes 4               |              |              |              |               |\n| Vanilla                   | 35.16 (1.57) | 38.67 (3.57) | 44.51 (5.78) | 52.45 (4.13)  |\n| Tool. (word segmentation) | **40.78** (2.58) | 42.48 (3.34) | 47.16 (5.42) | 54.40 (2.68)  |\n| Syn. (word segmentation)  | 37.94 (1.49) | **43.89** (3.67) | **50.70** (7.26) | **56.71** (3.70)  |\n| PowerPlantFlat            |              |              |              |               |\n| Vanilla                   | 27.85        | 35.81 (2.94) | 37.44 (3.88) | 41.13 (4.89)  |\n| Tool. (word segmentation) | **32.41**        | **39.43** (1.91) | **41.12** (4.35) | 42.05 (4.74)  |\n| Syn. (POS tag)            | 28.09        | 37.84 (2.59) | 39.72 (2.79) | **42.52** (3.71)  |\n\nThe results in the above two tables show that our proposed strategies also achieve significant improvements under few-shot settings.\n\nLast but not least, we emphasize that the goal of this work is to explore **zero-shot** NER with ChatGPT’s reasoning abilities. Standard CoT techniques [2] belong to **few-shot** learning strategies, which are **not** the main focus of our work.\n\n[1] Han, R., Peng, T., Yang, C., Wang, B., Liu, L., & Wan, X. (2023). Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors. ArXiv, abs/2305.14450.\n\n[2] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.\n\n**Question B: More comprehensive assessment.**\n\nBelow are the results on more datasets, including general-domain and domain-specific datasets (i.e., biomedical domain). The syntactic augmentation and two-stage voting are conducted under decomposed-QA paradigm.\n\n* We found that the proposed reasoning techniques cannot guarantee performance improvements on CoNLL-2003 and WNUT-16 and even hurt the performance. This is presumably because the label logic of these two datasets is not suitable for decomposition, and the syntactic information generated for them is noisy.\n* However, the proposed methods achieve significant improvements in biomedical domain datasets BC5CDR [1], BioNLP11 [2], and CRAFT [3] [4]. This demonstrates that the proposed methods can also improve zero-shot NER of other challenging specific domains besides the electric power domain evaluated in our paper.\n* Plus the **seven** datasets evaluated in our paper (Tables 1 and 2 of our paper), we have achieved significant improvements on **ten** datasets in total, including Chinese and English, on both domain-specific and general-domain. This demonstrates the effectiveness of the proposed reasoning strategies.\n\n| Dataset           |  CoNLL03     | WNUT17       | BC5CDR       | BioNLP11     | CRAFT        |\n|-------------------|--------------|--------------|--------------|--------------|--------------|\n| Vanilla           | **69.42** (0.91) | **46.61** (2.97) | 61.28 (3.11) | 51.29 (2.48) | 21.66 (1.41) |\n| Decomposed-QA     | 59.67 (0.36) | 42.39 (1.99) | **65.45** (0.89) | **55.3** (0.54)  | **23.99** (2.65) |\n| Syn.              |              |              |              |              |              |\n| Front             |              |              |              |              |              |\n| Noun phrases      | **57.13** (0.41) | **39.75** (2.42) | 64.41 (1.74) | 52.73 (1.89) | 22.92 (3.04) |\n| POS tag           | 55.14 (0.98) | 39.74 (1.98) | **66.24** (2.40) | 53.97 (0.99) | **23.59** (2.01) |\n| Constituency tree | 56.36 (1.07) | 39.66 (1.51) | 64.7 (0.80)  | 53.98 (1.24) | 23.84 (1.94) |\n| Dependency tree   | 54.27 (1.54) | 38.36 (0.37) | 65.56 (1.20) | **54.37** (0.83) | 23.51 (2.06) |\n| Back              |              |              |              |              |              |\n| Noun phrases      | **58.89** (1.39) | 36.47 (1.10) | **60.44** (0.57) | 51.99 (1.27) | 21.71 (1.89) |\n| POS tag           | 56.12 (1.54) | **38.12** (0.55) | 58.19 (1.55) | 54.41 (2.62) | 22.69 (3.76) |\n| Constituency tree | 55.66 (2.17) | 37.75 (2.04) | 47.81 (1.99) | 43.58 (1.44) | 23.86 (1.30) |\n| Dependency tree   | 58.36 (1.11) | 37.49 (1.56) | 59.69 (0.76) | **55.23** (0.77) | **22.84** (1.59) |\n| Tool.             |              |              |              |              |              |\n| POS tag           | **62.79** (2.54) | 43.81 (2.15) | **66.4** (1.44)  | **52.38** (0.35) | 24.54 (3.87) |\n| Constituency tree | 60.96 (0.34) | **44.3** (1.02)  | 65.02 (3.00) | 51.44 (0.81) | 23.88 (2.24) |\n| Dependency tree   | 59.23 (3.13) | 41.6 (1.30)  | 62.79 (2.62) | 42.71 (0.73) | **24.86** (2.45) |\n| Tool. + Syn.      |              |              |              |              |              |\n| Front             |              |              |              |              |              |\n| POS tag           | **63.46** (1.02) | 43.9 (2.21)  | 64.24 (1.83) | 49.87 (0.85) | **25.05** (2.12) |\n| Constituency tree | 59.59 (0.99) | **44.68** (2.59) | **65.43** (2.51) | **50.6** (1.39)  | 24.5 (2.76)  |\n| Dependency tree   | 57.93 (2.05) | 40.96 (3.57) | 59.93 (1.88) | 40.18 (1.63) | 24.46 (2.50) |\n| Back              |              |              |              |              |              |\n| POS tag           | 58.08 (0.27) | **39.32** (4.42) | **60.84** (3.47) | **47.9** (1.88)  | 13.71 (1.90) |\n| Constituency tree | **58.95** (2.19) | 37.91 (5.46) | 54.57 (1.86) | 42.27 (2.78) | 20.64 (1.85) |\n| Dependency tree   | 54.68 (1.27) | 36.7 (3.07)  | 57.28 (1.37) | 45.62 (3.46) | **24.05** (4.05) |\n\n[1] Wei C H, Peng Y, Leaman R, et al. Assessing the state of the art in biomedical relation extraction: overview of the BioCreative V chemical-disease relation (CDR) task[J]. Database, 2016, 2016: baw032.\n\n[2] Pyysalo S, Ohta T, Rak R, et al. Overview of the ID, EPI and REL tasks of BioNLP Shared Task 2011[C]//BMC bioinformatics. BioMed Central, 2012, 13: 1-26.\n\n[3] Bada M, Eckert M, Evans D, et al. Concept annotation in the CRAFT corpus[J]. BMC bioinformatics, 2012, 13(1): 1-20.\n\n[4] Crichton G, Pyysalo S, Chiu B, et al. A neural network multi-task learning approach to biomedical named entity recognition[J]. BMC bioinformatics, 2017, 18(1): 1-14.\n\n**Question C: Other LLMs.**\n\nTo achieve similar complex reasoning and instruction comprehension abilities, the model's size and training techniques are generally required to be similar to ChatGPT. However, we currently cannot afford to run such a large model locally. So, we focus on the evaluation of ChatGPT. As a powerful but closed-source model, the assessment of ChatGPT is important for this community. And we leave the evaluation of other LLMs to our future work.\n\n**Question D: Potential ways to reduce the gap between fully-supervised SOTA.**\n\nWe believe that the gap can be further reduced from the following aspects:\n\n(a) Combining different syntactic information to achieve enhanced reasoning abilities\n\n(b) Adding entity type information.\n\n(c) Exploring label logic and combinations in decomposed-QA.\n\n(d) Using parsing tools with better support for different languages, or combining information from different parsing tools.\n\n(e) Utilizing a small number of labeled demonstrations and exploring effective few-shot learning strategies to improve performance.\n\nFor (a)-(d), please refer to responses to Question E for preliminary results. \n\nFor (e), please refer to responses to Question A for preliminary results.\n\n**Question E: Next steps to address limitations and refine reasoning techniques.**\n\nWe believe that reasoning techniques can be improved from the following aspects:\n\n(a) Combining different syntactic information to achieve enhanced reasoning abilities. This might alleviate the negative impact of the noisy syntactic information. The following table shows the preliminary experimental results of combining four types of syntactic information. The results show that the ensemble effectively improve the performance.\n\n|                     | Ontonotes 4  | PowerPlantFlat |\n|----------------------------|--------------|----------------|\n| Vanilla                    | 35.16 (1.57) | 27.85          |\n| Tool. + Syn. + Consistency |              |                |\n|   Word segmentation        | 42.46 (2.20) | 40.31          |\n|   POS tag                  | 40.86 (2.48) | 38.21          |\n|   Constituency tree        | 41.36 (3.58) | 35.76          |\n|   Dependency tree          | 40.49 (3.49) | 39.97          |\n| Ensemble                   | **43.26** (1.99) | **44.01**          |\n\n(b) Adding entity type information. Providing an introduction to entity types to enhance the model's understanding of each type. The following table shows the preliminary experimental results. The results show that the effectiveness of this strategy varies on different datasets. We will explore this in detail in our future work.\n\n|        | Ontonotes 4  | PowerPlantFlat |\n|---------------|--------------|----------------|\n| Vanilla       | 35.16 (1.57) | **27.85**          |\n|   + type info | **43.81** (3.89) | 25.78          |\n| Decomposed-QA | 38.79 (1.66) | **36.57**          |\n|   + type info | **44.39** (2.12) | 35.87          |\n\n(c) Exploring label logic and combinations in decomposed-QA. There might exist different relations between labels. Combining similar/related labels in decomposed-QA might achieve further improvements. Here, we conduct preliminary experiments. Below are the used one-by-one label orders and label group orders of Ontonotes 4 and PowerPlantFlat. We show them all in English for readability.\n* Ontonotes 4: \n  * One-by-one order: \\[[Person], [Location], [Organization], [Geo-political entity]]\n  * Label group order: \\[[Person, Organization], [Geo-political entity, Location]]\n* PowerPlantFlat:\n  * One-by-one order: \\[[Device Identity], [Device Name], [System Identity], [System Name], [Component Name], [Location], [Person]]\n  * Label group order: \\[[Device Identity, Device Name], [System Identity, System Name], [Component Name], [Location], [Person]]\n\nThe results below reveal that appropriate label combinations can also achieve superior performance in decomposed-QA paradigm.\n\n| Dataset           | Ontonotes 4  | PowerPlantFlat |\n|-------------------|--------------|----------------|\n| Vanilla           | 35.16 (1.57) | 27.85          |\n| QA - one-by-one   | 38.79 (1.66) | 36.57          |\n| QA - label combinations | 40.01 (1.58) | 36.37          |\n\n(d) Using parsing tools with better support for different languages, or combining information from different parsing tools. Here, we evaluate another parsing tool, stanza, from Stanford University. To exclude the influence of other reasoning techniques and only observe the initial effects of different parsing tools, we conduct tool augmentation on four types of syntactic information under the vanilla paradigm. The results are shown in the table below. \n\n| Dataset (Language) | ACE05 (English) |              | PowerPlantFlat (Chinese) |        |\n|--------------------|-----------------|--------------|--------------------------|--------|\n| Parseing tool      | Hanlp           | Stanza       | Hanlp                    | Stanza |\n| Word segmentation  | \\               | \\            | **31.76**                    | 28.55  |\n| POS tag            | 30.71 (1.43)    | **31.51** (0.97) | **25.91**                    | 20.6   |\n| Constituency tree  | 28.06 (2.03)    | **28.64** (1.08) | **21.39**                    | 19.76  |\n| Dependency tree    | 28.32 (1.70)    | **28.38** (1.76) | **23.52**                   | 20.97  |\n\nThe results show that the quality of parsing tools and their help in reasoning are related to the language used. Stanza, from English developers, has higher support for the English dataset, ACE05; Hanlp, from Chinese developers, has better support for the Chinese dataset, PowerPlantFlat. Therefore, when using our methods, we can choose parsing tools that are more suitable for the corresponding language.\n\n**Reasons To Reject: The augmentation strategies proposed lack significant novelty.**\n\nTo the best of our knowledge, our work is the first to comprehensively adapt these prevalent reasoning techniques to zero-shot NER. Also, our work is the first to use syntactic augmentation approaches in the NER task with large language models.\n\n**Reasons To Reject: Not including standard chain-of-thought (CoT) methods.**\n\nOur work aims to present an empirical study of ChatGPT's reasoning capabilities on zero-shot NER. However, standard CoT methods belong to few-shot methods, which are beyond the scope of this work. Besides, we claimed in our paper that our proposed methods achieve remarkable improvements on zero-shot NER, which is supported by the results in Table 1&2.\n\n**Reasons To Reject: Not evaluating on datasets CoNLL-2003 and WNUT-17.**\n\nWe have evaluated our proposed strategies on seven datasets in our paper, as shown in Tables 1&2. These datasets include both Chinese and English datasets and both domain-specific and general-domain scenarios. The experimental results fully support our conclusion that our proposed reasoning techniques achieve significant improvements on zero-shot NER.\n\n\n**Missing reference**\n\nThis work has been cited in Line 132 of Section 2 Related Work in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors have comprehensively explored decomposition and augmentation techniques to enhance the zero-shot NER capabilities of large language models. Their proposed strategies, including label-wise task decomposition, syntactic augmentations, and self-consistency mechanisms, are shown to significantly boost NER performance across both English and Chinese datasets spanning different domains. The experiments demonstrate the efficacy of adapting common reasoning techniques in facilitating multi-step inference for zero-shot NER using the ChatGPT model. This empirical study provides valuable insights and guidance for low-resource NER settings where leveraging the reasoning and generalization abilities of large pre-trained models is critical. The authors have conducted thorough experiments to validate their proposed reasoning-based approaches for improving zero-shot NER.", "reasons_to_reject": "The augmentation strategies proposed lack significant novelty - both the decomposed and syntactic augmentation approaches are basic and commonly used techniques in NLP tasks. The paper mentions chain-of-thought (CoT) but does not include experiments on standard CoT methods or typical in-context learning approaches with LLMs. Additionally, the authors did not evaluate on some commonly used NER datasets like CoNLL-2003 and WNUT-17. While the empirical study is solid, the enhancements explored are not particularly innovative. Comparing against standard CoT and few-shot learning strategies on more NER benchmarks would provide a more comprehensive assessment and additional context for the claims.", "questions_to_authors": "Question A:\nThe authors mention chain-of-thought (CoT) prompting but do not compare against standard CoT techniques. Could the authors include experiments on conventional CoT methods to better analyze the improvements from the reasoning-based approaches?\n\nQuestion B:\nFor a more comprehensive assessment, could the authors evaluate their methods on additional widely-used NER datasets like CoNLL-2003 and WNUT-17? This would provide more context on the generalization capability.\n\nQuestion C:\nThe authors used ChatGPT in the experiments. How do the authors think their techniques will transfer to other LLMs? Could the authors provide any preliminary analysis on another large model to demonstrate wider applicability?\n\nQuestion D:\nThe results show significant improvements over the vanilla method, but there is still a large gap compared to supervised state-of-the-art. What are some potential ways to further enhance the reasoning and reduce this gap?\n\nQuestion E:\nThe authors identified some optimization directions based on the error analysis. Could the authors elaborate on their next steps to address these limitations and refine the reasoning techniques?", "paper_topic_and_main_contributions": "This paper presents an empirical study of zero-shot named entity recognition (NER) using the large language model ChatGPT. The authors adapt common reasoning techniques used with LLMs for logical and arithmetic reasoning to the NER task. Their main contributions are proposing four strategies tailored for NER to elicit reasoning from ChatGPT: task decomposition by entity labels, syntactic augmentation, self-consistency, and two-stage majority voting. Experiments on English and Chinese benchmarks, including domain-specific datasets, demonstrate that their proposed techniques significantly improve zero-shot NER performance across domains. The analysis of error types provides insights into optimization directions. Overall, this is a thorough investigation of reasoning abilities for zero-shot NER using an available robust LLM. The adapted reasoning strategies are novel in facilitating multi-step inference for NER. The empirical results have important implications for low-resource NER scenario using LLM.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Try adding: Han, R., Peng, T., Yang, C., Wang, B., Liu, L., & Wan, X. (2023). Is Information Extraction Solved by ChatGPT? An Analysis of Performance, Evaluation Criteria, Robustness and Errors. ArXiv, abs/2305.14450. to your reference."}}, {"review_id": "0WSwcUhzqg", "reviewer": "Reviewer_B3Rk", "summary": "", "questions": "Did you explore applying this approach to GPT-4 yet? GPT-4 usually outperforms ChatGPT by a considerable margin.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We hope these could address your concerns.\n\n**Question: Exploration of applying the approaches to GPT-4.**\n\nDue to the price being 20-30 times more expensive than GPT-3.5 (20 times for input and 30 times for output), we currently cannot afford the extensive use of GPT-4. We tested some individual examples and found that GPT-4 has excellent reasoning and comprehension abilities. Our method might achieve a more significant improvement on GPT-4. We will verify it in our future work.\n\n**Reasons To Reject: Efficiency concerns.**\n\nUsing label combinations in decomposed-QA could significantly reduce the rounds of inquiries. Combining similar/related labels in one round of inquiry can improve the efficiency. Due to time and budget constraints, we only explored one-by-one label orders in this work. We will explore label combinations according to label relations and logic in our future work. Here, we conduct preliminary experiments. Below is the used one-by-one label order and label group order of Ontonotes 4 and PowerPlantFlat. We show them all in English for readability.\n* Ontonotes 4: \n  * One-by-one order: \\[[Person], [Location], [Organization], [Geo-political entity]]\n  * Label group order: \\[[Person, Organization], [Geo-political entity, Location]]\n* PowerPlantFlat:\n  * One-by-one order: \\[[Device Identity], [Device Name], [System Identity], [System Name], [Component Name], [Location], [Person]]\n  * Label group order: \\[[Device Identity, Device Name], [System Identity, System Name], [Component Name], [Location], [Person]]\n  \nThe results below reveal that appropriate label combinations can also achieve superior performance in decomposed-QA paradigm.\n\n| Dataset           | Ontonotes 4  | PowerPlantFlat |\n|-------------------|--------------|----------------|\n| Vanilla           | 35.16 (1.57) | 27.85          |\n| QA - one by one   | 38.79 (1.66) | 36.57          |\n| QA - label groups | 40.01 (1.58) | 36.37          |\n\n\n\n**Reasons To Reject: Experimental results.**\n\n(a) Comparison to fully-supervised SOTA.\n\nThe goal of this work is to explore the performance boundary of zero-shot NER with ChatGPT. With various tailored reasoning techniques, our results show that the zero-shot results still need to catch up with fully-supervised SOTA. We consider this conclusion also contributes to the community.\n\n(b) Improvements of few-shot prompting. \n\nWe conduct preliminary experiments on few-shot settings. We encourage the model to explore syntactic information as their intermediate thinking. The following are the adaptations of our proposed zero-shot strategies to the few-shot setting.\n* Decomposed-QA: We provide the complete dialogue of each demonstration and let the model complete the dialogue of the test sample.\n* Syntactic prompting: For the test sample, we ask the model to first perform syntactic analysis and then recognize entities. For demonstrations, we use parsing tools to generate intermediate syntactic parsing results. \n* Tool augmentation: We provide both the text and the syntactic information for the demonstrations and test sample.\n* Self-consistency with two-stage majority voting: the method is the same as in the zero-shot setting.\n\nThe following table shows the experiment results under the **3-shot** setting. We randomly sample three sets of demonstrations and report the means and standard deviations. \n\n|                            | Ontonotes 4  | PowerPlantFlat  |\n|----------------------------|---------------|-----------------|\n| Vanilla                    | 38.71 (3.34)  | 35.81 (2.94)    |\n| Decomposed-QA              | **43.30** (1.84)   | **43.75**(3.06)    |\n| Syn.(front)                |               |                 |\n| Word segmentation          | 40.12 (3.22)  | **37.33** (2.70)    |\n| POS tag                    | **44.11** (3.52)  | 36.42 (0.26)    |\n| Constituency tree          | 38.81 (2.38)  | 33.66 (2.50)    |\n| Dependency tree            | 35.05 (1.41)  | 36.16 (1.43)    |\n| Syn. + Consistency         |               |                 |\n| Word segmentation          | 41.28 (3.65)  | 38.7 (2.27)     |\n| POS tag                    | **44.93** (4.02)  | 36.19 (0.65)    |\n| Constituency tree          | 41.89 (3.50)  | 36.05 (2.65)    |\n| Dependency tree            | 37.98 (2.56)  | **39.06** (0.78)    |\n| Tool.                      |               |                 |\n| Word segmentation          | 42.33 (3.14)  | **39.43** (1.91)    |\n| POS tag                    | **42.51** (2.44)  | 37.04 (0.72)    |\n| Constituency tree          | 38.51 (4.17)  | 35.14 (2.84)    |\n| Dependency tree            | 36.12 (1.93)  | 33.54 (1.44)    |\n| Tool. + Consistency        |               |                 |\n| Word segmentation          | 40.49 (12.05) | **41.09** (2.71)    |\n| POS tag                    | **43.28** (2.69)  | 38.66 (2.05)    |\n| Constituency tree          | 40.12 (4.31)  | 35.54 (2.66)    |\n| Dependency tree            | 38.39 (2.50)  | 35.79 (1.87)    |\n| Tool. + Syn.               |               |                 |\n| Word segmentation          | **43.11** (2.52)  | **39.69** (2.61)    |\n| POS tag                    | 42.44 (2.67)  | 36.35 (1.81)    |\n| Constituency tree          | 38.31 (3.30)  | 34.45 (2.53)    |\n| Dependency tree            | 35.21 (2.26)  | 34.1 (1.06)     |\n| Tool. + Syn. + Consistency |               |                 |\n| Word segmentation          | **42.18** (2.29)  | **41.43** (1.78)    |\n| POS tag                    | 41.53 (3.25)  | 37.95 (2.03)    |\n| Constituency tree          | 41.57 (4.54)  | 35.32 (3.82)    |\n| Dependency tree            | 40.33 (4.87)  | 35.85 (2.14)    |\n\nFor each dataset, we choose the two most effective syntactic augmentation methods and evaluate them on different numbers of demonstrations. The results are shown below. Due to time constraints, for combinations of decomposed-QA, syntactic augmentation, and self-consistency under few-shot settings, we leave them as our future work.\n\n| # Demonstrations                 | 0            | 3            | 5            | 10            |\n|---------------------------|--------------|--------------|--------------|---------------|\n| Ontonotes 4               |              |              |              |               |\n| Vanilla                   | 35.16 (1.57) | 38.67 (3.57) | 44.51 (5.78) | 52.45 (4.13)  |\n| Tool. (word segmentation) | **40.78** (2.58) | 42.48 (3.34) | 47.16 (5.42) | 54.40 (2.68)  |\n| Syn. (word segmentation)  | 37.94 (1.49) | **43.89** (3.67) | **50.70** (7.26) | **56.71** (3.70)  |\n| PowerPlantFlat            |              |              |              |               |\n| Vanilla                   | 27.85        | 35.81 (2.94) | 37.44 (3.88) | 41.13 (4.89)  |\n| Tool. (word segmentation) | **32.41**        | **39.43** (1.91) | **41.12** (4.35) | 42.05 (4.74)  |\n| Syn. (POS tag)            | 28.09        | 37.84 (2.59) | 39.72 (2.79) | **42.52** (3.71)  |\n\nThe results in the above two tables show that our proposed strategies also achieve significant improvements under few-shot settings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clear and easy to understand. The figures are very clear. The appendices are also thoroughly detailed.\n2. The paper proposes a novel approach utilizing LLMs for NER and conducts extensive experiments. The paper also conducts a detailed error analysis. This could assist future research focused on the application of LLMs for NER.\n3. The paper conducts experiments on both Chinese and English datasets.", "reasons_to_reject": "1. Efficiency concerns. The proposed decomposed question-answering paradigm breaks down the NER task into simpler subproblems by labels. Therefore, the approach requires multiple inquiries to ChatGPT. For one example, the number of inquiries is proportional to the number of entity types in the schema. Such efficiency is relatively low, which may affect the real-world application of this algorithm. \n2. Experimental results. The performance of the proposed approach is much lower than the previous SoTA. Although the comparison may not be entirely fair, as the SoTA employs a fully supervised methodology whereas this approach is zero-shot, the performance of this approach is still quite a distance away from being practically usable. I wonder if the model's performance could potentially be improved by using few-shot prompting.", "questions_to_authors": "Did you explore applying this approach to GPT-4 yet? GPT-4 usually outperforms ChatGPT by a considerable margin.", "paper_topic_and_main_contributions": "The paper introduces a new approach for zero-shot named entity recognition (NER) using ChatGPT. The proposed approach contains two key components: decomposed question-answering paradigm and syntactic augmentation. The decomposed question-answering paradigm breaks down the NER task into simpler subproblems by labels. The syntactic augmentation method consists of two aspects: syntactic prompting and tool augmentation. The paper also adopts the self-consistency method to enhance performance. The paper conducts extensive experiments on various English and Chinese datasets. The experimental results demonstrate the effectiveness of the proposed approach.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YvnrJxMri6", "reviewer": "Reviewer_5LaW", "summary": "", "questions": "Experimental results show a large gap between the zero-shot LLM and the state-of-the-art fully supervised model, showing that the significant limitation of LLMs in NER. However, LLMs have been shown capable in many complex tasks, do you think the ability in NER still important? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. The major comments are answered below.\n\n**Question: Importance of LLMs ability in NER.**\n\nFirst, the zero-shot ability of LLMs could facilitate low-resource information extraction (IE). IE and knowledge construction are fundamental research fields with broad applications [1]. If the powerful zero-shot ability of LLMs can be extended to IE, we can achieve efficient IE and knowledge construction with extremely low resources.\n\nSecond, knowledge bases might improve the ability of LLMs. LLMs still encounter hallucinations or lack of knowledge when solving knowledge-intensive tasks [2]. With the assistance of up-to-date and scenario-specific knowledge bases, LLMs might provide more reliable applications. If LLMs can achieve efficient knowledge extraction on its own and use the extracted knowledge to assist it in handling other tasks, the problem of hallucinations might be significantly alleviated. \n\nTherefore, we believe that the ability of LLMs in NER is important and worth exploring.\n\n[1] Zou X. A survey on application of knowledge graph[C]//Journal of Physics: Conference Series. IOP Publishing, 2020, 1487(1): 012016.\n\n[2] Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models. ICLR, 2023.\n\n**Reasons To Reject: The effects of individual LLM and parser.**\n\n(a) Other LLMs.\n\nTo achieve similar complex reasoning and instruction comprehension abilities, the model's size and training techniques are generally required to be similar to ChatGPT. However, we currently cannot afford to run such a large model locally. So, we focus on the evaluation of ChatGPT. As a powerful but closed-source model, the assessment of ChatGPT is important for this community. And we leave the evaluation of other LLMs to our future work.\n  \n(b) Other syntactic parsers.\n\nHere, we evaluate another parsing tool, stanza, from Stanford University. To exclude the influence of other reasoning techniques and only observe the initial effects of different parsing tools, we conduct tool augmentation on four types of syntactic information under the vanilla paradigm. The results are shown in the table below.\n\n| Dataset (Language) | ACE05 (English) |              | PowerPlantFlat (Chinese) |        |\n|--------------------|-----------------|--------------|--------------------------|--------|\n| Parseing tool      | Hanlp           | Stanza       | Hanlp                    | Stanza |\n| Word segmentation  | \\               | \\            | **31.76**                    | 28.55  |\n| POS tag            | 30.71 (1.43)    | **31.51** (0.97) | **25.91**                    | 20.6   |\n| Constituency tree  | 28.06 (2.03)    | **28.64** (1.08) | **21.39**                    | 19.76  |\n| Dependency tree    | 28.32 (1.70)    | **28.38** (1.76) | **23.52**                   | 20.97  |\n\n\nThe results show that the quality of parsing tools and their help in reasoning are related to the language used. Stanza, from English developers, has higher support for the English dataset, ACE05; Hanlp, from Chinese developers, has better support for the Chinese dataset, PowerPlantFlat. Therefore, when using our methods, we can choose parsing tools that are more suitable for the corresponding language.\n\n**Typos Grammar Style And Presentation Improvements.**\n\nThank you for kindly pointing out the typo error. We will fix it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This work shows the LLM can still be improved by the information produced by \"traditional\" NLP parsers. \n* A variety of syntactic information and settings are evaluated in the experiments. \n* Experimental results lead to the conclusions that are useful for the NLP community. ", "reasons_to_reject": "* Only one LLM (ChatGPT-3.5-Turbo) and one syntactic parser (Hanlp) are included in the experiments; the effects of individual LLM and parser are not clear. ", "questions_to_authors": "Experimental results show a large gap between the zero-shot LLM and the state-of-the-art fully supervised model, showing that the significant limitation of LLMs in NER. However, LLMs have been shown capable in many complex tasks, do you think the ability in NER still important? ", "paper_topic_and_main_contributions": "This paper explores the ability of an LLM (ChatGPT-3.5-Turbo) in zero-shot NER with additional syntactic information. The study incorporates a variety of ways to inform syntactic information in the prompt to ChatGPT 3.5 for performing the task of named entity recognition (NER). The experimental results show that the NER can be improved by providing syntactic information such as word segmentation (for Chinese) and POS tagging, while both constituency parsing and dependency parsing help relatively limited.\nThis is an interesting work showing the powerful LLM can still benefit from the information produced by the \"traditional\" NLP parsers. \n\nThe goal of this work is clear and well-motivated. The writing is easy to follow. A variety of settings were conducted in experiments. \nHowever, only one LLM (ChatGPT-3.5-Turbo) and one syntactic parser (Hanlp) were included. As a result, the effects of individual LLM and parser are not clear. More LLMs and parsers can be added for comprehension. In addition, is an even better result reachable when the golden-ground (human labeled) word-segmentation and POS tagging are fed? ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Figure 5 is unreadable for non Chinese speakers. \n\n* Putting footnote numbers after comma, period, and semi-colon. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a clever adaptation of recent advances in LLMs - task decomposition and step-by-step reasoning to the task of NER. Various strategies are used to incorporate syntax prediction as an intermediate step and break down the eventual decision making problem into stage-wise decisions and their impact is carefully studied. All the reviewers like this paper. There was an extensive discussion on this paper and more analyses were provided by the authors in support of their results. These new results look convincing to me.
Decision: Accept-Main