Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation
Authors: Yuanyuan Liang
Jianing Wang
Hanlun Zhu
Lei Wang
Weining Qian
Yunshi Lan
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Chain-of-Thought
Few-Shot
Knowledge Base Question Generation
TL;DR: The study aims to develop a method for Few-Shot Question Generation over Knowledge Bases by leveraging Large Language Models.

Abstract: The task of Question Generation over Knowledge Bases (KBQG) aims to convert a logical form into a natural language question. For
the sake of expensive cost of large-scale question annotation, the methods of KBQG under low-resource scenarios urgently need to be developed. However, current methods heavily rely on annotated data for fine-tuning, which is not well-suited for few-shot question generation. The emergence of Large Language Models (LLMs) has shown their impressive generalization ability in few-shot tasks. Inspired by Chain-of-Thought (CoT) prompting, which is an in-context learning strategy for reasoning, we formulate KBQG task as a reasoning problem, where the generation of a complete question is splitted into a series of sub-question generation. Our proposed prompting method KQG-CoT first retrieves supportive logical forms from the unlabeled data pool taking account of the characteristics of the logical form. Then, we write a prompt to explicit the reasoning chain of generating complicated questions based on the selected demonstrations. To further ensure prompt quality, we extend KQG-CoT into KQG-CoT+ via sorting the logical forms by their complexity. We conduct extensive experiments over three public KBQG datasets. The results demonstrate that our prompting method consistently outperforms other prompting baselines on the evaluated datasets. Remarkably, our KQG-CoT+ method could surpass existing few-shot SoTA results of the PathQuestions dataset by 18.25, 10.72, and 10.18 absolute points on BLEU-4, METEOR, and ROUGE-L, respectively.

Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generation
Yuanyuan Liang', Jianing Wang', Hanlun Zhu', Lei Wang"
Weining Qian', Yunshi Lan
East China Normal University, Singapore Management University leonyuany @ stt.cctu..ccm wtw.ccnu.cuu.cn, (lygwjn, timberflowing] @ gmail.com lei. sai.wang.201 @ whhessssmu.cdiee (wnqian, yslan) | @ dddase.com .uas..ccnu.duu.cn
Abstract
KB
Unlabeled data pool
The task of Question Generation over Knowl- edge Bases (KBQG) aims to convert a logi- cal form into natural language question. For the sake of expensive tion annotation, the methods of KBQG under low-resource scenarios urgently need de- veloped. However, current methods heavily rely on annotated data for fine-tuning, which not well-suited for few-shot question genera- tion. The emergence of Large Language Mod- els (LLMs) has shown their impressive gen- eralization ability in few-shot tasks. Inspired by Chain-of-Thought (CoT) prompting, which is an in-context learning strategy for reason- ing, formulate KBQG task as reasoning problem, where the generation of complete question split into a series of sub-question generation. Our proposed prompting method KQG-CoT first selects supportive logical forms from the unlabeled data pool taking account of the characteristies of the logical form. Then, construct task-specific prompt to guide LLMs generate complicated questions based on se- lective logic forms. To further ensure prompt quality, we extend KQG-CoT into KQG-CoT+ via sorting the logical forms by their com- plexity. We conduct extensive experiments over three public KBQG datasets. The results demonstrate that prompting method consis- tently outperforms other prompting baselines on the evaluated datasets. Remarkably, our KQG-CoT+ method could surpass existing few- shot So'T results the PathQuestions dataset by 18.25, 10.72, and 10. 18 absolute points BLEU-4, METEOR, and ROUGE-L, respec- tively.
Logical forms
of large-scale ques-
Supportive logical forms
(JOIN (R location. location.
containedby) (JOIN
Prompi
geography.riverr.cmm m.01mell))
LLM
Question
Where is the river that flows through poughkeepsie
Figure I: Overview of KQG-CoT framework.
context derived from Knowledge Bases (KBs) is in the form of logical. KBQG has attracted increas- ing interests from both the industry and academia due to its potential for data augmentation in QA systems (Xiong et al., 2022; Chen et al., 2023) and its ability to assist dialogue systems in creating coherent questions (Lee et al., 2018).
Existing studies (Kumar et al., 2019; Ke et al., 2021; Fei et al., 2022; Guo et al., 2022; Chen et al., 2023) for KBQG tasks have predominantly uti- lized neural network-based approaches and demon- strated impressive performance by conducting fine- tuning extensive training datasets. However, as the collection of KBQG data is labor-intensive, researchers start paying attention to the few-shot KBQG tasks (Xiong et al., 2022), where a great challenge is posed for suppliers with limited re- sources: 1) A great deal of annotated data is de- manded to allow the existing fine-tuned models to generalize well over different logical forms. How- ever, due to the limitations of fow-resouree avail- ability, training conventional models by fine-tuning on the full data becomes unrealistic. 2) A logical form is composed of entities, relations, and query grammar. Having logical forms with various com- binations of these basic components is crucial to
1 Introduction
Question generation task requires a system to pro- duce natural language questions based on the given context. KBQG (Guo et al., 2022) is one of the im- perative question generation tasks when the given
*Corresponding author

---

uphold the model's capability for compositional generalization. The lack of data leads to a com- positional challenge to the KBQG tasks (Gu et al., 2021). 3) Certain logical forms can become com- plex when operations such as aggregation, superla- tives, and comparisons are involved. Representing these logical forms presents additional challenges. Moreover, developing KBQG method that incor- porates diverse and elaborate expressions becomes particularly difficult in such low-resource scenar- ios (Xiong et al., 2022; Guo et al., 2022).
tiple steps. We concatenate the above rationales with the queried logical form to form a prompt. which guides a LLM to outcome a reasoning pro- cess of generating a complex question aligning with the logical form. We further improve KQG-CoT to KQG-CoT+ via sorting the supportive logical forms by complexity.
As previous methods rely heavily on the train- ing instances to fine-tune KBQG model. KQG- CoT does not need numerous logical form ques- tion pairs train the models. We test the perfor- mance of our prompting methods under few-shot setting on three public datasets, namely WebQues- tions (Kumar et al., 2019), PathQuestions (Zhou et al., 2018), and GrailQA (Gu et al., 2021). We conduct a comprehensive comparison with a range of commonly used CoT baseline methods including Auto-CoT (Zhang et al., 2023c), Active-CoT (Diao et al., 2023), Random-Coo (Brown et al., 2020) and so on. The experimental results show that we can outperform all of them with an observable mar- gin. Besides, we also compare with a set of SoTA systems trained with full data or few data. Our few-shot method could achieve competitive results to the full training methods. Remarkably, our few- shot method could surpass existing few-shot SoTA results of PathQuestions dataset by 18.25, 10.72 and 10.18 absolute points BLEU-4, METEOR and ROUGE-L, respectively.
Recentl, LLMs such as GPT-3 and Codex (Gao et al., 2022; Suzgun et al., 2022; Wei et al., 2022; Wang et al., 2023a) have proven their strong gen- eralizability wide range of few-shot and zero- shot tasks with CoT, including text interpretation, computer vision, planning and reasoning. Mcan- while, line of work (Kasner et al., 2022; Moiseev et al., 2022: Andrus et al., 2022; Trajanoska et al., 2023; Xie et al., 2023) validates that LLMSs have the strong capability to accurately capture the se- mantics of relations between values in the data, enabling to transform the structured instructions to narrative text. The above studies inspire to explore few-shot KBQG tasks by prompting LLMs with COT.
However, how to apply LLMs to KBQG with CoT is still unclear. On one hand, KBQG differs from tasks like code generation or question an- swering, as it involves incorporating KB-specific items into the input instead of self-contained narra- tives. Therefore, formatting the input in an easily understandable manner while considering the KB schema is crucial. On the other hand, the challenge lies in designing effective CoT prompts (Wei et al., 2022) that can enhance the performance of LLMs in the context of few-shot KBQG.
KQG-CoT provides simple but effective solu- tion few-shot KBQG problem, we expect it could serve as an important baseline for future investiga- tion to KBQG tasks under low-resource scenarios.
Our main contributions are summarized as fol- lows:
By encoding and clustering the skeletons of logical forms, we successfully retrieved sup- portive logical forms that are partieularly suit- able for constructing effective prompts.
In this work, we propose KQG-CoT framework, which is the first attempt for training-free free few-shot KBQG with LLMs. As shown in Figure our framework consists of two main steps, the objects of which are supportive logical forms selection from an unlabeled data pool and prompt construc- tion. To acquire coherent logical forms, we employ a clustering technique to carefully choose multiple logical forms that serve representatives, consid- ering both their syntactic and semantic characteris- tics. To construct prompt, inspired by the principle of CoT (Wei et al., 2022), we take the selected logi- cal forms as exemplars and write rationales to split the generation of a complete question into mul-
We reorganized the sequence of examples and utilized the CoT method to construct prompts that are highly effective for large language models.
The experimental results indicate that our method surpasses the baseline by a signifi- cant margin and achieves performance levels that are comparable to fine-tuned methods.

---

2 Related Work
the input (Ruis et al., 2023) help LLMs under- stand the tasks. Multiple studies (Su al., 2022; Rubin et al., 2022) explored the selection of ex- amples that are similar to the query during prompt construction. Recent researches (Lu et al., 2022a; Liu et al., 2022; Diao et al., 2023; Wang et al., 2023c) highlight that the order of these examples in the prompt has a substantial influence. CoT is prompting strategy decomposing complex tasks into sub-tasks, helping the model derive the cor- rect answers progressively (Wei et al., 2022; Zhou et al., 2023). It has been widely used mathemati- cal word problem solving, common sense reason- ing, and symbolic reasoning. Our work incorpo- rates CoT strategy into KBQG tasks, where itera- tive process enables LLMs to ultimately obtain a complex question aligning with the logical form.
Knowledge Base Question Generation. The early approaches for KBQG tasks are template- based methods. Berant et al. (2013 and Talmor and Be- rant (2018a) utilized search engines and manual an- notation to construct the natural language questions based on logical forms. However, template-bant based methods rely on manual intervention, which is hard to be scaled up. With the advancement of deep neu- ral networks, neural network-based methods have emerged as prominent and widely adopted ap- proach. Kumar et al. (2019) and Chen et al. (2023) proposed end-to-end models based on Transformer and Graph2seq models, which are capable of gen- erating complex, multi-hop questions based on subgraph. Follow-up studies (Fei et al., 2022; Guo et al., 2022) developed more complicated models for KBQG, which ensure the relevance between the generated questions and subgraphs. Xiong et al. (2022) proposed method for low-resource KBQG, where an auto-pron auto--rromper is developed to paraphrase a logical form into a description, so that pre-trained language model can be fine-tuned with the augmented data. Our work is different from this one as our method focuses solving few-shot KBQG challenge with frozen LLMs. Few-shot Learning for Text Generation. In re- cent years, significant progress has been made in the field of few-shot learning for text generation. One line of work develops meta-learning frame- works for text generation (Mi et al., 2019; Madotto et al., 2019; Zeng al., 2021; Hospedales al., 2022), which aims acquire an optimal initial- ization that enables accurate and rapid adaptation to new task, even when limited data available. Other line of work proposes different augmentation algorithms to synthesize the data for training (Song et al., 2019; Zhao et al., 2022), so that conven- tional text generation models can be applied to the augmented data. Most recently, LLMs are lever- aged to solve few-shot text generation tasks such as text summarization (Yang al., 2023: Zhang et al., 2023b; Liu et al., 2023), machine translation (Wang et al., 2023b; Hendy et al., 2023), dialogue gener- ation (Zhang et al., 2023a; Valvoda et al., 2022: Kang et al., 2022) and so on. There is no existing study applying LLMs to few-shot KBQG tasks. In-Context Learning with LLMs. Without gradi- ent updates, In-Context Learning (ICL.) effectively tackles a wide range of NLP tasks by incorporating a small number of prompted examples as part of
Methodology
3.1 Problem Formulation
A KB consists of a set of triples. A logical form is structural expression of a subgraph in the KB, which may consist of complex operations (e.g.. ag- gregation, comparative and superlative) and can be utilized execute against KB. The task of KBQG requires a system to generate a natural lan- guage question when given a logical form and the corresponding KBs with consistent semantics.
3.2 Method Overview
Recently, the LLM has shown its impressive in- context few-shot learning capabilities. Instead of fine-tuning pre-trained model to adapt it to a downstream task, we can simply apply it to a new task with few examples as prompt during infer- ence (Yang et al., 2022; Li et al., 2023). For the KBQG task, we adopt two-stage method to de- sign CoT prompts, which effectively enable the LLM to comprehend complex logical forms and generate questions. Concretely, the first stage Sup- portive Logical Forms Selection focuses on iden- tifying supportive examples that represent various syntax patterns of logical forms. To accomplish this, we encode the structure of logical forms, per- form clustering, and employ sampling techniques to select top-k supportive logical forms. Once these supportive examples are selected, leverage LLMs with CoT prompts to generate natural lan- guage questions. This leads us to the second stage, Prompt Construction, which involves producing sub-questions as rationales. Through this process,

---

we can ultimately formulate complex question that adequately captures the semantic of the logi- cal form. A schematic diagram of our method displayed in Figure 2.
the K-means (Hartigan and Wong, 1979) clustering algorithm to group the encoded structure into & clusters based on their syntactic similarity.
Logical Form Sampling. Each cluster contains a group of logical forms with the similar structure, we randomly pick up structure from each group and obtain l representative structures. As each structure may correspond to multiple logical forms. We further identify & logical forms with distinct mantics deriving from the ls selected structures. To this end, we iteratively sample logical forms hold- ing the maximum diversity of semantics. Specifi- cally, for the first logical form, we randomly pick up one from the candidates. Then we search logical forms for another structure. We greedily pick up candidate with least semantic similarity to the selected logical forms, where the similarity is mea- sured by the encoding of the original logical forms. We repeat the process until we have gone through structures as shown in Figure 2.
3.3 Supportive Logical Forms Selection
Zhang et al. (2023c) has shown that when con- structing demonstrations, we need to mitigate the effect of few-shot CoT errors by differentiating the design of demonstrations, In KBQG tasks, support- ive logical forms are those that can cover diverse logical rules, so as to offer more syntax informa- tion for LLMs to generate questions. Unlike the narrative inputs, the logical form a combination of program structures and schema items (i.c., en- tities and relations). Therefore, it is essential to take both aspects into consideration when select- ing supportive logical forms. In approach, utilize Structure Encoding and Clustering, fol- lowed by Logical Form Sampling process to select supportive logical forms.
To help the LLMs fully understand the logical forms, we substitute the entities in the original log- ical forms with their surface names in the KB. In this way, we obtain & supportive logical forms.
Structure Encoding and Clustering. To ensure the logical forms can be drafted for unseen ques- tions, we extract their structures by converting the schema items into symbolic variables. Specifi- cally, keep the grammars in the logical form unchangeable. Then, we replace the relation with symbol "y" and we replace the entity with "e". This structure also known as a abstract query graph (Chen et al., 2021), which reflects the topol- ogy and the component classes logical forms. For instance, the raw logical form is:
3.4 Prompt Construction
Since some logical forms have complicated mantics and even nested syntactic structures included. Following the CoT method, construct reasoning chain prompt based on the supportive logical forms retrieved above. For each example, we need to generate a reasoning chain based on log- ical forms to elicit LLMs generate questions from simple to complicated. To this end, we hold two criteria when constructing reasoning chains:
(AND medicine. medicine.routeddddmmm
(JOIN cenddiiineeroute.. drug. wtt.ddrrugg..eeeeeeee forrketed_foruulattti mx))
(i) The templates should break up the generation of complicated question into step by step process.
It becomes the following structure after conversion:
(AND r (JOIN r e)).
(ii) The templates should clearly identify the sub- component in logical form that requires LLMs to focus on for each step.
Once we have obtained the structure of the log- ical forms, which filters out the semantic mean- ing of the logical forms. We encode the structure representation into a fix-length embedding. In de- tail, we view the structure as sequence of to- kens. We encode the contexts of the sequence with Sentence-Trane Transformers (Reimers and Gurevych, 2019), which an advanced model for text em- bedding. The encoded vectors are well-suited for calculating the similarity between sentences. We extract the final hidden state of as the vectorized representation the sentence. After that, we utilize
Therefore, we first break down a logical form in nested manner, where the follow-up logi- cal forms include the preceding logical forms. Specifically, the first step usually generates a simple question querying one-hop relation from the topic entity. The second step usually gener- ates a question querying two-hop relation chain involving the above one-hop relation. As we can see from Figure 2, the first step of prompt

---

Supportive Logical Forms Selection
Premmp: Construction
Logical Forms
Demonstrations
medicine routed drug (JIN medicins.rvuted drug eted formulations m=
Irpet (AND rppror (JOIN sports.gpot..eiim coaches John Russo))
Subgpmphi1__ _OOIN your ponrssppen spont foom aouche John Russo)
Subguph2: (AND sponts.port "Subguphl)
(AND music produzer (JOIN ausic producer.ttcck.. produced OIN music reconding producer m.01 m.0l (BWw6)))
Subquestion sport seam cnach john nusso
Subquestion2 Which sport does john russo coach?
Input (JOIN (R pwwsseeppeeeeeeece _f
pe ___eeeo___hh_ birth) (JOIN music producer.elcuses
Simcueess
produced Poguctry In Metion))
(AND (JOIN
Subgraph UOIN music producer coleases produced Poguetry in Motion))
Subgraph2: (JOIN people persor place birth) Suhgraph1 )
(JII ON
Subquestion music producer of re produce poguetry motion
Subquestion2: Where the birth place of the music producer of poguetry in motin?
Clusters
Inpot: (COUNT aviation.aineraft manufacturer (IOIN organization.
organ.zttion legal structure S.A.))
Supportive Logical Forms
Preduction
(AND sport (JOIN sperts spori.lesm canches
Subgraph :0OIN onganization organization logal structure S.A.)
John Russo))
Subgruph2: (AAD aviation aircraft manufacturet Subgraph1
(COLNT (AND masicc.lhum (JOIN music alhum. festured
Subggaphi3 (COUNT NT Subgraphi2) )
aetists Sierra Leone)))
Subquessionn organizstion legal structure of
Subquestion2 aircraft manufacturer the legall structure of s.a.
(JOIN (R people person.place of birthi (JOIN music. producer, releases produced Poguetry In Mation))
Subspessioon What the number of arrerat meadiettrer legal
af'
Figure 2: KQG-CoT framework. The supportive logical forms are selected from unlabeled data pool by extracting the structures, clustering the structures and sampling the most representative ones. A total of demonstrations are automatically constructed using reasoning chains. The tested logical form appended to the demonstrations form the complete prompt, which can elicit the LLM to generate series of subquestions sequentially from simple to complicated. Finally, the last subquestion can be extracted as the final prediction.
parses the entire logical form into one-hop relation subgraphl "(AND sports. sports.sport. team_ John Russo)" which leads to a simple subques- tionl "sport team coach russo ". The second step includes the parsed logical form appended to the previous step as a component and gener- ates question "Which sport johm russo coach?" ?" based on the subgraph2 and subquestionl. As a result, we continuously expand the logical form until complete question is formed. This step-by- step process ensures that the generated question is semantically coherent and grammatically accurate.
Dataset
#Q
#R
#E
#T
WQ
22,989
672
25,703
2/99/5.8
PQ
9,731
378
7,250
2/3/2.7
GQ
64,331
3,720
32,585
1/4/1.4
Table 1:
Statistics of the evaluated datasets. #Q de- notes the number of questions. #R and #E denote the total number of relations and entities, respectively. #T denotes the mininum/maximuu//vrrrga number of triplets involved in each question.
4 Experiment
In this section, we first introduce the KBQG datasets used to evaluate the performance of our proposed method and the comparable baseline methods. Next, we present the implementation details and demonstrate the experimental results.
During inference, we concatenate all the demon- strations and queried logical form as the final prompt. Based on the example in Figure 2, the prompt includes "Input: (AND... Input: (JOIN Input: (COUNT... ... S.A.". After receiving the prompt, LLMs outcome the predictions that clari- fies the intermediate generation steps of subques- tionl, subquestion2, and subquestion3. And the last subquestion will be our final predicted ques- tion, which is "What is the number of aircraft man- ufacturer in the legal structure of s.a. ?".
4.1 Data and Metrics
We evaluate our prompting method on the follow- ing three public datasets:
WebQuestions (WQ) (Kumar al., 2019)1 is KBQG dataset combining instances from We- QuesstionSSP (Serban et al., 2016) and Com-
httpp:///itthu.... homge///////yyaaaaa//nnggggg

---

WQ
PQ
GQ
Method
M
R
B
M
Standard Prompt
24.86
29.01
52.74
55.87
42.24
76.83
29.17
33.52
42.95
Random-Col
25.02
29.37
53.16
56.42
42.61
77.03
29.81
33.75
43.31
Manual-CoT
28.44
30.24
54.30
60.37
42.88
77.48
30.18
33.61
44.89
Active-CoT
26.02
29.55
54.01
58.78
43.86
76.78
30.27
33.71
44.07
Auto-CoT
28.42
29.65
53.47
59.59
43.16
77.13
30.17
34.22
44.47
KQG-CoT (Ours)
28.89
30.41
54.38
60.81
43.54
77.35
30.51
34.26
44.91
KQG-CoT+ (Ours)
29.73
31.08
55.14
61.71
44.27
78.41
31.24
34.94
45,36
Table Few-shot evaluation of existing prompting methods with Frozen LLMs on three KBQG datasets. The best and second best results are boldfaced and underlined respectively.
plex WebQuestionss (Talmor and Berant, 2018b). provides questions, answers, and annotated sub- graphs. This dataset commonly evaluated in existing work (Guo et al.. 2022).
by sorting the demonstrations from short long after sampling. We denote this method as KQG- CoT+.
As there is no existing attempt for few-shot KBQG tasks with LLMs, adopt five general prompting methods under few-shot scenarios our baselines.
PathQuestions (PQ) (Zhou al., 2018)² is monly used KBQG dataset constructed from KBQA dataset. contains questions inquiring chain of relations, wherein the path between the topic entities and answer entities 2-hop or 3-hop. GrailQA (GQ) (Gu et al., 2021)² is large-scale KBQA dataset built on Freebase, which covers 86 domains. It covers complex questions which require counting, ranking and even superlative inquiry. Each question is associated with s- expression, which can be viewed as a logic form.
Standard Prompt (Brown et al., 2020) is a stan- dard prompting method of in-context learning, where & random logical forms and questions are concatenated to form the prompt. The prediction is one-step generation.
Random-Coo is an intuitive CoT prompting base- line where logical forms are randomly selected from the data pool and we follow the original work (Brown et al.. 2020) to describe the sub-task in narrative.
We collect the annotated the logic form from the training set as the data pool and leave the orig- inal questions untouched. The questions in the validation or test set are sampled evaluate our method. Statistics of evaluated datasets are shown in Table 1.
Manual-CoT (Wei al., 2022) is CoT prompting with & human-writted exemplars as demonstrations and the sub-task is presented in narratives.
Active-CoT (Diao et al., 2023) is an ensemble framework for CoT prompting. The multiple logi- cal forms randomly selected as validation set. Then multiple measurements (e.g., disagreement, variance) are leveraged as the uncertainty value for each logical form to produce the final question. Auto-CoT (Zhang et al., 2023c) automatically con- structs prompt by selecting & demonstrations with cluster-based algorithm and the sub-task is pre- sented in narratives. We simply adopt the prompt- ing method KBQG tasks by encoding all logical form in a textual way.
Following previous KBQG studies, we rely on a set of well-established metrics as for KBQG evaluation:
BLEU-4 (Papineni et al., 2002), ME- TEOR (Banerjee and Lavie, 2005) and ROUGE- L (Lin, 2004). BLEU-4 and ROUGE-L can be viewed as precision and recall for text generation tasks, respectively. METEOR is a comprehensive metric beyond exact matches, which also accounts for partial matches and variations in word order. We denote them as B, M and R, respectively.
4.2 Comparable Methods
4.3 Implementation Details
We denote our prompting method KQG-CoT. Previous studies (Lu et al., 2022b) have proven that the order of the exemplars significant to the prompt results, we implement an improved version
For encoding logical forms, we utilize all- MiniLM-L66-v2 checkpoint from the Sentence- Transformers library in Huggingface for effective
t/gitt::/github.... com/ http////kkkeRRNNN Phttps: http///kiiaaa lab. github. Aeathub.i//Griiiii io/GrailQM/
'https://huggingfacee co/ co/sentence- weaeeanceetraassrrrroro transformers/ all-MiniLN-L6-v2

---

encoding. As this is few-shot scenario, we manu- ally write the rationales for the demonstrations in the chain prompt. We utilize text-davinci-O0 from OpenAI API to generate questions and set the number of clusters as k = 12
consistently outperforms other method across all KBQG datasets by a remarkable margin. Specif- ically, KQG-CoT+ improves the performance of the competitive Auto-CoT by 0.72 to 2.12 absolute values for all datasets. Meanwhile, KQG-CoT also outperforms existing CoT prompting methods on BLEU-4 of all the datasets.
4.4
Main Results
2) Comparing CoT methods with standard prompting, we notice that all the CoT prompt- ing methods outperform the standard prompting method, which indicates that, to generate questions with complex logic and long dependency, splitting the entire generation task into sub-tasks are crucial for maintaining the coherence and accuracy of the
WQ
Method
B
M
R
Fill Training
L2A (Du et al., 2017)
6.01
26.95
25.24
Transforner (Vaswani et al., 2017)
8.94
13.79
32.63
MHQG (Kumar et al., 2019)
11.57
29.69
35.53
BiGraphh2Sg (Chen
2023)
29.45
55.45
T5-Large (Raffel et al., 2020)
28.78
30.55
55.12
JoinGT (Ke et al., 2021)
30.02
32,05
55.60
IGND (Fei et al., 2021)
30.62
31.41
55.82
questions.
LFKQG al., 2022)
31.66
32.69
3) Comparing Auto-CoT, KQG-CoT and KQG- CoT+, even though all these methods adapt clus- tering to select le demonstrations, KQG-CoT and KQG-CoT+ are more effective we elaborately design encoding algorithm and prompt templates for KBQG tasks, which makes fit more into the question generation from the logical forms.
DSM (Guo al. 2022)
28.62
64.25
Few-shol Evaluntion
KQG-CoT KQG-CoT+
28.89
30.41
54.87
29.73
31.08
55.46
Table 3: Comparison between few-shot evaluation of KQG-CoT/KQG-CoTT and full-trained evaluation of other systems on WQ.
Comparison with Other Systems. We further compare our prompting methods with other KBQG systems on the WQ and PQ datasets. According to our knowledge, we are the first work on the KBQG task using the GQ dataset, so there are no existing methods available for comparison.
PQ
Method
B
M
R
Full Training
L2A (Du et al. 2017)
17.00
50.38
19.72
Transfornrr (Vaswani et al., 2017)
56.43
43.45
73.64
MHQG (Kumar et al,, 2019)
25.99
33.16
58.94
In Table 3, we can see that with 12 demonstra- tions, our method can outperform majority of full- trained systems WQ dataset, where all training
BiGruph2Seq (Chen et al., 2023)
61.48
44.57
77.72
AutoQGS (Xiong
2022)
65.13
47.50
76.80
TS-Large (Raffel et al., 2020)
58.95
44.72
76.58
IGND (Fei et al., 2021)
61.69
45.11
77.28
LFKQG (Fei et al., 2022)
63.92
46.91
78.40
data leveraged train model. KQG-CoT+ prompting method can achieve 29.73%, 31.08%
JointT (Ke al., 2021)
48.25
78.87
DSM (Guo et al., 2022)
61.03
86.06
and 55.46% for BLEU-4, ROUGE-L and ME- TEOR respectively, which are close to the SoTA results.
Few-shor Evaluation
BiGraph2Seq (Chen al., 2023)
12.07
JoinGT (Ke al., 2021)
43.15
35.91
69.57
AutQGSS (Xiong et a 2022)
43.46
33.55
68.23
KQG-Co
60.81
43.54
77.35
In Table 4, can see that for PQ dataset, our method can still achieve better results than most of existing full-trained KBQG models. Compared with existing methods under few-shot settings, our methods can significantly improve the BLEU-4 over AutoQGS by around 20 absolute points. It is worth noting that AutoQGS takes 0.1% training instances for training and we simply leverage 12 in- stances for inference, which highlights superiority of our methods.
KQG-CoT+
61.71
44.27
78.41
Table 4: Comparison between few-shot evaluation of (OG-CC///QGGCCOT and few-shot/full-trine evalu- ation of other systems on PQ.
Comparison with Baselines. Table 2 showcases the experimental results of our methods and base- line approaches. We have the following observa- tions based on it:
1) Comparing all CoT prompting methods, in the few-shot setting, our KQG-CoT+ prompting
4.5 More Analysis
Human Evaluation. We further conduct human evaluation by randomly sampling 300 examples from the test set of WQ dataset. The generated
http:/ppeaaa.ccmmoooo///eeeeee codex/ 'Detailed prompt design of KQG-CoT+ is presented in Appendix A.3.

---

Input: (AND military.militarry www.imarrlcoooiiiii conffict (JOIN catirry.miiitrycoooo catpaa__ooonicfffooesstetreeeeee (JOIN
(R military. military aesource.onfflett Bendix AN/FPS-20))
Manual-CoT: Which military conflict involves the bendix an/fps-20 and what are its force strengths?
Active-CoT: What military conffict has force strengths using bendix an/fps-20?
Auto-CoT: What are the force strengths in the bendix an/fps-20 military conflict?
KQG-COT+: Which military conflict has force strengths with bendix an/fps-20?
Ground Truth: Which militury conflict has force strengths with conflicts hendix an/fps-20?
Inputx(AND measurement,
mewsuremmet system (JOIN measurement mea..measrremet system.
heaa_ccapaa capacity units Joule per kelvin))
heat
Manual-CoT: What is the measurement system that uses joules per kelvin heat capacity units?
Active-CoT:: What the measurement system for heat capacity units of joule per kelvin?
Auto-CoT: Which measurement system uses joule per kelvin its heat capacity unit?
KQG-COT+: What measurement system uses joule per kelvin a units to measure heat capacity?
Ground Truth: What system uses joule per kelvin as the unit to measure heat capacity?
Table 5:
Illustrative examples from KQG-CoT+ and baseline methods on GQ.
Model
Synt.
Comp.
Relev.
that CoT plays an important role in generating com- plicated questions. Then we remove the K-means algorithm and randomly select supportive logical forms. The decrease of the results indicates that our clustering algorithm could provide more diverse logical forms as our demonstrations. We further encode the entire logical forms without extracting their structures. The results decrease which indi- cate that the structure is significant indicator obtain the clusters?.
Ground Truth
4.88
4.92
4.91
Standard Prompt
3.67
3.76
3.99
Random-CoT
4.05
4.21
4.12
Manual-CoT
4.60
4.54
4.72
CoT
4.56
4.71
4.75
Auto-CoT 4.38
4.77
4.55
KQG-CoT+
4.63
4.80
4.78
Table 6: Results of human evaluations WQ. Synt., Comp. and Relev. denote syntactic correctness, com- plexity and relevance, respectively.
RaRRand CoT:BLEU-4 - KQG-COT:BLEBE
.. --RRaddo--oo COT:ROUGE-L- KQG-COT:ROUGES
Method
GQ
R
40.33
42.51
43.82
44.25
KQG-CoT+
31.24
34.94
45.36
38.58
41.79
42.27
42:54
38.86
(a) w/o CoT
30.11 1
33.58
43.88
37.41
29.33
30.24
30.65
(b) K-means + Random
29.81
33.75
43.31
26.81
27.26
(c) w/o structure encoding
30.03
33.41
43.76
25.12
25.84
27.85
28.24
28.88
Table Ablation study our KQG-CoT+ method on GQ.
Kw1
KM4
Ka8
Kx12
Figure 3: The BLEU-4 and ROUGE-L scores of our method and Random-CoT with increasing number of shots on GQ.
questions are rated on scale of considering the aspects of syntactic correctness, complexity, and relevance to the given logical forms. We ask three annotators to score the generated questions with --point being poor and 5-point being perfect. The score of each question is averaged over all annotators. We present the results in Table 6, where we can observe similar trend between human and automatic evaluation. Our approach outperforms all comparable methods, the evaluated scores of which are close to the ground truth.
Effect of k. We investigate the effect of in Fig- ure 3. As observed, with an increase of the number of demonstrations, both our methods and Random- CoT show increasing BLEU-4 and ROUGE-L scores. This indicates that the number of demon- strations is significant in activate the potentials of LLMs. Compared with Random-Con our method shows larger gain when the value of k becomes large, this indicates that our methods indeed pick up the most representative logical form as the demon- strations.
A blation Study. We conduct ablation study to as- sess the effectiveness of components of our model and display the results in Table 7. We first exclude the CoT reasoning chain, and observe a perfor- mance drop of the evaluate metrics. This indicates
Case Study. To provide a comprehensive compar-
"The ablation study on WQ and PQ is presented in Ap- pendix A.

---

ison between KQG-CoT+ method and the base- line models on GQ dataset, we present multiple example cases in Table Our method elicits the intermediate generation steps and provides more guidance to LLMs so that our KQG-CoT+ gener- ates questions that are grammatically correct and semantically close to the given logical form. In con- trast, baseline methods may encounter issues such inconsistency in the logical form, misplaced modifiers, or unsmooth expressions.
performance of KBQA methods, we initiated data augmentation procedure for the WebQuestions dataset. It's important to highlight that the aug- mented dataset was merely half the size of the orig- inal dataset. Next, we trained the KBQA method RnG-KBQA (Ye et al., 2022) by combining the augmented and original datasets, resulting in the improved version called RnG- RaG-KBQA+. The results, as outlined in Table 10, demonstrate that con- ducted relatively straightforward augmentation on a limited dataset subset. Nevertheless, the Fi score of the original KBQA method witnessed a notable increase of 2.8%. This demonstrates that our proposed KBQG method provides significant assistance to downstream KBQA tasks..
Effectiveness of Structured Encoding and Clus- tering. To demonstrate the effectiveness of the proposed Structured Encoding and Clustering in selecting diverse structures, we conducted quan- titative assessment of the average semantic simi- larity between the logical forms extracted using our method and the baseline method K=8 on the GrailQA dataset. The results are presented in Table 8. The data from the initial segment, shown in the table below, reveals that the logical forms chosen by our method exhibit lower average semantic similarity. When viewed collectively, these find- ings offer strong evidence for the efficacy of our proposed upproach.
Method
FI-Score
KBGAKBQA 75.6
RnG KBQAK
78.4
Table 10: The result of our approach in improving the performance of KBQA methods.
5 Conclusion
In this paper, we presented the KQG-CoT approach to tackle few-shot KBQG tasks. KQG-CoT re- trieves relevant logical forms from unlabeled data and incorporates their characteristics. It then gener- ates explicit prompt showcase the reasoning pro- cess for complex question generation based on the selected examples. Experimental results demon- strate that our approach achieves state-of-the-re performance compared to baselines and even shows competitive results to full- full-training methods.
Method
Average similarity
Random 0.285
Active-CoT 0,274
Auto-CoT 0.265
KQG-CoT
0.252
Table 8: The average semantic similarity between the logical forms of different methods.
Impact of Sorted Order. To assess the impact of the sorted order of demonstrations in KQG- CoT+, we compared the performance of Auto-CoT and Active-CoT using the same sorted order of demonstrations in KQG-CoT+ (i.c., Auto-CoT+ and Active-CoT+) and conducted experiments on the GrailQA dataset. The Table shows that, com- pared the Active-CoT+ and Auto-CoT+ methods, our proposed KQG-CoT+ method still exhibits sig- nificant improvements.
Limitations
Our proposed prompting method, KQG-CoT, par- tially relies on handcrafted prompts when writing the subquestions. However, handcrafted prompts are usually based on the personal knowledge and experience of the exports, which can introduce sub- jective biases.
Method
R
A Acknowledgements
Active-CoT+
30.40
34.04
44.22
Auto- CoT+
30.52
34.59
44.77
This work was supported by Natural Science Foun- dation of China (Project No. 62206097) and Shanghai Pujiang Talent Program (Project No. 22PJ 20330403,0)). We sincerely thank the anonymous reviewers for their valuable comments and feed- back.
KQG-CoT+
31.24
34.94
45.36
Table The result data for Auto-CoT+, Active-CoT+, and KQG-CoT+ on the GrailQA dataset.
KQG-CoT Improve KBQA Task. To confirm the efficacy our approach in enhancing the
Further analysis will he presented in Appendix A.4.

---

References
methods natural language processing, pages 2573- 2582.
Berkeley R Andrus, Yeganch Nasiri, Shilong Cui, Ben- jamin Cullen, and Nancy Fulda. 2022. Enhanced story comprehension for large language models through dynamic document-based knowledge graphs. Pruceedings of the AAAI Conference on Artificial Intelligence, 36:10436-1044
Zichu Fei, Xin Zhou, Tao Gui, Qi Zhang, and Xuan- jing Huang. 2022. LFKQG: controlled generation framework with local fine-tuning for question gener- ation over knowledge bases. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 6575-6585, International Committee on Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop trinsic Evaluation Measures for Machine Transla- tion and/or Summarication, pages 65-72, Ann Arbor, Michigan, Association for Computational I Linguis- tics.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. arXiv pr arXiv:2211.10433 10435.
Intrinsic and Ex-
Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond i.i.d.: Three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021, page 3477-3488.
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. 2013. Semantic parsing on Freebase from question question-answer answer pairs. In Proceedings of the 2013 Conference on Empirical Methods Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
Shasha Guo, Jing Zhang, Yanling Wang. Qianyi Zhang, Cuiping Li, and Hong Chen. 2022. DSM: Question generation over knowledge base via modeling diverse subgraphs with meta-learne.. In Proceedings of the 2022 Conference on Empirical Methods Natural Longuage Processing, pages 4194-4207. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
John Hartigan and Manchek A Wong. 1979. Algo- rithm as 136: A k-means clustering algorithm. Jour- nal of the royal statistical society, series e (applied statistics), 2811::000-108.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? comprehensive evaluation. arXiv pr 002
Yongrui Chen, Huiying Li, Yuncheng Hua, and Guilin Qi. 2021. Formal query building with query struc- ture prediction for complex question answering over knowledge base. arXiv pr arXiv:2109.0361.
Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. 2022. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis Machine In- telligence, 44(09):5149-5169,
Yu Chen, Lingfei Wu, and Mohammed Zaki. 2023. Toward subgraph-guided knowledge graph question generation with graph neural networks. IEEE trans- actions on neural networks and learning systems, PP.
Minki Kang, Jin Myung Kwak, Jinheon Back, and Sung Ju Hwang. 2022. Knowledge-consisten di- alogue generation with knowledge graphs. ICML. 2022 Workshop on Knowledge Retrieval and Lan- guage Models.
Shizhe Diao, Pengeheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models.
7elnnk Kasner, Toannis Konstas, and Ondrej Dušek. 2022. Mind the labels: Describing relations in knowledge graphs with pretrained models. arXiv pr urXiv2220000737..
Xinya Du, Junru Shao, and Claire Cardie. 2017. Learn- ing to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1352-1352, Vancouver, Canada. Association for Computational Linguistics.
Pei Kc, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin- feng Song. Xiaoyan Zhu, and Minlic Huang. 2021. JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Find- ings of the Association for Computational Linguis- ties: ACL-IICNLP 2021, pages 2526-2538, Online. Association for Computational Linguistics.
Zichu Fei, Qi Zhang, and Yaqian Zhou. 2021. Itera- tive gan-based decoder for question generation. In Proceedings of the 2027 conference on empirical

---

Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrish- nan, Guilin Qi, Lianli Gao, and Yuan-Fang Li. 2019. Difficulty-cnttrollae multi-hop question generation from knowledge graphs. In The Semantic Web-ISWC 2019: 18th International Semantic Web Conference, pages 382-398. Springer.
Fei Mi, Minlie Huang, Jiyong Zhang, and Boi Faltings. 2019. Meta-learning for low-resource natural lan- guage generation in task-oriented dialogue systems. In Proceedings of the Twenty-Eightt International Joint Conference on Artificial Intelligence, IJCAI-19, pages 3151-3157. International Joint Conferences on Artificial Intelligence Organization.
Che-Hao Lee, Tzu- Yu Chen, Liang-Pu Chen, Ping-Che Yang, and Richard Tzong- Han Tsai. 2018. Auto- matic question generation from children's 's stories for companion chatbot. In 2018 IEEE International Con- ference on Information Reuse and Integration (IRI). pages 491 191-494.
Fedor Moiseev, Zhe Dong. Enrique Alfonseca, and Mar- tin Jaggi. 2022. SKILL: Structured knowledge infu- sion for large language models. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1581-1588. Seattle, United States. Association for Computational Linguistics.
Tianle Li, Xueguang Ma, Alex Zhuang. Yu Gu, Yu Su, and Wenhu Chen. 2023, Few-shot in-context learning for knowledge base question answering. In The 61st Annual Meeting of the Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Compu- tational Linguistics, pages 311-318. Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing DOMNLP-IJCNLL pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023. Gpteval: Nlg evaluation using gpt-4 with better human align- ment. arXiv pr arXiv:2303.16634.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2671, 1, Seattle, United States. Association for Computational Linguistics.
Yao Lu, Max Bartolo, Alastair Moore. Sebastian Riedel, and Pontus Stenetorp. 2022a. Fantastically ordered prompts and where to find them: Overcoming few- Proceedings of the
shot prompt order sensitivity. 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086 8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Laura Eline Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023, Large language models are not zero-shot com- municators.
Yao Lu, Max Bartolo, Alastair Moore, Schastian Riedel, and Pontus Stenetorp. 2022b. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers). pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Iulian Vlad Serban, Alberto Garcia-Durán, Caglar Gulcehre, Sungjin Ahn. Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating fac- toid questions with recurrent neural networks: The 30M factoid question-ansee answer corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
Andrea Madotto. Zhaojiang Lin, Chien- Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 5454-5459, Florence, Italy. Asso- ciation for Computational Linguistics.
Berlin, Germany. Association for Computational Linguistics.
Yiping Song. Zequn Liu, Wei Bi, Rui Yan, and Ming Zhang. 2019. Learning to customize model struc-

---

tures for few-shot dialogue generation tasks. An- nual Meeting of the Association for Computational Linguistics.
chain of thought reasoning in language models. The Elevent International Conference on Learning Representations.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin. Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. 2022. Selec- tive annotation makes language models better few- shot learners. arXiv pr 2xXv:2209.01975.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fci Xia, Ed H. Chi. Quoe Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Se- bastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv: arXiv:2210.09261..
Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. Empirical study of zero-shot ner with chatgpt.
Guanming Xiong, Junwei Bao, Wen Zhao, Youzheng Wu, and Xiaodong He. 2022. Autoggs: Auto-prompt for low-resource knowledge-basee question genera- tion from spargl. In Proceedings of the s ACM International Conference on Information Knowledge Management, page 2250-2259. Association for Com- puting Machinery.
Alon Talmor and Jonathan Berant. 2018a. The web knowledge-base for answering complex ques- tions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long Pupers), pages 641 141-651, New Orleans, Louisiana. Association for Computational Linguistics.
Xianjun Yang. Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. 2023, Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv pr wrXiv:2302.08081.
Alon Talmor and Jonathan Berant. 2018b. The web as a knowledge-bast for answering complex ques- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguisties.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022, An empirical study of gpt-3 for few-shot knowledge- based vqu. In The 36th AAAI Conference on Artificial Intelligence (AAAI).
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022. RNG-KBQA: Genera- tion augmented iterative ranking for knowledge base question answering. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Milena Trajanoska, Riste Stojanov, and Dimitar Tra- janov. 2023. Enhancing knowledge graph construc- tion using large language models. arXiv.
Josef Valvoda, Yimai Fang, and David Vandyke. 2022. Prompting for a conversation: How control a dia- log model? In Proceedings of the Second Workshop on When Creative AI Meets Conversational AI, pages 1-8, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Jiali Zeng, Yongjing Yin, Yang Liu, Yubin Ge. and Jinsong Su. 2021. Domain adaptive meta-learning for dialogue state tracking. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2493- 2501.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Miaa Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems. Curran Associates, Inc.
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. arXiv pr UrXiv:2301.07065
Tianyi Zhang, Faisal Ladhak, Esin Durmus. Percy Liang- Kathleen McKeown, and Tatsunori B. Hashimoto. 2023b. Benchmarking large language models for news summarization.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023a. Plan-and-solve prompting: Improving zero- shot chain-of chain-of-thouuht reasoning by large language models. arXiv pr arXiv:2305.0409)
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023c. Automatic chain of thought prompt- ing in large language models. In The Eleventh Inter- national Conference on Learning Representations.
Longyue Wang, Chenyang Lyu. Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023b. Document-level machine translation with large lan- guage models.
Yingxiu Zhao, Zhiliang Tian, Huaxiu Yao, Yinhe Zheng. Dongkyu Lee, Yiping Song, Jian Sun, and Nevin Zhang. 2022. Improving meta-learning for low- resource text classification and generation via mem- ory imitation. In Proceedings of the 60th Annual
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, 2023c. Self-consistency improves

---

Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 583-595, Dublin, Ireland. Association for Computational Lin- guistics.
ing (Diao et al., 2023), random sorting, and sort- ing based on the number of logical form jumps. The detailed experimental results are presented Table 11. It becomes evident that arranging the demonstrations in ascending order of the number of logical form jumps leads to the most favorable outcomes. This finding highlights the structural complexity of logical forms when organizing the demonstrations.
Denny Zhou, Nathanael Schärli, Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. 2023. Least-to-most prompting enables com- plex reasoning in large language models. In The Eleventh International Conference Learning Rep- resentations.
GQ
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu, 2018. An interpretable reasoning network for multi-relation question answering. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 2010-2022, Santa Fe, New Mexico, USA. As- sociation for Computational Linguistics.
Method
R
KQG-CoT+
31.24
34.94
45.36
(a) Uncertainty
30.36
33,91
45.05
(b) Similarity
31.20
34.63
45.28
(c) Random
30.81
34.26
44,91
(d) -L2s
30.52
33.66
44.83
Appendix
Table II: The results of using different sorting methods for demonstrations on the GQ dataset are as follows: Our KQG-CoT+ method is sorted in ascending order the number of logical form jumps. Random sorting is done randomly. L2S sorting is performed in ascending order length. Uncertainty sorting based on de- scending order of uncertainty values. Lastly, similarity sorting based on descending order similarity values between the logical forms of demonstrations and tests.
A.1
Ablation Study on More Datasets
We display Table 12 show more ablation studies on WQ and PQ datasets. We can also recognize the significance of our CoT reasoning chain, K-means algorithm, and structure encoding.
A.2 Illustrative Examples of KQG-CoT+ Prompt
We present a selection of illustrative examples showcasing proposed prompts and predictions WQ, GQ and PQ in Table 13, Table 14 and Table 15, respectively. As
A.3 Detailed Prompt Design of KQG-CoT+
To enhance the guidance provided to LLM in ques- tion generation, we have included a descriptive sen- tence in the demonstrations, which states: "Let's engage in step-by- exercise of generating questions from logical forms. We have provided several examples, each comprising 'Input' logi- cal form and a corresponding 'Subquestion' that we aim to generate. By deconstructing the input logical form into basic components, we can gen- erate questions iteratively until we get the final question. For each 'Subgraph', we can construct relevant 'Subquestion' phrase to assist in generat- ing the subsequent question in the sequence.".
A.4 Effect of Demonstration Order
During the experiment, made noteworthy ob- servation regarding the impact of demonstration order on the performance of our method. We con- ducted a comprehensive exploration of various sort- ing techniques, including uncertainty-base based sort-

---

WQ
PQ
Method
B
KQG-CoT+
29.73
31,08
55.14
61.71
44.27
78.41
(a) w/o CoT
28.75
30.12
54.24
60.83
44.06
77.88
(b) K-means Random
25.02
29.37
53.16
56.42
42.61
77.03
(c) w/o structure encoding
28.52
29.73
54.28
60.34
43.26
77.59
Table
Ablation study our KQG-CoT+ method on WQ and PQ.
Demonstrations
Input: (JOIN (R location.coumiae country.official ollhccillllanuggeee (JOIN location.countr..loe country.laggungessppllle.eeo. romansh language))
Subgraphl: (JOIN location. country, languages spoken romansh language)
countryllanguages
Subgruph2: (JOIN (R location.couu wocation.cuutty.com www...ooooltt.oliiiiii. language) ( Subgraphl)
Subquestion country languages spoken romansh language
Subquestion2: What the main language spoken the country that romansh language used
Input: (AND (JOIN people.cause, of _eeeth.parent, cause _f death drug) (IOIN (R people.deceased penpleaddcceed_pppoooccueee person.cause_.rm _f death) julia biggs))))
(IOIN film.ator.film (JOIN hin..erfoormane..hhee
Subgraphl: (JOIN pcople.cause. of death cause of death drug)
Suhgraph2: (IOIN film.perforance.. character julia higgs)
Subgraph3: (IOIN film.actor.iimn Subgraph2)
Subgraph4: (JOIN (R people.deceasee deceased__person _ecell__e__soooe cause_of @auue__o_deethh Subgraph3)
Subgraph5: (AND Suhgraph1 Suhgraph4)
Subquestionl: parent
of death drug
Subquestion2: performance character julia biggs
Subquestion3: film actor who performance julia biggs
Subquestiond; cause of death of film actor who performance julia biggs
Subquestion5: Which drugs caused the death of the actor who played julia biggs? ?
Input: (JOIN (R lucation.coumtry.uureee wwwrenccrreeeeeeeee _usdd) (JOIN cttilooo.oomtty..mm unthem (JOIN
government.nationn anthem country.anthem aruba dushi tera)))
Prediction
Input: (JOIN (R him.performance.uu actor) (AND (JOIN film, hain.performancc.hhee chhamacccrrcccceerr..e..ee. simon birch) (JOIN film film starring (JOIN fllm.perfomance..o actor ian michael smith))))
Subgraphl: (JOIN fll.performance character simon birch)
Subgruph2: (JOIN flm.performance actor ian michael smith)
Subgraph3: (JOIN film.fil.starme starring Subgraph2)
Subgraph4: (AND Subgraph1 Subgraph3)
Subgraph5: (IOTN (R film,perfor www..performmnce..oom actor) Subgraph4)
Subquestionl: I: performance character simon birch
Subquestion2: performance actor ian michuel smith
Subquestion3: performance actor ian michael smith star in
Subquestion4: performance character simon birch the film that ian michael smith star in SubquestionS: Who plays simon birch in the movie that ian michael smith acted in
Table 13: Prompt with demonstrations and prediction on WQ, the preceding section displays the prompt, and followed section displays outputs generated by L.LMs.

---

Demonstrations
Input: (ARGMIN basc. www.ecaxaactooogy.ccooo exopllane sstronomy.astromomiin discovery.disoovery date)
Subgraph1: (ARGMIN base, buse. exoplanetoooogy eeapeeeooooy...olaneeeeee exoplanet astromomyy.sttroommiall discovery.discovery.dd Subquestion:: Which exoplanet first to he found
Input:
(AND digicams. higicams.digital camera (AND (It digicams. camera. camera.weight weight 250.0
#float)(OONN (R digicams.camern_veewiiiiie aiaeeeeneieeiiieeeeyeeeiiiiaaaa caaeerassaaaa (JOIN digicams, camera_ Scheema
cameras Sony Alpha 700))))
Subgraph1: (It digicams..igge. jigiiiamsdigiilllcmmeeeaaa camera.weight
Subgraph2: (IOIN digicams, camera_ viewfinder typeetyepp.... digital cameras Sony Alpha 700)
Subgraph3: (JOIN (R digicums camera viewlinder. httpeiiiieerryye.dddeee type.digital
aupediggial_ccmeruus cameras) Subgraph2)
Subgraph4: (AND Subgraph1 Subgraph3)
Subgraph5: (AND digicams.diig digital camera Subgraph4)
Subquestionl: digital cameras that weight less thun 250.0
Subquestion2: viewfinder lype digital cameras sony alpha 700
Subquestion3; digital cameras the same viewfinder type the sony alpha 700
Subquestion4; digital cameras the same viewlinder type the sony uipha 700 and weight less than 250.0
Subquestion5: Are there any digital cameras that use the same viewfinder as the sony alpha 700 that weight less than 250.0?
Input: (AND music.genre (JOIN (R music.genre munsiicgerre.parettggee pareet_geeeaneeeeee (JOIN music.genre geni.. albums confessions tour)))
Prediction
Subgraphl: (JOIN music.gente wwwggeeeerruuuuee albums confessions tour)
Subgraph2: (JOIN (R music, veuueeparent genre) Subgraphl)
Subgraph3: (AND music.gence genre Subgraph2)
Subquestionl: the music genre albums confessions tour
Subquestion2: the albums confessions tour is part
parent genre
Subquestion3:
The alhums confessions tour part of what parent genre musical genre?
Table 14: Prompt and prediction on GQ, the preceding section the prompt, and the blue text following represents the prediction.
Demonstrations
Input: (JOIN (R people wwwlleepppooneeeee gender) (JOIN people.person
person parents) sviatoslav of kiev))
Subgraph1: (JOIN (R people.person. pew.sooppareetts.... sviatoslav ii kiev)
Subgraph2: (JOIN (R weople.person.gee
Subgraphl)
Subquestionl: sviatoslav ii of kiev 's parents
Subquestion2: What is the gender of sviatoslav ii of kiev dad
Input: (JOIN (R htwpl//esssddppssopppeeeeee death) (JOIN (R people peopplepenoom.tiidrren) (JOIN (R (roplee.ensoo..hhiddree))
kennedy)))
Subgraph1: (JOIN (R hateeperson.cchdrreeeee p kennedy)
Subgraph2: (JOIN (R poople.erson.chidrre Subgraph
Subgraph3: (JOIN (R people.dcceased person.place place of death) Subgraph2)
Subquestion I: kennedy 's children
Subquestion2: children of kennedy 's children
Subquestion3; What is the place of death of kid of p.j kennedy 's son
Input: (JOIN (R muusic.recrdinggreleessy (JOIN (R music.recordinggtraae tracks) o holy night))
Prediction
Subgraph1: (JOIN (R muusic.recording.tra... o holy night)
Subgraph2: (JOIN (R muusic.eccoding..eleesss Subgraph1)
Subquestionl: o holy night trucks
Subquestion2: What is the releases of recording of o holy night tracks
Table 15: Prompt and prediction on PQ, the preceding section is the prompt, and the blue text following it represents the prediction.


Reviews:
[{"review_id": "Q7baqslAqx", "reviewer": "Reviewer_PKjA", "summary": "", "questions": "Question A: What are the primary contributions of this paper in comparison to Auto-CoT?\nQuestion B: Is the order of the selected logical forms significant? What is the reasoning behind choosing the last subquestion as the final prediction?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your meticulous and insightful feedback on our manuscript. In response to your feedback, we are providing a general response to the questions you've raised here.\n\n**Question A:** What are the primary contributions of this paper in comparison to Auto-CoT?\\\n**Response :** The primary contributions of this paper in comparison to Auto-CoT lie in the following ways:\n1. While our method and auto-CoT utilize K-means clustering to select supportive examples for constructing demonstrations, we take a different approach regarding the logical forms from input questions. We holistically consider the structural information of logical forms. Illustrated in the Supportive Logical Forms Selection (Section 3.3) of our paper, our method initially extracts structures of logical forms, which omit the semantics of the inputs at this stage. Subsequently, we cluster and sample the K main structures of logical forms. By amalgamating the sampled structures of logical forms with semantic similarity, we employ a greedy approach to select logical forms with the lowest average semantic similarity. As evident from our approach, the manner in which we unravel the semantic and syntactic components for logic forms differs from that of Auto-CoT, yet it holds significant importance for KBQG tasks.\n2. Our utilization of the CoT methodology varies. Unlike Auto-CoT, where the output is generated with single turn, the generation process of our method follows incremental expansion. Progressing from shorter to more extensive structures, structured inputs can be incrementally transitioned, thereby aligning more closely with the CoT concept. In our paper, for input logical forms, we progressively expand them using their keywords. This ensures that each guiding segment of CoT remains accurate a feat not easily achievable with unstructured inputs. Moreover, to our best known, we are the pioneers in employing LLMs and CoT to tackle the KBQG task.\n3. Of utmost significance, prior research [1], [2] underscores the crucial importance of the order of demonstrations. This aspect is overlooked by auto-CoT, whereas our method arranges demonstrations based on the complexity of corresponding logical forms. Our results indicate superiority over random ordering. We order demonstrations based on the complexity of the logical form. The experimental results in Table 2 validate the feasibility of this approach for the given task. Therefore, we believe that this exploration holds great significance in KBQG tasks or other tasks involving structured inputs.\nIn summary, although our approach shares some similarities with auto-CoT, our primary emphasis lies in the novel contributions our method brings to addressing the KBQG task.\n\n\n**Question B:** Is the order of the selected logical forms significant? What is the reasoning behind choosing the last subquestion as the final prediction?\\\n**Response:** Just as we mentioned in our paper, the logical forms we have selected will subsequently form the demonstrations in the prompt. Both [1] and [2] emphasize the importance of the order of demonstrations. Moreover, KBQG tasks aim at generating questions with complex expression and nested logics. Hence, a progressive generation order is important, which has been proven by comparing the KQG-CoT and KQG-CoT+ methods in Table 2 of our paper. So the order of the selected logical forms is very significant. \nRegarding the decision to choose the last subquestion as the final prediction, this rationale becomes evident through both the Prompt Construction (Section 3.4) outlined in the paper and Figure 2. Each substructure of the logical form corresponds to a distinct phrase, and as we iteratively elaborate on these substructures, the corresponding phrases naturally grow in comprehensiveness. As we approach the culmination of the logical form in its entirety, the corresponding statement reaches its fullness, ultimately aligning with the answer we aim to achieve.\n\n[1] Lu et al.,  Fantastically Ordered Prompts and Where to Find Them:Overcoming Few-Shot Prompt Order Sensitivity.\\\n[2] Dong et al.,  A Survey for In-context Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contribution of this work is that it leverages Language Models (LLMs) with few-shot learning and employs the Chain of Thought (CoT) approach for Question Generation over Knowledge Bases (KBQG).", "reasons_to_reject": "It is important to clarify the specific contributions of this paper in relation to Auto-CoT (Zhang et al., 2023c).", "questions_to_authors": "Question A: What are the primary contributions of this paper in comparison to Auto-CoT?\nQuestion B: Is the order of the selected logical forms significant? What is the reasoning behind choosing the last subquestion as the final prediction?\n", "paper_topic_and_main_contributions": "The paper utilized LLMs with few-shot learning to tackle the task of Question Generation over Knowledge Bases (KBQG) with the Chain of Thought (CoT) approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Oujlw8R6mV", "reviewer": "Reviewer_BZaK", "summary": "", "questions": "Question 1: could you explain how you embed the input logical form (AND medicine.routed_drug (JOIN medcine.routed_drug.marketed_formulatations m.0hqslx))with its structure info (AND r (JOIN r e))? Do you encode the original logical form or just encode its structure info? And why you do not convert the entity ID m.0hqslx into its entity name? \n\nQuestion 2: If you only encode its structured info, why do you need to cluster them? I think we just need to use string matches to find the most similar structure in the demos. In Figure 2 for a new input, like (COUNT (AND aviation.aircraft_manufacturer (JOIN organization.organization.legal_structure S.A.))), it's better to just select the demo with the same structured info (COUNT (AND r (JOIN r e))), which is the [2] supportive logical forms.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wholeheartedly express our gratitude to you for offering comprehensive and insightful feedback on our manuscript. We are both honored and pleasantly surprised by your high evaluation. However, in the same vein, you have also highlighted the shortcomings of our work in the \"Reasons To Reject\" section and raised two questions. Below, we will provide detailed explanations for both aspects.\n\n**Statement 1:** While the paper addresses the relevant challenge of Question Generation over Knowledge Bases, the methodology, inspired by Chain-of-Thought prompting, is not sufficiently differentiated from existing approaches.\\\n**Response:** As you mentioned in the \"Reasons To Accept\" section, our approach is innovative and adeptly employs CoT prompts to address KBQG challenges in low-resource environments. As showcased in our paper's Methodology (Section 3) and Figure 2, the innovation within our method is primarily evident through the following facets:\n1. As we strive to identify supportive logical forms, we meticulously consider the inherent structures of these forms. This allows us to extract supportive logical forms that exhibit semantic diversity. \n2. In the utilization of the CoT methodology, we astutely leverage the structured nature of logical forms. We progressively expand these forms from simpler, shorter constructs to more intricate and elaborate structures, seamlessly aligning them with the CoT framework.\n3. Importantly, we are pioneering the integration of LLMs and CoT to tackle the KBQG task. Additionally, we arrange demonstrations based on the complexity of the logical forms. The experimental results presented in Table 2 serve to validate the feasibility of this approach for the given task. Consequently, we firmly believe that this exploration bears significant significance not only within KBQG tasks but also in other tasks necessitating structured inputs. \n\nIn summary, our approach not only draws insights from previous methods but also introduces innovation, creating a notable distinction from existing approaches.\n\n**Statement 2:** Additionally, the empirical results, though comprehensive, do not provide a clear justification for the proposed techniques' superiority over established benchmarks.  \\\n**Response:** Prior research [1] has demonstrated that the performance of In-context learning (ICL) (of which CoT is a subset) heavily relies on **Demonstration Selection**, **Order of Demonstration Examples**, and **Reasoning Steps Formatting**. Therefore, the design of our approach takes these three aspects into careful consideration. Furthermore, in the \"More Analysis\" (Section 4.5) section of our paper, we conduct **Ablation Study** to validate the impact of each component. From the results presented in Tables 7, 8, and 9, the contributions of each part to the overall effectiveness of our method are evident. Thus, both theoretically (an aspect that does not require further validation in our work again) and experimentally, there are clear reasons to support the superiority of our proposed method over established benchmarks.\n\n**Statement 3:** How can this KBQG help the downstream tasks, like KBQA is not studied? \\\n**Response:** In the \"Introduction\" section, we mentioned that KBQG can assist QA systems in data augmentation and aid in generating coherent questions for dialogue systems, among other applications. As a result, it has been garnering increasing interest from both industry and academia. Past research[2],[3] has already demonstrated that for solving KBQA and other tasks, effective data augmentation through KBQG can enhance training data and the quality of generated questions has a significant impact on the performance improvement of the task models. \\\n Our proposed method, requiring only the extraction of a few logical forms from the knowledge graph, can generate high-quality questions. This feature positions it as a valuable asset for downstream KBQA tasks. The specific benefits of KBQG for various downstream tasks and the mechanisms behind these benefits lie outside the scope of this study and are part of our future research plans. We intend to utilize the KBQG technique developed in this study for data augmentation in addressing KBQA tasks under few-shot settings, as well as quantifying the impact of increased data volume on KBQA improvement in our future research endeavors.\n\n**Question 1:** could you explain how you embed the input logical form (AND medicine.routed_drug (JOIN medcine.routed_drug.marketed_formulatations m.0hqslx))with its structure info (AND r (JOIN r e))? Do you encode the original logical form or just encode its structure info? And why you do not convert the entity ID m.0hqslx into its entity name? \\\n**Response:** We perform embedding on the structures of logical forms because the extracted structures of logical forms does not include entity IDs, so eliminating the need for conversion to their corresponding entity names. \\\nInitially, we attempted to embed the original logical forms and directly apply k-means clustering. However, we found that this approach yielded suboptimal results. We find that the diversity in logical forms, stemming from their inclusion of keywords, relations, and mid-components, might lead to this outcome. Considering that the structures of logical forms share similarities with the arrangement of sentence parsing syntax trees, we chose to extract and incorporate the principal structural components of these logical forms. So, we perform embedding on the structure of the logical form and then proceed with clustering.\n\n**Question 2:** If you only encode its structured info, why do you need to cluster them? I think we just need to use string matches to find the most similar structure in the demos. In Figure 2 for a new input, like (COUNT (AND aviation.aircraft_manufacturer (JOIN organization.organization.\nlegal_structure S.A.))), it's better to just select the demo with the same structured info (COUNT (AND r (JOIN r e))), which is the [2] supportive logical forms. \\\n**Response:** According to intuition, it might be more effective to select K most similar structures and construct demonstrations from them. However, we chose not to pursue this approach due to several reasons:\n1. Based on findings from the auto-CoT paper, using a method that selects the most similar structures and then combining automated methods for constructing CoT reasoning steps can lead to errors and subpar results.\n2. We are addressing the issue of few-shot KBQG, which involves manually constructing intermediate reasoning steps when forming CoT-style demonstrations. If we were to construct demonstrations anew for each selected logical form, it would be cumbersome, time-consuming, and resource-intensive.\n3. Furthermore, the method you mentioned, which we experimented with under the \"Similarity\" row in Table 8 of Appendix A and he experimental results demonstrate that our proposed method significantly outperforms the approach you mentioned. \n\nHence, our method involves initially using clustering to select representative logical forms, followed by manual construction of intermediate reasoning steps. Regardless of the input, the demonstrations remain consistent. This approach ensures that even if we need to manually create intermediate reasoning steps for K demonstrations, it remains manageable and guarantees the correctness of the reasoning process. In conclusion, both theoretically and experimentally, it has been demonstrated that the find the most similar structure of the input approach is not feasible.\n\n**Regarding your observation on Figure 2, where you pointed out \"In Prompt Construction Subgraph1: (JSON sports.sport.team_coaches John Russo) should be (JOIN),\"** we want to express my heartfelt gratitude for your careful reading. This writing error was a result of my oversight. We will thoroughly review and rectify these errors in the subsequent revisions to ensure the accuracy and clarity of the paper.\n\n[1]Dong et al., A Survey for In-context Learning. \\\n[2]Guo et al., DSM: Question Generation over Knowledge Base via Modeling Diverse Subgraphs with Meta-learner. \\\n[3]Maharana et al., GRADA: Graph Generative Data Augmentation for Commonsense Reasonin", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a novel approach to the timely challenge of Question Generation over Knowledge Bases in low-resource settings, ingeniously leveraging Chain-of-Thought prompting. The methodological innovation, combined with impressive empirical results surpassing established benchmarks on the PathQuestions dataset, highlights both the study's immediate relevance and its potential to significantly impact the domain. The comprehensive experimentation offers robust evidence of the proposed techniques' effectiveness, making this paper a valuable contribution to the field.", "reasons_to_reject": "While the paper addresses the relevant challenge of Question Generation over Knowledge Bases, the methodology, inspired by Chain-of-Thought prompting, is not sufficiently differentiated from existing approaches. Additionally, the empirical results, though comprehensive, do not provide a clear justification for the proposed techniques' superiority over established benchmarks. \nHow can this KBQG help the downstream tasks, like KBQA is not studied? The paper could benefit from deeper analysis and a clearer exposition of the methodological advancements to ensure a more substantial contribution to the field.", "questions_to_authors": "Question 1: could you explain how you embed the input logical form (AND medicine.routed_drug (JOIN medcine.routed_drug.marketed_formulatations m.0hqslx))with its structure info (AND r (JOIN r e))? Do you encode the original logical form or just encode its structure info? And why you do not convert the entity ID m.0hqslx into its entity name? \n\nQuestion 2: If you only encode its structured info, why do you need to cluster them? I think we just need to use string matches to find the most similar structure in the demos. In Figure 2 for a new input, like (COUNT (AND aviation.aircraft_manufacturer (JOIN organization.organization.legal_structure S.A.))), it's better to just select the demo with the same structured info (COUNT (AND r (JOIN r e))), which is the [2] supportive logical forms.", "paper_topic_and_main_contributions": "In the study, the challenge of Question Generation over Knowledge Bases (KBQG) in low-resource scenarios is addressed by introducing a Chain-of-Thought (CoT) inspired prompting technique, reformulating KBQG as a reasoning task. The proposed method, KQG-CoT, selects supportive logical forms from an unlabeled pool and constructs task-specific prompts to guide Large Language Models in generating questions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2. In Prompt Construction Subgraph1: (JSON sports.sport.team_coaches John Russo) should be (JOIN)."}, "extra_text": {}}, {"review_id": "9YKp0cx63R", "reviewer": "Reviewer_jeur", "summary": "", "questions": "A. For Prompt Construction (Section 3.4), the proposed method seems to adopt the bottom-up approach to extract logic form. How about top-down approach? It would be helpful to compare with other compositional semantic extraction approach of previous works [1].\n\nB. Are the any supportive quantitative metrics and/or visualizations to demonstrate that the sampled k structures from Structure Encoding and Clustering are in fact diverse? \n\nC. How is semantic diversity/ similarity measured (Line 310)? \n \n[1] Drozdov et al., Compositional Semantic Parsing with Large Language Models. ICLR 2023.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude to the reviewers for their thorough and insightful feedback on our manuscript. In the 'Reasons To Reject' and 'Questions For The Authors' sections, you raised some inquiries regarding our work. Below, we will provide a detailed response to the questions you have raised.\n\n**Statement 1:** Performance gain of KQG-CoT (presented in Table 2) is marginal over the Auto-CoT, Active-CoT in some metrics, eliciting the questions regarding the absolute necessity of using Logic Form in comparison with narratives in the given task. \\\n**Response:**\n1) KBQG is an extremely challenging generative task that requires models to comprehend the given logical form and generate coherent, semantically consistent questions. Not only does this demand robust natural language processing capabilities from the models, but it also necessitates a profound understanding of diverse domain knowledge and reasoning abilities. Additionally, the complexity of the few-shot KBQG task goes beyond the technical aspects, encompassing the diversity and openness of answers. The expressions of questions corresponding to logical forms from various domains and topics vary significantly, demanding the models to possess cross-domain and cross-style adaptability.\n2) As evident from Table 3 and Table 4 in the paper, achieving substantial improvements with new methods is a formidable challenge. Even some well-known efforts to enhance performance in this task have not yielded significant advancements.\n3) As observed from Table 2, our KQG-CoT approach not only exhibits a value closely approaching that of Active-CoT in one category but also shows evident enhancements in the remaining eight categories compared to Auto-CoT and Active-CoT. Moreover, the performance gain is even more pronounced in KQG-CoT+, with the maximum improvement reaching 2.12%.\n4) In comparison to the textual format of narratives, the decomposed structured logical form is more suitable for CoT. Structured logical forms have the ability to progressively expand from simple structures to more intricate ones, aligning with the CoT's construction of a series of logically connected thinking steps or ideas. These steps or ideas interconnect with each other, forming a complete thought process.\n\nTaking into account the aforementioned points, it becomes apparent that employing Logic Form is crucial relative to the use of narratives.\n\n**Statement 2:** The proposed work is quite incremental. For instance, K-Means clustering algorithms for k-demonstration selection have been proposed by AutoCoT. The importance of demonstrations order for LLM prompting (in the paper's case, it is the difference between KQG-CoT and vs KQG-CoT+) has also been discussed in previous works. \\\n**Response:** We understand the description of \"quite incremental\" to mean that you may perceive the innovation of our approach as relatively modest compared to existing research. However, while we acknowledge that our method might not be entirely revolutionary in terms of innovation, we believe that our approach represents significant improvements and innovations compared to previous work. This includes several specific aspects:\n1) While both our method and AutoCoT employ the K-Means method for selecting k-demonstrations, there is a distinction in our approach. Prior to utilizing the K-Means method, we extract the structure of the logical form. Specifically, we cluster the main structure of the logical form using the K-Means method. After clustering, when identifying corresponding logical forms from these structures of the logical forms (where multiple logical forms might share the same structure), we further filter based on semantic similarity. This ensures that the extracted logical forms exhibit diversity. Hence, even though both methods utilize the K-Means method, their emphasis differs. Our method focuses on the extraction of the structure of the logical form and the selection of diverse logical forms based on semantic similarity. **The K-means method is merely one component of our approach; what holds significance is the innovative work we have undertaken to tailor our approach to the demands of KBQG tasks**.\n2) While previous approaches have discussed the importance of demonstration order for LLM prompting, **there is still no definitive conclusion on the specific optimal order to date**. Published papers, such as those referenced in [2],[3], and [4], have all been exploring viable orders. In our paper, the distinction between the KQG-CoT+ method and the KQG-CoT method lies in our attempt to establish an ordering based on the complexity of the logical form. The experimental results in Table 2 validate the feasibility of this approach for the given task. Therefore, we consider this exploration to be highly meaningful.\n3) As you mentioned, we are among the first to pioneer the use of LLMs to address KBQG task. Through detailed comparative analysis and experimental results, we have demonstrated that our KQG-CoT+ method offers significant improvements over baselines and traditional approaches. Furthermore, we have conducted ablation experiments to validate the specific contributions of each module in our method. **Our endeavor provides valuable guidance on effectively harnessing the capabilities of LLMs in the realm of KBQG**.\n\nIn light of the points discussed above, we believe that characterizing our work as \"quite incremental\" is not suitable.\n\n**Statement 3:** The effectiveness of the proposed Structured Encoding and Clustering in selecting k diverse structures need further quantitative and qualitative evaluation studies. \\\n**Response:** In our paper, we conducted ablation experiments to substantiate the distinct contributions of each module. While you might hold the view that this is not entirely exhaustive and further experimentation is warranted, we promptly executed additional experiments in response. These experiments comprised two main components: firstly, we quantitatively assessed the average semantic similarity between the logical forms extracted using our method and the baseline method at K=8 on the GrailQA dataset; secondly, we employed the visualization tool TSNE to reduce the dimensionality of selected logical form vectors to 2D, thereby generating visual plots. \\\nThe outcomes of the first segment are presented in the table below, indicating that the logical forms chosen by our method display a lower average semantic similarity. The results of the second segment demonstrate that our method's selected logical forms exhibit greater dispersion, which corroborates the findings of the first part. However, due to constraints in displaying images during the rebuttal phase, we intend to incorporate this information in the revised version of the paper. When considered collectively, these results provide robust support for the efficacy of our proposed method.\n| Method | Average similarity | \n| :---         |     :---:      |\n| Random   |0.285     |\n| Active-CoT    | 0.274       | \n| Auto-CoT    | 0.265       | \n| KQG-CoT    | 0.252       | \n\n\n**Question A:** For Prompt Construction (Section 3.4), the proposed method seems to adopt the bottom-up approach to extract logic form. How about top-down approach? It would be helpful to compare with other compositional semantic extraction approach of previous works [1]. \\\n**Response:** Thank you very much for professionally pointing out this new approach to extracting logical forms. It's an important aspect that we overlooked while conducting this work. The reason we chose to adopt a bottom-up approach to extract logical forms is that bottom-up allows for the incremental addition of information, continuously enhancing the complexity of logical forms. This aligns with the CoT concept, which starts with simple questions and gradually addresses more complex ones. In our revised version, we will also explore a top-down method to extract logical forms, thereby enriching the scope of our work.\n\n**Question B:** Are the any supportive quantitative metrics and/or visualizations to demonstrate that the sampled k structures from Structure Encoding and Clustering are in fact diverse? \\\n**Response:** This question, along with the preceding Statement 3, pertains to the same aspect, inquiring whether there is concrete data to substantiate the specific impact of sampling k structures from the Structure Encoding and Clustering method. The response aligns with the answer provided for Statement 3.\n\n**Question C:** How is semantic diversity/ similarity measured (Line 310)? \\\n**Response:** In lines 310 to 318 of our paper, we elucidate how we measure semantic similarity. The specific details are provided below. \n“Specifically, for the first logical form, we randomly pick up one from the candidates. Then we search logical forms for another structure. We greedily pick up a candidate with least semantic similarity to the selected logical forms, where the similarity is measured by the encoding of the original logical forms. We repeat the process until we have gone through k structures as shown in Figure 2.” \n\nRegarding the section titled \"Typos Grammar Style And Presentation Improvements\": \\\n**1.**“Line 17: split Line 272: drafts -> be drafted Line 223: consists -> consist” \\\n**Response:** Thank you very much for your thorough reading and for pointing out the writing errors in our manuscript. These mistakes were due to our oversight, and we apologize for any inconvenience they may have caused to your reading experience. we will carefully review and correct these errors in the upcoming revisions to ensure the paper's accuracy and clarity. \\\n**2.**“Table 5: Additional explanations/ color coding for the examples of different CoT outputs would be better appreciated.” \\\n**Response:** Thank you very much for your suggestion. In our next version, we will differentiate the examples of different CoT outputs in Table 5 by adjusting font size and color, and by adding additional explanations. \\\n**3.**“Table 3,4: Only 1-2 best performing methods for \"Full training\" would be sufficient since the purpose of the table is to report the previous SoTAs under the full training assumption.” \\\n**Response:** In Table 3 and 4, we have provided a detailed presentation of some prominent full-training methods for KBQG developed in recent years. Our intention behind this is twofold: firstly, to demonstrate the level at which the effectiveness of our method resides, and secondly, to offer readers a clear overview of the evolution of KBQG techniques over time.\n\n[1]Drozdov et al., Compositional Semantic Parsing with Large Language Models. ICLR 2023. \\\n[2]Lu et al., Fantastically Ordered Prompts and Where to Find Them:Overcoming Few-Shot Prompt Order Sensitivity. \\\n[3]Liu et al., What Makes Good In-Context Examples for GPT-3? \\\n[4]Diao et al., Active prompting with chain-of-thought for large language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The work is among the first to explore the capability of LLMs in KBQG tasks.\n\n2. Empirical studies are extensive and provide different aspects of the proposed method.", "reasons_to_reject": "1. Performance gain of KQG-CoT (presented in Table 2) is marginal over the Auto-CoT, Active-CoT in some metrics, eliciting the questions regarding the absolute necessity of using Logic Form in comparison with narratives in the given task.  \n\n2. The proposed work is quite incremental. For instance, K-Means clustering algorithms for k-demonstration selection have been proposed by AutoCoT. The importance of demonstrations order for LLM prompting (in the paper's case, it is the difference between KQG-CoT and vs KQG-CoT+) has also been discussed in previous works. \n     \n3. The effectiveness of the proposed Structured Encoding and Clustering in selecting k diverse structures need further quantitative and qualitative evaluation studies.", "questions_to_authors": "A. For Prompt Construction (Section 3.4), the proposed method seems to adopt the bottom-up approach to extract logic form. How about top-down approach? It would be helpful to compare with other compositional semantic extraction approach of previous works [1].\n\nB. Are the any supportive quantitative metrics and/or visualizations to demonstrate that the sampled k structures from Structure Encoding and Clustering are in fact diverse? \n\nC. How is semantic diversity/ similarity measured (Line 310)? \n \n[1] Drozdov et al., Compositional Semantic Parsing with Large Language Models. ICLR 2023.", "paper_topic_and_main_contributions": "The authors propose leveraging Large Language Models (LLMs) and Chain-of-thought (CoT) prompting for the Question Generation over Knowledge Bases (KBQG) tasks. Specifically, the proposed method is broken into 2 steps (1) Supportive Logic Form Selection and (2) Prompt Construction. In the former step, the authors propose encoding the structural knowledge via the logic form. Then, K-Means clustering together with logic form selection are conducted to select the diverse k structures for the sequential prompt construction step. In the next step, motivated by CoT, authors construct reasoning chains from the selected logic form structures as demonstrations for LLM's in-context learning of the KBQG task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Drozdov et al., Compositional Semantic Parsing with Large Language Models. ICLR 2023."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 17: split\nLine 272: drafts -> be drafted\nLine 223: consists -> consist\n\nTable 5: Additional explanations/ color coding for the examples of different CoT outputs would be better appreciated.\n\nTable 3,4: Only 1-2 best performing methods for \"Full training\" would be sufficient since the purpose of the table is to report the previous SoTAs under the full training assumption."}}]

Author rebuttal:

Meta-review: This paper proposes a chain of thought prompting-based approach for question generation over KBs in low-resource settings. The reviewers all believe the task is important, and it has been under-explored with LLMs in prior work. On the negative side, the improvements of the proposed method over competitors (e.g., Auto-CoT) are fairly small, which does call into question the method's generalizability. I would suggest that the authors consider adding a more thorough and well-documented human evaluation to the next version of their paper, as that would help determine its effectiveness more than simple (and flawed) string overlap-based methods. The authors were forthcoming with new results and details during the discussion period, which is much appreciated by both the reviewers and this AC. Overall, there are flaws with this paper, but I believe it will  still make a valuable contribution to EMNLP (either main conference or findings).
Decision: Accept-Main