Skill-Based Few-Shot Selection for In-Context Learning
Authors: Shengnan An
Bo Zhou
Zeqi Lin
Qiang Fu
Bei Chen
Nanning Zheng
Weizhu Chen
Jian-Guang Lou
Venue: EMNLP 2023 Main (2023)
Keywords: In-Context Learning
Few-Shot Selection
Semantic Parsing
TL;DR: We propose Skill-KNN, a skill-based few-shot selection method for in-context learning, to facilitate semantic parsing. Skill-KNN significantly outperforms raw-input-based selection methods on semantic parsing tasks.

Abstract: *In-context learning* is the paradigm that adapts large language models to downstream tasks by providing a few examples.
*Few-shot selection*---selecting appropriate examples for each test instance separately---is important for in-context learning.
In this paper, we propose **Skill-KNN**, a skill-based few-shot selection method for in-context learning.
The key advantages of Skill-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks.
The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself.
Technically, Skill-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant surface features.
Experimental results across five cross-domain semantic parsing datasets and six backbone models show that Skill-KNN significantly outperforms existing methods.

Skill-Based Few-Shot Selection for In-Context Learning
Shengnan An Bo Zhou Zeqi Lin Qiang Fu Bei Chen Nanning Z Weizhu Chen Jian-Guang LOU* National Key Laboratory of Human-Machine Hybrid Augmented Intelligence,
National Engineering Research Center of Visual Information and Applications,
Institute of Artificial Intelligence and Roboties, Xi'an Jiaotong University Microsoft Corporation, Northeasterrn University
(an1006634493e) mnzheng@mail, xjj....j......... edu. cn, 2171 386@stu. neu. edu. cn ([Zeqi qifu, beichen, wzchen, jlou))microoee
com
Abstract
Input Query: Which
employee
age 30 come from?
In-context learning is the paradigm that adapts large language models to downstream tasks by providing few examples. Few-shot selection selecting appropriate examples for each test in- stance separately- is important in-context learning. In this paper, propose SKILL- KNN, skill-based few-shot selection method for in-context learning. The key advantages of SKILI--KNN include: (1) it addresses the prob- lem that existing methods based on pre trained embeddings be easily biased by surface natural language features that are not impor- lant the target task; (2) does not require training or fine-tuning of any models, making it suitable for frequently expanding or chang- ing example banks. The key insight is to opti- mize the inputs fed into the embedding model, rather than tuning the model itself. Techni- cally, SKILL-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant sur- face features. Experimental results across five cross-domain semantic parsing datasets and backbone models show that SKILL- KNN sig- nificantly outperforms existing methods.
Q
Select
Example Bank
which city the most employees live and how many live there?
Similar Example:
Prompt
Generate
In-Context Learning
Wrong Result!
(a) Raw- Raw-Input- Leww-Innut-Basse Selection
Input Query:
Which citiee do mors than one employse under age from?
Prompting LLMs Generate
requires gresier-ther thaler---hnnn less-
Description: than constraints..
Q Select
||||
Example Bank
have more Similar Example: one donator whose smount is less than 8.5.
Prompt
Generate
In-Context Learning
Correct Result !
(b) Skill-Based Selection
Figure 1: In-context learning with different selection methods. (a) Examples from raw-input inputtbbased based selection just share similar entities with the input query. (b) With the skill-based description, the selected examples con- tain the desired task-specific skills.
1 Introduction
In-context learning (Brown et al., 2020) has be- come prevailing paradigm for utilizing large lan- guage models (LLMs) (Hendrycks et al., 2020; Patel and Pavlick, 2021; Rae al., 2021; Zhang al., 2022a; Hoffmann et al., 2022; Srivastava et al., 2022; Chowdhery et al., 2022; Smith al., 2022; Wei et al., 2022a). It employs frozen task- agnostic backbone model to serve various down- stream tasks without requiring parameter updates for each task. Under in-context learning, the LLMs generate output for an input query by condition- ing the prompt that contains input-output exam- ples. Due limited context length of the language Work done during the internship at Microsoft.
model, only few examples can be presented in the prompt. Prior studies have found that the perfor- mance of in-context learning is sensitive to the se- lected in-context examples (Liu et al., 2022: Zhang et al., 2022b; Chen al., 2023b). Therefore, essential research question is: how to select proper examples from large example bank?
Raw-input- selection is one widely applied solution (Gao et al., 2021: Liu et al., 2022: Hu et al., 2022). involves embedding raw inputs examples using an off-the-shed embedding model
Corresponding authors.

---

Input Query
Context
DB Schemn: employes Question: Which oitess mora than one employee under age 30 come from?
city. ...)... ...
Schema: (name,, capanlly
Skill-Based Selection
Question: Find
capway
Query: WHEME cagacity
CRRUU
BY location count())
||||
Example Bank
Schema:
(schcol, donator,
Prompting- Rewriting
donulioo SQL SE18CT WHESE 8.5 CROUP
Question:
that BY achool HAVING count?")
Skill-Based Descriptions
empabyee (naere, age, city. ..)
Candidate 1:This took requires the grester-than less
Question: Which more than employee age
Candidate 2: This taak requires to apply two constraints selected column.
In-C nn-onnext Learning
Offelte-Shelf Embedding Model
SQL Query: SELJECT crry FROM empleyee WHISKE age 30 GBOOP BY city HAVIIG country) >
Candidate
The bird's-eye view of SKILL-KNN, learning with skill-based descriptions.
Figure 2:
selection method to facilitate in-context
and then selecting the most similar examples. It can be conveniently applied in various downstream tasks. However, this method can be easily biased by surface natural language features that are not im- portant for the target task. For instance, in semantic parsing tasks', the raw-input-based selection just finds out examples with similar entities (as illus- trated in Figure la), while the better in-context examples should contain the required executable operations in logical forms, which can be regarded the task-specific skills.
annotated demonstrations, which does not require any fine-tuning process and has rule-based con- straint. Additionally, alleviate the sensitivity the order of annotated demonstrations during gener- ation, we design two variants of SKILL-KNNN we sample set of candidate descriptions by shuffling annotated demonstrations, then select candidate based on consistency and distinctiveness, respec- tively.
The experimental results show that SKILL-KNN brings considerable boost for in-context learning compared the raw-input- selection. We evaluate SKILL-KNN on five challenging semantic parsing datasets: Spider (Yu et al., 2018b), Dr. Spi- der (Chang et al., 2023), KaggleDBQA (Lee et al., 2021), BIRD (Li et al., 2023c), and COGS (Kim and Linzen, 2020). We take six models for in-context learning: text-chat-davinci-O0 code- davinci-002, text-davinci-003, code-cushman-002, gpt-35-turbo, and gpt-4. Across these tasks and models, SKILII-KNN consistently performs best among non non-oracle selection methods and, at times, is even comparable to oracle methods. For instance, with eoxt-cha-ddainnc--00 78.3% exccution accuracy on Spider, while the best raw-input-based selection method reaches 74.6% and Target- KNN2 attains 78.6%. Furthermore, our ablation study indicates that SKILII-KNN retains its superiority when constraints are imposed on the annotated demonstrations, including reducing the number of demonstrations, restricting the database diversity, and decreasing the operation coverage. Our contributions are three-fold: 1) we propose skill-based few-shot selection method SKILL- KNN, which leverages the power of prompting
To overcome this limitation, we aim to make this embedding-basee selection better aware of the intrinsic skills behind the raw inputs. We consider to harness the power of prompting LLMs to con- vert the desired skills from raw inputs, which maintains the training-frre free advantage during selec- tion. There has been much work trying to fine-tune the embedding model for each task based on the example bank (Rubin et al., 2022; Poesia et al., 2022; Hu et al., 2022; Ye et al., 2023). However, fine-tuning-base methods are difficult to apply in practical scenarios: it is laborious to train and save the embedding model for each task, and it is also inconvenient to re-train the model a dynamic example bank that can be updated frequently.
SKILL-KNN achieves
Specifically, we introduce SKILL-KNN, a training-free skill-based selection method (briefly illustrated in Figure lb). Overall, SKILL. KNN will first generate skill-based descriptions from raw input queries, then feed these descriptions into an off-the- she---hhheff embedding model to select most simi- lar examples. To generate skill-based descriptions, we prompt a frozen LLM with just a few human-
'Semantic parsing means to parse an NI. utterance into machine-undere understdanndt logical form (c.g.. SQL query).
"One of the oracle methods, detailed in Scction 4.2.

---

Table Part of our annotated skill-based descriptions for text-to-SQU tasks.
Query
Danchese
Skill Bosed
allengy type (allergy olleryyype]
Show all mijon.
has allengy (suid, allegg)
To safve this takk in the datahoor, we need to sellect distinet values in the column..
sudem (stud. Iname, fname,
major....
number ditfarent colleges playess who pla for Columbus Crew are from.
team country (comery 18, country nome, expital.... ...]
name]
To salve this taak datahose, we join two tables
and count the sumbeer of distiies values in the column.
match seasen [sesson, player position, country, team, ..]
Which catalog coments product that starts from "2"! Show the cattlog enry tames.
catalog stucture (cutalog level number, citalog id....
(cosalog id, catilog
catalog publisher....
dataisuw,, we and apply a constraint on the formar of values in this columa..
To salvee this
column
catslog costeus jeatnlng entry ki, coralig level munther... ...
LLMs to generate skill-based descriptions; 2) we design two variants of SKILL-KNN based on con- sistency and distinctiveness, respectively; 3) our comprehensive experiments across various seman- tie parsing tasks and backbone models demonstrate the effectiveness of SKILL-KNN, and our analy- sis of annotated demonstrations provides further insights for better utilization of SKILL-KNN.
in which x; is the input* of one example (Ii, Ui) E B. Based on Equation 2, we can select k most similar examples from B. In addition, these exam- ples will be sorted in the prompt according to their similarities the test input query: examples with higher similarity scores will be placed closer to the test input query.
This standard implementation of R(I,, B) is a raw-input- bastu selection. It just searches for ex- amples with similar inputs (i.c., the 2, and z; in Equation 2). Some recent researches propose fine-tune the embedding model (from Emb(-) Emb'(.)) (Rubin et al., 2022; Poesia al., 2022: Hu et al., 2022; Ye et al., 2023). In this paper, we want to explore how to improve the effectiveness few-shot selection without training or fine-tuning of any models.
2 Preliminaries
In this section, we introduce embedding- few- shot selection as the preliminary of our method.
2.1
In-Context Learning with Few-Shot Selection
Consider a downstream task T that contains a set of input-output examples {(x; -> )5))" (termed example bank B3y², and pre-traane pre-trained large language model with frozen parameters O. Given test input query Et, the large language model with in-context learning will generate an output y/s by sampling from the following distribution,
SKILL-KNN
SKILL-KNN involves a rewette-then-retrieee pro- cess to better exploit the potential of in-context learning. Figure 2 gives bird's-eye view of our method. To mine and utilize task-specific skills, SKILL-KNN contains a prompting- rewriting stage and a skill-based selection stage. Prompting- rewriting will prompt LLMs to generate skill-based descriptions from the given input query. Skill-based selection will return few- shot examples based these generated descrip- tions. In the following, we elaborate the design of SKILL-KNN.
1/ LL [R( B) o az).
(1)
in which is the sampling temperature, R(x,, B)) returns sequence of examples selected from B according to Ft, and ( means to sequentially con- catenate two sequences. In the later part of the paper, we omit the frozen & and set r - 0 by de- fault, which means to perform greedy decoding. Few-shot selection aims to design the algorithm R(x, B3 that can work well for task T.
2.2 Embedding- Based Few-Shot Selection
3.1 Generating Skill-Based Descriptions
A standard implementation of R(Iz, B) is to lever- age an off-the-shelf embedding model Emb(.) and calculate the embedding similarity of raw in- puts (Liu et al., 2022),
We prompt frozen large language model to rewrite each input query as based description, which does not require any fine-tuning process. Specifi- cally, we first annotate the skill-based descriptions for 16 examples in B, then prompt the large lan-
sim(a: x.) = Emb(xx))mb(as Emb Emb(x;)]
(2)
'For raw-input-ber based selection, we only use the question in the input query for embedding and omit the database schema, as it contains much trivial and redundant information for the question and could confuse the embedding model.
'In semantic parsing tasks, cach input query contains a natural language question along with the database schema.

---

guage model with these annotated demonstrations and generate for other examples in B and for each test input query.
Consistency-Based
Distinctiveness
O +
Note that we annotated skills with natural lan- guage descriptions rather than rule-based con- straints. It is important to note that off-the-shelf embedding models are primarily pre-trained nat- ural language (NL) data and may not be well-suited for handling specifically designed structural con- straints. By annotating skills with NL descriptions, we can better align with the aff-the-shelf embed- ding models, which in turn allows us to leverage their generalizability when encoding unannotated NL descriptions more effectively. Thus, these natu- ral language skills can better suit the aff-the-shelf embedding model, and our annotated demonstra- tions better generalize to unseen data.
Central Sumllarty
Maximum Simlbrity
Candidate Skil-Basse Raprasentations
Figure 3: Two variants of SKILL-KNI The blue and green points represent two candidate sets of skill-based representations.
( - sa). Then, two variants use these didate descriptions from the view of consistency and distinctiveness, respectively. Figure briefly illustrates the basic ideas behind these two vari- ants. Appendix B.1 provides further analysis for the motivation behind the design of two variants.
Formally, with a set of annotated demonstrations {Ia + Saf in which Sa the annotated skill-based description for the raw input Za, we generate the 8; for each unannotated input 24 by prompting the large language model,
Consistency-Based Variant. From the view of consistency, we take the central embedding of all candidate descriptions during selecting examples,
(S St)
T Et Ei
1
Σ Emb(s') Si
m
8; LLM + sa) xi).
(3)
(5)
in which S₁ and S, represent two sets of candidate descriptions for the test input I; and one example (x:, Mi) E B, respectively. This variant is inspired by prior work on improving the consistency of chain-of-thougee reasoning (Wang al., 2022; Li et al., 2022a). As illustrated in the left in Figure 3, Equation 5 can be regarded as an embedding- majority voting among all candidate descriptions during selection.
then, these descriptions are fed into the off-the- shelf embedding model to select similar examples,
(mmb(s,))m(((s)) sim 84) = (amnb(:)[lEmbbbe)
(4)
Table shows part of our annotated demonstrations for text-to-SLL tasks, and all our annotations are contained in Appendix E. Note that different text- to-SQL tasks can share the same set of annotated demonstrations in our experiments.
Distinetiveness-Basee Variant. Considering that the central embedding can sometimes be overwhelmed by trivial candidates, we want to highlight the most distinctive and informative description among all candidates. Formally, we consider the maximum similarity score between two sets for selection,
Equation 4 defines the basic version of SKILL- KNN. Moreover, notice that the generated skill- based descriptions sometimes could be sensitive the order of annotated demonstrations. Such a sensitivity also observed some previous work (Zhao et al., 2021; Lu et al., 2022). Therefore, we design two variants of SKILL-KNN to further address this sensitivity issue.
Emb Famb(s)) (6)
sima S.) = max
j.k [Emb [Emb(s)]][nnnn) [[m[[E(((s)))))
3.2 Variants
in which sf St and sf S,. As illustrated in the right in Figure 3, Equation 6 means that take the minimum distance among two set of candidates for selecting similar examples.
To alleviate the influence from the sensitivity to prompt order, we design two variants of SKILL- KNN that change the order of annotated demonstra- tions and perform rewriting multiple times. Specifi- cally, for each input 2; both two variants generate a set of candidate descriptions S; { s ....., s;") according to Equation by changing the order in
Experimental Setup
In this section, we will introduce the tasks, com- pared selection methods, backbone models, and

---

Table 2: Our main experimental results (%) across various LLMs and tasks. Numbers in bold are the best results across non-oracle methods, and results with underlines outperform least one oracle method.
Backhone
Mahhod
Spider
Dr. Spidze
KDRQA
BIRD
COGS
FWin
DB
NLQ
SQL
P..
P.A.
KNN w/ SBERT (Liu et 2022)
73.0
51.5
59.2
67.3
24.3
38.3
78.5
71.1
0
KNK w/ OpenAl Babbage (Luu all, 2022
74.0
53.2
61.0
69.2
35.3
38.1
86.3
74,1
0
w/ OpenAL
(Lin al., 2022)
51.5
59.2
MMR w/ OpenAl Ada (Ye et 2022)
74.6
34.1
00.7
71.3
31.3
57.2
86.8
73,9
0
SKILL-KMR of SBERT (basc)
76.8
55.3
60.3
72.1
34.7
38.9
93.8
58.3
0
tesicchat-hoiiss-002
consistenyy
$4.8
divinniimmss
78.3
$7.0
61.4
72.2
40.0
92.6
58.8
SNILL-KNNN o/ OpenAl Ada (buse)
72.1
55.4
60.5
70.9
38.7
38,9
91.9
95.6
752
54,8
59.6
71.1
40.0
95.2
93.5
56.5
59.8
97.0
Tregei-KNN (omele)
78.6
36.6
63.5
75.5
37.8
41.7
M6.8
53.2
Target Sketch Matching (vescke)
79.5
$4.2
65.1
76.2
40.4
40.8
96.9
95.7
Ramdum
$3.9
35.
KNK w/ SIEERT (Liu et 2002)
73.1
51.2
31.2
39.9
75.4
64.7
KNN W/ OpenAl Babbage (Luu et al. 2022)
73.6
50.7
59.6
69.1
37.5
38.22
95.4
72.4
0
OpenAT
2022)
MMR OpenAl Ada (Ye et 2022)
74.8
53.7
60.7
69.4
37.8
37.9
81.9
75.4
SKIT NN of SAERT (basc)
76.4
$3.0
60.2
72.4
38.6
40.2
93.1
86.8
code-davined 0012
cumbstercy
77.1
51.1
60.7
72.4
36.8
40.1
93.5
87.1
distinctveness
55.0
71.0
91.9
58.4
SKH1-KNN of OpenAl Ada (Fore))
76.2
$4.7
61.4
70.2
39.8
40.3
94.0
93.9
76.4
$4.9
60.2
71.8
39.5
40.8
96.2
58.8
distinctiveeess
55.0
Torget KNN (ocolla)
78.8
56.1
68.2
76.4
44.5
44.3
85.9
79.7
Target Sketch Matching (ocsele)
$0.9
$4.0
66.7
77.9
44.9
43.8
95.0
94.8
Randam
69.0
52.2
55.1
64.5
65.5
61.2
KNN w/ SBFRT in 2022)
60.9
50.3
56.1
82.1
71.1
KNN w/ OpenAl Babloge (Liu et 2022)
72.2
534
$8.0
69.5
26.8
39.7
$8.8
77.2
0
KNK w/ OpenAl Ada (Lin 2022)
70.8
50.3
57.2
66.3
20.3
82.8
64.2
MMR OpenAl Ada (Ye
58.6
S SKILL- SKILL-KNN of SBERT (base)
74.9
54.4
59.0
70.2
32.9
38.0
94.8
$9.4
0
text davinci-ooo 003
consestency
75.3
$4.8
59.0
70.6
32.0
38.1
94.3
80.2
0
didimcciuuuss
$4.1
S BNILL-KNN of OgenAl Ada (basc)
74.2
55.2
59.5
68.5
34,0
38.4
95.7
94.3
consestency
74.3
55.0
59.5
71.0
36,8
40.2
96.7
92.7
97.4
Turget-NN KNN (omele)
63.9
73.3
$4.5
Turget Sketch Matching (oracle)
76.4
$2.5
61.4
72.0
31.5
39.8
97.6
95.3
Random
72.2
51.5
56.6
66.8
35.3
56.7
55.6
W/ SBERT et al. 2022)
47.5
54.4
20
62.5
KNN w/ OpenAl Babbuge (1 et all, 2022)
71.6
48.6
$6.4
66.8
36.4
36.4
77.5
71.6
0
KNK w/ OpenAT Actx in
2022)
68.9
48.1
$7.3
67.4
31.9
34.8
71.3
54.7
OpenAl (Ye 2022)
57.1
SKILL KNN w/ SBERT (base)
24.5
52.1
$8.0
69.3
35.1
37.0
90.6
$4.1
0
code-culman-0tt 0022
Conslstency
74.7
52.3
59.00
69.6
35.3
38.3
91.1
56.6
didinccivvses
$2.0
58.8
NN OpenAl (raee)
73.6
52.4
58.4
69.0
38
93.2
$6.6
+ combiseency
73.5
$3.0
$3.9
69.8
40.0
38.6
95.0
$2.8
4
didiaactveness
51.2
Turget KNN (omcle)
52.5
66.6
73.8
72.8
Target Sketch Matching (oczele)
77.2
31.7
62.2
75.2
39.7
39.8
91.9
94.0
hyper-parameters in our experiments.
Dr. Spider (Chang et al., 2023) is a diagnostic evaluation benchmark constructed based on Spider. It contains 15,269 examples which can be divided into sub-tasks, according to the type of designed perturbations: database perturbations (DB), natural language question perturbations (NLQ), and SQL query perturbations (SQL). We take the train set of Spider as the example bank, since Dr. Spider is purely an evaluation benchmark.
4.1 Tasks
We evaluate on five challenging cross-domain se- mantic parsing datasets. Due to the cross-domane property, the model can not easily solve these tasks by just copying some similar surface features from the provided in-context examples.
Spider (Yu et al., 2018b) is large-scale text-to- SQL dataset. It contains train set with 7,000 ex- amples and a dev set with 1,034 examples. More- over, the train set and dev set do not share any database. We take the train set of Spider as the example bank, and evaluate on the dev set.
KaggleDBQA (Lee et al., 2021) (KDBQA) is a small while complex dataset towards realistic eval- uation of text-to-SQL semantic parsers. It contains real Web databases with original formatting and

---

Table Performance of gpt-35-turbe and gpt-4 on Spi- der dev set.
Table 5: Performance of SKILL-KNN and two baselines on GSM8K.
Backhone
Method
Exec. Acc.
Backbone
Method
Accuracy
Random
76.1
Random
KNN w/ SBERT (Liu et al., 2022)
76.7
text-chat davinci--iiiee 0012
KNN w/ SBERTT (Iái 2022)
69.9
SQL (Poureeza and Ralici, 2023)
74.2
gpt-4
SKILL- SKILL-KNN KNN SBERT (base)
71.0
SKILL. KNN SBERT (buse)
consistency
82.7
distinctiveness
82.3
the input query (detailed in Appendix D.3). We use the output format designed in An al. (2023) and evaluate on two sub-tasks, primitive substitution
Random
74.3
KNN w/ SBERT (Liu et al., 2022)
73.7
KNN OpenAl Babbage
2022)
73.4
gpt-35 turbo
SKILL KNN w/ SBERT (base)
76.2
(P.S.) and primitive structural alternation (P.A.).
consislency
76.3
distinctiveness
76.8
4.2
Selection Methods
Table 4: Comparison with the fine-tuning based selec- tion methods on Spider dev set.
We mainly compare SKILL-KNN with training- free selection methods.
Backbone
Method
Exec, Acc.
Random. We randomly select examples from B as in-context examples. For each test case, we take random selections 3 times and average the results.
EPR (Rubin 2022)
74.4
CEIL. (Ye al., 2023)
75.0
text-chat-davinci
TST (Poesin et al., 2022)
76.3
SKILL-KNN w/ SBERT (base)
76.8
KNN (Liu et al., 2022). We test three off-the-sheil embedding models: Sentence-BERT (SBERT) with all all-mpnet-base-v2 checkpoint (Reimers and Gurevych, 2019). OpenAl embedding model with eextsimilarity--abbbe-000 checkpoint (OpenAI Babbage), and OpenAl embedding model with text-embedding--atee -aa----d--0222 checkpoint (OpenAI Ada). KNN with OpenAl embedding models can serve as strong baselines for training-free selection methods, as these large models have been well pre-trained for judging text similarity (Neelakantan al., 2022).
EPR (Ruhin al., 2022)
75.9
codo-davinci-002
SKILL-KNN SBERT (base)
76.4
consistency
77.1
distinctiveness
77.4
EPR (Rubin al., 2022)
69.7
lext-davinci-003
SKILL-KNN w/ SBERT (base)
74,9
consistency
75.3
distinctiveness
76.6
EPR (Rubin al., 2022)
74.6
code-cushman-Oee
SKILL-KNN w/ SBERT (base)
74.5
consistency
74.7
distinntivvneess
72.8
MMR (Ye et al., 2022) is dynamic selection method to enhance the diversity of selected exam- ples from KNN. adds a penalty term according to the similarity to the already selected examples. We take OpenAI Ada for embedding and follow the implementation details in Ye et al. (2022).
275 unrestricted questions. Since ther is not much data in KDBQA, we take it pure test set and use the train set of Spider as the example bank.
BIRD (Li et al., 2023c) a large scale text-to-SQL dataset with real-world database contents. It has 9,428 training examples and 1,534 test cases in the dev set. Each case in BIRD is equipped with description of the required external knowledge, which is not contained in above three text-to-SQL. tasks. Since the database schema in BIRD is too large, we first take grounding to reduce the size of schema (detailed Appendix D.4).
SKILL-KNN (ours). We test SKILL- KNN with SBERT and OpenAl Ada. For the base version of SKILL-KNN (i.e., without consistency or dis- tinctiveness), we shuffle the order of annotated demonstrations generate = 5 skill-based de- scriptions for each input query and average the results. There is a balance between achicving op- timal performance and minimizing computational costs. We provide more experimental analysis Appendix B.2. For two variants, we take all 5 gen- crated descriptions as the candidate set.
COGS (Kim and Linzen, 2020) is synthetic benchmark for testing compositional generalization in semantic parsing. It can also be regarded cross-domain setting, containing a significant distri- bution shift between train set and test set. The logi- cal form in COGS represents the thematic roles in
hhttps://ww/ wwert.net/. "https://platfors hwtppppaatttrropppaaiiiommm wwwpapp.ceooeaaaaaoeooo wtt.l/cccooo////eeeee/ee wwwiiddc///ggiiiddd gusd/// ddings/what- are httpss//wwhatarreeeeeennnnnee

---

16
text-chat that-davinci-002
code davinci 002
text deviddiiiiiooo 003
code cushman 002
Figure 4: Performance of SKILL-KNN -NNN (base version) with different number of annotated demonstrations.
Besides these training-free baselines, we also compare with three fine-tuning-base based methods. These methods leverage the example bank for fine- tuning the embedding model.
code-davinci-002,, text-davinci-003. code-cushman- 002, spt-35-turbo, and gpt-4. For generating skill- based descriptions, we always use gpt-3.5-turbo, as it is cheap and fast. We select examples in all experiments. We use execution- values accuracy as the evaluation metric for text-to-SQL. tasks and exact-match accuracy for COGS.
4 in-context
EPR (Rubin et al., 2022) requires a scoring LM to produce positive/negative examples for fine-tuning the embedding model, and we use GPT-J (Wang and Komatsuzaki, 2021), a 6B-parameter LM as the scoring LM?.
Main Results
CEIL (Ye et al., 2023) proposes a compositional selection method for tn-context learning. It models the compositional interaction between the given input and in-context examples, and fine-tunes the selection model through carefully-designe con- trastive learning objective.
Table 2, Table 3, and Table 4 report the main exper- imental results. We also count the number of wins, i.e., how many tasks (and sub-tasks) the method performs best
SKILL-KNN performs better than raw-input- based selection methods. Across all backhone models and tasks, our skill-based selections achieve the best performance among non-oracle methods. Especially, SKILL-KNN with SBERT can even outperform KNN with OpenAl embedding models. These results clearly demonstrates the necessity and effectiveness our prompting-bui based rewrit- ing. Appendix A.2 contains more experimental comparisons with existing selection methods.
TST (Poesia al., 2022) retrieves few-shot exam- ples from a training bank using target similarity tuning. learns to recognize utterances that de- scribe similar target programs despite differences in surface natural language features.
In addition, we also compare with two oracle methods, which ground truth output sequences are allowed to be leveraged for few-shot selection.
Target-KNN (oracle). We select examples with similar output embeddings. We use OpenAl Bab- bage and OpenAl Ada to encode the ground truth, and take the best result of two models for each task.
SKILLI-KNN performs comparable/better than fine- tuning-based method. Results in Table 4 show that SKILL- KNN can perform comparable or even better than fine-tuning-base methods. It demonstrates that optimizing the input to the em- bedding model can also effectively help down- stream tasks without any fine-tuning.
Target Sketch Matching (oracle). We select in- context examples with similar sketches of ground truth. For text-to-SQL tasks, we calculate the over- lap SQL key words (detailed in Appendix D.6).
Variants with consistency and distinctiveness can outperform the base version of SKILL-KNN. As shown in the #Wins column in Table 2, two vari- ants of SKILL-KNN outperform the base version in most situations. It demonstrates that injecting consistency and distinctiveness can effectively al- leviate the order sensitivity. Overall, we recom- mend choosing the distinctiveness variant as it wins
For COGS, we follow the target-side structural sim- ilarity setting in An et al. (2023).
4.3 Backbones and Hyper-parameters
We conduct experiments with six language models as the b text-chat-davinci-002.
OpenAl language models cannot used as the scoring function since OpenAI API does not provide this functionality. "In this work, the "hackhone" refers to the frozen large language model for prompting.
"We use the official evaluation scripts for Spider in http:/hhww...oommaoyo/ttttttttte

---

Yell-cMi o/ SAERT
Vell-rN o/ SRIIT Constrain Operation Covorage
sa11-000 w/ SRIRT
05 Diverity
w/o Corstrantt
Executinn Arousty
Earntte Accuneyy
10
70
74
72
74
72
74
text-chat-dawlnc-002
code-davind-002
text-davinc-003
code-cushman-002
Figure 5: Performance of SKILL- -NNN (base version) with constraints on selecting examples for annotating.
more times than the consistency variant (19 vs 15). When looking into detailed results, different mod- els prefer different variants of SKILL-KNN: text- chat-davinci-002 and code-davinci-002 prefer to the distinctiveness variant, while text-davinci-003 and code-cushman-002 prefer to the consistency variant. We identify the potential factor that could lead to different preferences in Appendix B.5.
of annotated demonstrations from 16 to 4/8/12. The results depicted in Figure 4 illustrate that gradual increase in the number of annotated demonstra- tions can yield marginal improvements. Note that SKILL-KNN maintains the advantage compared with the raw-input- based selection even with only four annotated demonstrations. This indicates that the LLM can effectively learn how to rewrite from a limited number of examples and generalize to a wider range of unseen skills. This generalizability is also supported by our case study in Appendix C.
SKILL-KNN is more robust to perturbations than raw-input-base selections. Results on Dr. Spider reflect the robustness towards perturbations in data. For instance, with text-chat-dee wwwhhhhaviiii--ni0 KNN with SBERT performs lower than the random baseline two out of three types of perturbations, while all three versions of SKILL-KNN outperform the random baseline on all three perturbations. indicates that - leads more robust in-context learning than raw-input- www-input-basee based methods.
SKILL-KNN retains its superiority when the selection of annotation examples is constrained. We constrain the selection of annotation examples from two perspectives (detailed in Appendix D.1); first, we limit the SQL operation coverage in anno- tation examples; second, we restrict the annotation examples few databases. Figure 5 shows that applying these constraints to the selection of anno- tation examples leads to only a minor decline performance, while still maintaining a substantial advantage over raw-input-baat based selection.
SKILL-KNN can be effective in the math rea- soning task. Our study primarily evaluated the effectiveness of SKILL-KNO in the semantic pars- ing/code generation field. To further examine the generalizability of SKILL-KNN beyond semantic parsing, we have applied it to challenging math reasoning task, GSM8K (Cobbe et al., 2021). Re- sults in Table 5 evidence that SKILL-KNN can also be effective in tasks beyond semantic parsing.
Related Work
In-context learning has recently become a stan- dard paradigm for effectively leveraging large lan- guage models (Brown et al., 2020; Hendrycks et al., 2020; Patel and Pavlick, 2021; Rae et al., 2021; Zhang et al., 2022a; Hoffmann et al., 2022; Srivas- tava al., 2022: Chowdhery al., 2022: Smith et al., 2022; Wei al., 2022a). Such a convenient paradigm has been widely applied in various sce- narios such as code generation (Chen et al., 2021; Bareiß et al., 2022: Li et al., 2022b; Chen et al., 2023a; Li et al., 2023b), arithmetic reasoning (Wei et al., 2022b; Wang et al., 2022; Li et al., 2022a; Shi et al., 2023; Qin et al., 2023), and semantic parsing. From the view of leveraging skills for in- context learning, most existing work considered explicitly injecting symbolic systems into the re- sponse of the model (Cheng et al., 2023; Creswell et al., 2023; Schick et al., 2023; Shen et al., 2023;
6 Analysis
The most important mechanism in SKILL-KNN is the prompting-basee based rewriting which requires a few manually annotated demonstrations. Here we investigate how would these demonstrations affect the performance of SXILL-KNN, We experimen- tally analyze two factors: the number of annotated demonstrations and the selection of examples for annotation. The following analysis takes the base version of -XILL-KNN with SBERT as the embed- ding model and the Spider dataset for evaluation.
More annotated demonstrations bring marginal improvements. We decrease the default number

---

Lu et al., 2023). This work aims to uncover the intrinsic skills from the raw inputs of examples.
tate few-shot selection, which a novel perspective to harness the power of large language models.
Semantic parsing with deep learning methods has been explored much existing work (Dong and Lapata, 2016; Yu et al., 2018a; Xu et al., 2017: Guo et al., 2019; Zhong et al., 2020; Wang et al., 2020; Lin et al.. 2020: Scholak et al.. 2021; Qi et al.. 2022; Li et al., 2023a). Under the recent in-context learning paradigm, there have been some prelimi- nary observations: Shin et al. (2021) showed that GPT-3 is better generating English-like descrip- tions rather than the raw logical forms; Rajkumar al. (2022) revealed that prompt design essential for semantic parsing with Codex; Liu et al. (2023) showed that ChatGPT has surprising zero-shot performance on Spider and its variants; Pourreza and Rafiei (2023) demonstrated that explicitly tak- ing multiple stages for generating SQL leads to better in-context learning performance. These ob- servations indicate that in-context learning has great potential on solving semantic parsing tasks, and this work aims to further activate this potential from the view of improving sho-shot selection.
Conclusion
This work proposes SKILL-KNN to facilitate in- context learning on semantic parsing tasks. By gen- erating skill-based descriptions without any fine- tuning, SKILL-KNN and its two variant outper- form raw-input-based selections in various tasks.
Limitations
GPU resources. Our experiments have a high cost on GPU resources, since in-context learning requires extremely large language models. Specif- ically, all experiments are conducted on the x NVIDIA A100 GPU station. During inference time, takes about x 8 GPU hours to generate for each 10,000 examples. Thus, it totally takes 400 ~ 500 X 8 GPU hours to reproduce our Table 2.
Task type. We mainly evaluate SKILI-KKNN on cross-domen semantic parsing tasks, and we be- lieve it can also help other challenging tasks where some intrinsic task-specific skills are needed. How- ever, for tasks that require only surface feature sim- ilarity of in-context examples, we suppose the ad- vantage of SKI.I-KKNN could be diminished.
Few-shot selection one essential part for in- context learning. The standard approach to use an off-the-shelf embedding model to encode raw inputs and select top-k similar examples (Gao et al., 2021; Liu et al., 2022; Hu et al., 2022). To improve in semantic parsing tasks, much prior work tried fine-tuning-based methods: Rubin et al. (2022) fine- tuned the embedding model based on the condi- tional probability under the language model; Poesia et al. (2022) trained the model to fit the target-side similarity; Hu et al. (2022) fine-tuned the model based on the similarity of between state changes; Ye et al. (2023) considered the interaction among in- context examples during training the selector. Our work points in new direction that does not require further fine-tuning: leveraging task-specific skills by prompting large language models to rewrite in- put queries. Beyond semantic parsing, some more recent work tried to explore training-free selection methods for in-context learning from different per- spectives: Nguyen and Wong (2023) and Li and Qiu (2023) tried to distill the whole example bank into a small set of exemplars and focused on clas- sification tasks; Wu et al. (2023) improved clas- sification tasks through information compression; Ye al. (2022) and Ye and Durrett (2023) mainly focused explanation-be based tasks. This work pro- poses prompting extremely large models to facili-
Individual variants. We design two variants of SKILL- KNN based on consistency and distinctive- ness, respectively. An ideal variant should take into account both these two aspects. We take this as future direction for our work.
Ethies Statement
Due to the use of pre-trained language models, this work can be exposed to potential ethical risks as- sociated with general deep learning models, such as social bias and privacy breaches. We suppose this work would be helpful to alleviate potential ethical issues for in-context learning as it can better overcome the surface-form biases in example bank.
Acknowledgments
We thank all the anonymous reviewers for their valuable comments. Shengnan An and Nanning Zheng were supported part by NSFC under grant No. 62088102.

---

References
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023. selection-inferenn Exploiting large language models for interpretable logical reasoning. In The Eleventh International Conference on Learning Rep- resentations.
Shengnan An, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng. Jian-Guang Lou, and Dongmei Zhang. 2023. How do in-context examples affect compositional generalization?
Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1 Long Papers), pages 33-43.
Patrick Bareiß, Beatriz Souza, Marcelo d' Amorim, and Michael Pradel. 2022. Code generation tools (almost) for free? a study of few-shot, pre- trained language models on code. arXiv preprint arXiv22066.03335
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:7
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ( (Volume I: Long Papers), pages 3816-3830.
Shuaichen Chang, Jun Wang, Mingwen Dong, Lin Pan, Henghui Zhu, Alexander Hanbo Li, Wuwei Lan, Sheng Zhang, Jiarong Jiang, Joseph Lilien, Steve Ash, William Yang Wang, Zhiguo Wang, Vit- torio Castelli, Patrick Ng, and Bing Xiang. 2023. Dr.spider: A diagnostic evaluation benchmark to- wards text-to to SQL robustness. In The Eleventh Inter- national Conference on Learning Representations.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian- Guang Lou, Ting Liu, and Dongmei Zhang. 2019. To- wards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Com- putcattinal Linguistics, pages 4534-4535.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Mcasuring massive multitask language under- standing. In International Conference on Learning Representations.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023a. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute- optimal large language models. arXiv preprint 2rXiv:2203.1555:
Mark
Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0332
Yushi Hu,
Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022. In-context learning for few-shot dialogue state tracking. arXiv preprint arXiv:2203.08562
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKe- own, and He He. 2023b. On the relation between sensitivity and accuracy in in-context learning.
Rohit J Kate. Yuk Wah Wong, Raymond J Mooney, et al. 2005. Learning transform natural formal languages. In AAAI, volume 5, pages 1062-1068.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni. Yushi Hu, Caiming Xiong, Dragomir Radey, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages. In The Eleventh Inter- national Conference on Learning Representations.
Najoung Kim and Tal Linzen. 2020. Cogs: A compo- sitional generalization challenge based on semantic interpretation. In 1 Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2004.02311.
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. Kaggledbqa: Realistic evalua- tion of text-to-sql parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticy and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 2261-2273.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. ArXiv, abs/21 10. 14168.
Itay Levy, Ben Bogin, and Jonathan Berant. 2022. Diverse demonstrations improve in-context compositional generalization. arXiv preprint arXiv:2212.06800.

---

Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023a. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.
Xu Jiang, Karl Cobbe, Tyna Eloundou. Gretchen Krueger, Kevin Buttonn Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted
with human feedback.
Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023b. Towards enhancing in-context learning for code generation. arXiv preprini arXiv:2303. 17780.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger. David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang. Peter Welinder, and Lilian Weng. 2022. Text and code embeddings by con- trastive pre-training.
Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang. Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo. Xuanhe Zhou, Chenhao Ma. Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023c. Can Ilm already serve as a database interface? big bench for large-scale database grounded text-to-sqls.
Xiaonan Li and Xipeng Qiu. 2023. Finding support- ing examples for in-context learning. arXiv preprint MrXiv:2302.13555
Tai Nguyen and Eric Wong. 2023. In-context ample selection with influences. arXiv:230..1044.
arXiv preprint
Yifei Li. Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022a. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.0233:
Roma Patel and Ellie Pavlick. 2021. Mapping language models to grounded conceptual spaces. In Interna- tional Conference on Learning Representations.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. 2022b. Competition- code generation with alphacode. Science, 778(6624):10922-111
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gul- wami. 2022. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4870-4888.
Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text- to-sql with self-correction.
Jicxing Qi, Jingyao Tang, Ziwei Hc, Xiangpeng Wan, Chenghu Zhou, Xinbing Wang. Quanshi Zhang, and Zhouhan Lin. 2022. Rasat: Integrating relational structures into pretrained seq2seq model for text-to- sql. arXiv preprint arXiv:2205.0 06983.
Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S. Yu. 2023. A comprehensive evaluation of chatgpt's zero- shot text-to-sql capability.
Jachang Liu, Dinghan Shen, Yizhe Zhang, William B
Dolan, Lawrence Carin, and Weizhu Chen. 2022.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaso Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? arXiv preprint 2rXiv:2302.06476..
What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop Knowledge Extrac- tion and Integration for Deep Learning Architectures. pages 100-114.
Jack W Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides. Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11 1446.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang. Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023. Chameleon: alug-and-play compo- sitional reasoning with large language models. arXiv preprint 2.0.0042..
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098.
Nitarshan Rajkumar, Raymond Li, and Dzmitry Bah- danau. 2022. Evaluating the text-to-sql capabil- ities of large language models. arXiv preprint arXiv:2204.n 4.0098.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese nettrks In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNI.P) pages 3982-3992.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,

---

Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655-2671.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. Rat-sql: Relation-aware schema encoding and linking for text- to-sql parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7567-7578.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint ar 2rXiv:2302.04711.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https://8t //git//ittbb com/kinggoflllz/messt -transformer-jjar
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self- celficonsis im- proves chain of thought reasoning in language mod- cls. arXiv preprint arXiv:
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. Picard: Parsing incrementally for constrained auto-regressive decoding from language models. Proceedings of the 202| Conference on Empirical Methods in Natural Language Processing, pages 9895-9901.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint
Ozan Sener and Silvio Savarese. 2018. Active learn- ing for convolutional neural networks: A core-set approach.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le. and Denny Zhou. 2022b. Chain of thought prompt- ing clicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2023. Self-adaptive in-context learn- ing: An information compression perspective for in- context example selection and ordering.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das. and Jason Wei. 2023. Language models are multi- lingual thain-of-thought reasoners. In The Eleventh International Conference on Learning Representa- tions.
Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqinet: Generating structured queries from natural language without reinforcement learning.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr. Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715.
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional ex- emplars for in-context learning. arXiv preprint wwXiv:2302.05690
Xi Ye and Greg Durrett. 2023. Explanation selection using unlabeled data for in-context learning. arXiv preprint arXiv:2302.048/ 04813.
Shaden Smith, Mostofa Patwary. Brandon Norick. Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron train megatron-turing nlg 530b, large-scale generative language model, arXiv preprint arXiv:2201.11 19900.
Xi Yc, Srinivasan Iyer. Asli Celikyilmaz, Ves Stoy- anov, Greg Durrett, and Ramakanth Pasunuru. 2022. Complementary explanations for effective in-context learning. arXiv preprint (tt::2221113899
Tao Yu. Zifan Li. Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. Typesql: Knowledge- based type- aware neural text-to-sql generation. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pa- pers), pages 588-594.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adríà Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 046755.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, et al. 2018b. Spider: A large- scale human laman-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 911))

---

Proceedings the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 391 1-3921.
John M Zelle and Raymond J Mooney. 1996. Learning parse database queries using inductive logic pro- gramming. In Proceedings of the national conference on artificial intelligence, pages 1050-1055.
Luke S 'Zettlemoyer and Michael Collins. 2012. Learn ing map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv: /207. 1420.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022a. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00
Yiming Zhang, Shi Feng, and Chenhao Tan. 2022b. tive example selection for in-context learning, arXiv preprint arXhv:2211.0444
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- lernational Conference Machine Learning, pages 12697-12706. PMLR.
Victor Zhong, Mike Lewis, Sida Wang, and Luke Zettlemoyer. 2020. Grounded adaptation for zero- shot executable semantic parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6869- 6882.

---

This is the Appendix of the paper: Skill-Baseer
in Table 7 further demonstrate the advantage of SKILL-KNN.
Few-Shot Selection for In-Context Learning.
A
More Experimental Results
B
More Analysis
We report more experimental results with SKIIIL- KNN.
B.1 Motivation Behind Two Variants of SKILLL-KNN
The motivation behind the two variants stems from our consideration of disturbances from the prompt- order sensitivity as additive noises to the ground- truth skills during the rewriting process. The two variants are designed to address two types of noise:
A.1 Recall@ N Performance of SKILL-KNN
Table @ performance of SKILL-KNN with distinctiveness on Spider dev set. The backbone model is text-chat-davini-0022 and the sampling temperature is 0.7.
Zero-mean white noise which frequently occurs in results and originates from zero-mean distri- bution (c.g., zero-mcan Gaussian distribution). We assume its magnitude is relatively small compared to the ground truth. Zero-mean white noise can cause the loss or redundancy of partial information in the ground truth.
Top-N
3
5
7
10
Recall N (%) 80.3
85.0
87.4
89.2
89.9
90.8
Besides the performance of we also evaluate the top-k recall performance of SKILL-KNN. Since we cannot take beam search with OpenAI interfaces, we implement the top-N selection with sampling and re-ranking, following the self-consistency setting (Wang et al., 2022). Specifically, we first sample 100 sequences and then select the & most frequently occurring quences and evaluate their execution accuracy.
Spike noise which occasionally occurs in results and has a much larger magnitude than the ground truth. It strongly influences the information the ground truth and causes outliers.
The consistency-basee variant is more effec- tive at addressing zero-mean white noise, the averaging operation reduces the variance of zero- mean noise. The distinctiveness- based variant is better suited for handling spike noise, as it miti- gates the influence of outliers. The final results our Table 2 indicate that both types of noise occur in our LLM-based rewriting, as evidenced by the close win times of the two variants (19 vs 15).
We take the text-chat-davinci-o0e as the back- bone model and set the sampling temperature as 0.7. We evaluate SKILI-KNN with distinctiveness on Spider dev set. Table 6 shows that the recall rate gradually goes up with increasing N and can even achieve higher than 90% when we set N 10. Moreover, the top-1 in Table 6 is 80.3% which is higher than the greedy-search performance (78.3% in Table 2). It means that our SKIL.L-NNN can obtain further gains from ensemble methods such as self-consistency.
To further support that two variants are better at tackling two different types of noise, conduct additional analysis from two perspectives.
Table 8: The selection accuracy of different variants under different noise patterns.
A.2 Comparison with More Baseline Methods
Zero-Mean White Noise
Spike Noise
Table 7: Comparison with more baselines on Spider dev
Consistency Distinctiveness
95.4%
81.9%
90.4%
87.9%
set.
Backbono
Method
Evaluation of selection performance. We ex- amined the performance of each variant in select- ing better examples from the example bank under the noise patterns. We construct some syn- thetic data for automated evaluation: we take 1,000 unique embeddings of skill-based descriptions as ground truth, then we add noise on each ground- truth embedding to construct the sample set, and
Random
72.9
Best-of-5 (Nakano et al., 2022)
75.7
K-Center (Sener and Savarese, 2018)
73.4
text-chat-davinii--00
Influence (Nguyen and Wong, 2023)
73.4
DDP (Ye et al., 2023)
74.6
SKILL- SKILL-KNN KNN w/ SBERT (buse)
76.8
Despite the baselines mentioned in Section 4.2, here we reproduce more selection methods. Results

---

finally we assess whether each sample set could correctly select the original ground-truth embed- ding. The accuracies are shown in Table
tently outperforms at least one oracle method on the DB sub-task in Dr. Spider and two sub-tasks in COGS (marked with underlines). Such an ob- servation could be caused by the different diversity in selected examples. As indicated in Levy et al. (2022) and An et al. (2023), beyond the similar- ity to the test case, a higher diversity among in- context examples could also help better perform cross-domain generalization under in-context learn- ing. Oracle methods directly seek higher similarity, thus the selected examples may be less diverse, which could slightly hamper the generalization per- formance. To reflect the diversity of in-context ex- amples, we count the number of different databases among selected examples. Statistics in Table 10 shows that SKILL-KNN can lead to a higher diver- sity than oracle method, which is in line with our hypothesis.
Human evaluation of rewriting. In Spider dev set, we first identified examples where one vari- ant can always succeed with different LLMs while the other variant always failed. Then, check the frequency of spike noise occurrences in these examples through human evaluation. We checked 23 examples and found that: for examples where the consistency- based variant wins, the spike noise occurs in 0.4 sample/set; for examples that distinetiveness--aae based variant wins, the spike noise occurs in 1.6 sample/set.
The results of the above analysis further evi- variant performs
dence that the consisteney- better than the distinctiveness--aatt based variant under the zero-mean white noise but performs worse un- der the spike noise.
B.4 Why do skill-based descriptions perform better?
B.2 The Choice of Hyper-Parameter m
Since both SKILL-KNN and raw-input- meth- ods use the embedding similarity for selection, we suppose that the higher performance of SKIILL- KNN can be contributed by some desired prop- erties in the embedding space of skill-based de- scriptions. Based on this inspiration, we visualize the embedding space of both raw input queries and skill-based descriptions with t-SNE ( (Van der Maaten and Hinton, 2008). More details are con- tained in Appendix D.7.
Table 9: Performance with m 3/5/7/9 (Dataset: Spider, LLM: cole-cushman-0022 variant: consistency).
9
Exec. Acc. (%) 72.8 74.7
75.1 75.0
The two variants of SKILL-KNN require to gen- erate m candidate skill-based descriptions. In our experiments, we set m = 5 as trade-off be- tween achieving optimal performance and minimiz- ing computational costs. During our initial explo- ration, we had experimented with m 3/5/7/9. As shown in Table 9, the performance improves marginally when m > Therefore, we decided to set m 5.
Under the embedding space of skill-based de- scriptions, the distribution of test cases is closer to that of the example bank, thus benefiting cross-domain generalization. For the raw-input- based embeddings of test cases, Figure 6a shows that these embeddings some local parts. On the one hand, the example bank can not be fully utilized under this embedding space, since the top-k similar examples must be around the local parts of test cases. On the other hand, the different distributions represent that this space does not reveal the inner similarity between test cases and example bank, thus is helpless to facilitate cross-domain generalization. Under the skill-based embedding space (shown in Figure 6b), the distributions of test cases and the example bank are better matched. Therefore, the cross-domain generalization gap can be better bridged with skill- based descriptions.
B.3 Measuring Diversity of SKILL-KNN and Oracle methods
mainly centralized in
Table 10: Number of different databases among selected examples. This can reflect the diversity of in-context examples selected by different methods.
Number of Different Databases
Spider
Dr. Spider
KDBQA
SKILL-KNN (distinctiveness)
2.83
2.84
Target-KNi (oracle)
2.18
2.16
2.21
A surprising observation in Table 2 is that with the same backbone model for in-context learning, our BEILL-KNNY could sometimes outperforms or- acle methods. Specifically, SKILL-KNN consis-

---

Exomple BONK Cauna
Exampic
(a) Law-Input-Based Based Embedding Space
(b) Skill-Based Embodding Space
Figure 6: T-SNE visualization for the embedding space of (a) raw input queries and (b) skill-based descriptions. The orange points are from the dev set in Spider while the gray points are from example bank.
The skill-based embedding space is more sparse, thus the boundary of similar examples is more clear and can be more robust to perturbations. As shown in Figure 6a, the raw-input-basee embed- ding space is almost evenly distributed. It means that for one embedding in this space, the bound- ary to determine "which examples are similar" is not clear enough. Without a clear boundary for selecting similar examples, the performance could be non-robust to perturbations in data. Compared with the embeddings of raw inputs, the skill-based embeddings shown in Figure 6b are more clustered, thus the KNN-based selection can be more robust to data perturbations.
distinetiiennes--aseed variant in all tasks and under both perspectives. can help to explain why different models prefer different variants: text-chat- davinci-O01 and code-davinci-Ou1 prefer simpler in-context examples while text-davinci-009 and Node-cushman-O00 are more robust to different complexity.
C Case Study
Considering that the quality of generated skill- based description can be one key factor that influ- ences the effectiveness of SKILL-KNN, we man- ually check the generated skills for 100 examples. We find that 86/100 generated skills are exactly correct; 12/100 are almost correct but need some partial modifications (e.g., the number of joined tables); and only 2/100 generated skills are totally wrong.
B.5
What factors cause the different performances of two variants?
As mentioned in Section 5, different backbone mod- els prefer different variants of SKILL-KNN. As both two methods aim improve skill-based simi- larity, such performance differences indicate that beyond similarity, some other factors also influence in-context learning. An et al. (2023) and Rajkumar et al. (2022) indicated that the complexity of se- lected examples could be one potential factor. Here, we explore whether the two variants differ in the complexity of the selected examples.
Morcover, during manually checking the qual- ity of generated skills, we surprisingly find that there are some novel descriptions about skills that are not presented in our annotated examples. Ta- ble 11 shows some examples. It indicates that the prompting-based rewriting provide degree of generalization of unannotated skills.
D Detailed Settings Experiments
-onsistency-based variant always leads to more complex in-context examples than Distinetiveness-base variant. Here, we mainly check the complexity of in-context examples for text-to-SQU tasks from two perspectives: (1) the average number of tables, which reflects the complexity of database, and (2) the average length of SQL queries, which reflects the hardness of scarching. As shown in Figure 7, consistency- based variant leads to higher complexity than
In this section, we provide more details about our experimental settings.
D.1 Select Examples for Annotation
In our default setting, consider two principles to select examples for annotating the required skills: 1) ensuring coverage of all logical operations found in the example bank and 2) selecting examples from diverse databases. Specifically, we first find all used logical operations the example bank and

---

Consistency- Based
Comsistency-Base
Delinctionass Sased
60
13
12
Spider
Dr. Spicer DB
Dr. Spicer
Dr. Spider 8CL
KOBQA
Spkder
Dr. Splder
Dr. Spiser NLQ
Dr. Spider SCL
KDBQA
NLQ
D8
(a) Number of Tables
(b) Length of SQL Queries
Figure 7: Two perspectives that reflect the complexity of selected in-context examples from two variants of SKILL-
KNN. (a) The average number of tables contained in the database of each in-context example. (b) The average length SQL queries (split by space).
Table I1: Case study for generated skill-based descriptions in Spider. Texts in purple are novel descriptions that are not presented in our annotations.
Neeil Qjevr
Cenrreld
SQL
daplay
hone overiiiio witla the oage 1030 so 12000 and coris iv cursee wh santee 50.
caabas, wolke
DEL.ETT anall FROM WIBLE conaioion saary WITWEEN AND 13000 Aeparmen
coersies aape ouini se coarzses
deerzeensan
greedily cover these operations in few examples. Then, we randomly select more examples from various databases until there 16 examples.
guage question. Therefore, we add the evidence as part of the context for in-context learning. For
evaluating wny-inuut-basel methods on BIRD, we concatenate the natural language question and the additional evidence compute the embedding. For our SKILL-KNN, we also provide the evidence for rewriting. and we use 12 annotated demonstrations with evidence (shown in Appendix E).
In our ablation study shown Figure to con- strain the operation coverage, we just remove our first selection step; to constrain the database diver- sity, we just select examples from two databases.
D.2 Inference Hyper-Paramette
Since the database schema in BIRD is too large to be fully contained in the context for LLM, we reduce the size of schema through grounding in pre- processing. Specifically, we calculate the embed- ding similarity between the input question (along with the evidence) and each table name and column name. Based on this similarity, we preserve 8 tables each with 16 columns for each schema-questing pair.
During inference, we set the max decoding length to 200, and the sampling temperature to O.
D.3 Input-Output Formats
Figure 8 shows some input-output examples to il- lustrate the data formats in our experiments,
Note that the output format of COGS follows the transformation in An al. (2023) which converts the original long-chain format into a more com- pact function-calling format. Such transforma- tion is similar to the conversion from Lambda cal- culus to FunQL in Geo domain(Zelle and Mooney, 1996; Kate et al., 2005; Zettlemoyer and Collins, 2012). It improves the human readability by omit- ting two types of details in original format: the special marker for definite descriptions and the Skolem constants. Apart from the omitted details, this transformation keeps the main semantics in the domain of COGS, such as semantic roles, modifica- tions, and orders among clauses and modifications.
D.5 Evaluation on COGS
COGS totally contains 24,155 examples in train and 21,000 examples in gen set. To reduce the high computational cost, we sample 2,000 ex- amples from the train set as the example bank for in-context learning, and sample 1,000 examples from two sub-tasks primitive substitution (P.S.) and primitive structural alternation (P.A.) which are de- fined in An al. (2023).
D.6 Target Sketch Matching for SQL
As mentioned in Section 4.2, select in-context examples with target sketch matching (oracle) text-to-SQL. tasks, we calculate the overlap of SQL key words between the example from example
D.4 Evaluation on BIRD
Different from other text-to-SQI SQL tasks, BIRD ad- ditionally provides "evidence" for each natural lan-

---

bank and the labeled SQL query of the test input query. We mainly consider the following SQL key words along with several operations for calcula- tion: SELECT, WHERE, GROUP, HAVING, ORDER, DESC, ASC, LIMIT, JOIN, INTERSECT, EXCEPT, UNION, NOT, IN, OR, AND, BETWEEN, EXISTS, LIKE, DISTINCT, COUNT, AVG, MIN, MAX, SUM, CAST, CASE, WHEN, THEN, ELSE, END, IIF, REAL, FLOAT, NULL, STRFTIME, *, /, =, ?, ,,, +, %. Based on these key words, the target sketch similarity between two SQL queries y and yi is calculated as follows,
sime (ye, 1A) (K) n (W)))),
(7)
in which KW() returns a set of contained key words.
D.7 T-SNE Visualization
For the visualized embedding space in Figure 6, we use Sentence-BERT as the embedding model and take examples from both example bank and dev set in Spider. For SKILL-KNN, we take its consistency-baaee variant. To accelerate the vi- sualization process, just take examples with medium hardness (defined in Yu et al. (2018b)). We use the implementation of t-SNE from the sklearn library We set the learning rate of t-SNE as "auto", init method as "random", and perplexity as 3.
E Annotated Demonstrations
Table 12 lists 16 annotated demonstrations for text- to-SQL tasks and Table lists another 12 an- notated demonstrations with evidence (which is required in BIRD). Table 14 lists 16 annotated demonstrations for COGS. Appendix D.1 intro- duces how we select these examples.
"https://sciiitlllmm.oo////

---

Table 12; Annotated demonstrations for text-to- Sext-to-SQL tasks. All these examples are from the example bank of Spider.
DB ID
Input Query
Annoiated Skill Based Descriptions
I
Show all mujors.
solve this the datalasee we need select distinct values the columnn
Count the number of different colleges that players who play for Columbus Crew from.
To solve this task the datallase, we need to join two tables and count the number
march season
distinct values the column.
What are the hometowwn of gymmasis and the correspondig number gymaasis?
To solve this task in the database, we noed to join two tables, seloct one column.
gymnast
group these selections and count the number of selections each group
Show minimum, maximum, and average market value for all companies.
To solve this task the database, we need to retum the minimum value, the maximum valuc, and the avenge of valuess the column.
gAk company
Show the years, book titles, and publishers for all books, desconding onder year.
To solve this task the database, we need to select three columns and sort them in descending order acconding the values
culture compuny
column.
Which catolng costents have a product stoek mimber
To solve this task the database, we need to select one column and apply constraint format of values this column.
product_caullg
that starts from "2n1 Show the catalng, entry
What are the dates in which the mean sen level pressure between 30.3 and 31?
To solve this task the datailase, we need to select one oolumn and apply constraint that values another columa should certain range.
1
What is the salary and some of the employee who has most number of cerificutess aircrafts with distunce more than 5000?
To solve this task on the database, we noed to join three tables, spyly greater- thin construint, group the selections and calculate the number of each group, sort the selections in descending order, and select the top result.
f
solve this task the database, need calculate the average valuee colum, apply an non-inclusion constraint with another set of selections, which need group the seletions and find which groups have maximum value than the threshuld.
What is the everage longitue of stations that never had bike uvailability than 10?
1
display the information of employees whose salury in the runge of 8000 and 12000 and commission is not null department number does not equal 40.
solve this task the dutabase, need give full information ahout selections, upply an inserval constraint, and upply un optional constraint that two unequal judgments should be satisficd least onc.
hr_1
To solve this task the dutabase, we need to exclude the selections the second set from the first set: the first of selections need apply greater-hhn constraint, and the second set of selctions need to group the soletions and find which groups have a minimum value lower than the chreshold.
What are the ids of stations that have latitude above 37.4 and never bad bike availability below
1
To solve this task the database, we need to retum the union of two set of selections: the first set selections noad join two tables group the selections and find which groups have an average value greater than the threshold, and the secoed set selections need apply constaint the format of values.
What are the names and ids of stations that had more than bikes available on average or were installed in Deaember?
1
To soliee this task the databasc, we need retunt the intersection two set of selections: the first set selcctions need to join two tables, group the selections and find which groups have a number selections greater
Show with least two regions 10 citics affocted.
reaom
or equal the threshold, and the secoed set of selections need to join two tables, group the selections and which groups
sum of values larger or equal the threshold.
1 1
Last the foveramess of all distinet drivers in alphahetical order?
To solve this task the datalasse we need select distinct values one oolumo and sont these selections in ascending order according the selected values.
To solve this task the dutabase, we need apply greater than conseraint on difference between values, group the selections find which groups have number of selections greater than equal to the threshold.
display joh for those johs that were done by or more for more than 300 days.
hr_1
Find names iotal checking savings balances of accounts whose savings balance higher than the average savings balance.
solve this the datahase, we need select one column and values in another two columns, join three tables, and apply greater-than constraint where threshold is the average another of selected values.
small bank

---

For Rewriting:
For Semantic Parsing:
Context:
Context:
Generate the needed skills to solve the task the databuse
Generate SQQ query for the natural language task
echema.
database achema.
allergy type (allergy, allergytype)
department (Department ID, Name, Creation, Ranking,
bas, allergy (staid, allergy)
Budget in Billion,, Numn Employese))
student (stuid, Iname, fname, age, sex, major, advisor, city, coda)
head (bead. name, born, stale, age)
management (dopartment ID, head, ID, temporary, (
Task: Evidence: XEX
majors.
many acting (Optional) ### Evidence: XXx
there?
(Optional) #**
#** Skills: To solve this task the database, need select distinct values in the column.
### SQL query: SELECT coun(DiSTINCT temporary
FROM management
#*# Database schema:
#*# Database achoma:
department (department_5) budget In billions, num, employees)
creation, ranking,
stadium (Stadium, JD, Location, Name, Capacity, Highest, Lowest, Average)
bead (head id, name. born, state, age)
singer (Singer ID, Name, Country, Song Name, Song release year, Age, male) concert (concert ID, concert Name, Theme, Stadium, ID, Year)
management (department
head temporary. acting)
#*# Task: What are the maximum and minimum badget of the departments?
(Optional) H## Evidence: xxx
singer. concert (concert ID, Singer, ID) ### Task: How many singera do we have? (Optional) Evidence:
#1# Skills: To solvve this task in the database, we need to
Completion:
#i# SQL query:
return the minimum value and maximum value column.
Completion:
SELECT colan((*) FROM singer
(a) Formats of Text-to-SQL. Tasks (with Optional Evidence for BIRD)
For Rewriting:
For Semantic Parsing:
Context:
Generate the required alclls to parse the following sentences.
Please generate semantic representations for the following sentences.
#Bil Sentence: Isabella bilod that Elizabeth saur Skill: 'ave' has no object.
Hill Sentence: Emma proved that citizen offered a frog cake the stage
in which
### Somanties: PROVE EMMA NONE, NONE) CCOMP
OFFER ( CIIIZEN ON CAME, STAGE) IROG))
### Sentence: sandwich was fed to a girafle #a# Stell:: This sentence in the passive voice and has prepoeitiona) phrase (i.e., the recipient the verb.
child attempted smirk. (CHILD SMIRK CHILD, NONE, NONE)
#il Sentence:
phrasee) which describes
Semantics:
XCOMP
Sentence:
helped by dog.
Sentence:
was shortened.
#iill Steil::
Hil!!! Semantics:
Completion:
Completion:
This sentence the passive voice und bas prepositional phrase (l.e,, by noun phrase) which describea agent of the verb.
SHORTEN (NONE, BESIDE (COOKIE, TABLE) NONE
(b) Formats COGS
Figure 8: ouputt formats used in our experiments.

---

Table 13: Annotated demonstrations for text-to- SoLt task with evidence. All these examples are from the example bank of BIRD.
D8
Ingu Quary
Evidense
Anmoraad Stall Ouieaad Diseippiee
aapastuve
Pleae lai av three soder that caused a lois the ormqury.
excsed a los to the compary refers to 8
To solve tha lak is the dasikac, ne need to select one colanss. eely line then conernast and retarn fhece sovals
diney
Calcsha pasentage the
DIVIDECCOUNToeee ader where "Daaaa", COUNTrvius percestiges
cdcckac pessntase
Additonally caleala posentase
omsorite cau the liss redl nomber redtllyy wanbor dride by anether
Igpakaze
Arars the kenidau who will and 2000 fawe mayy ard fives the Repuatliuss punt??
the kniidamn who will end in 2009 refine ou ENI 2000; fooms the Hepadkic as parts sdiarn to paaty in Repubiionn
To wakee this sak in the dacirons, we seed ualaa rwo cotarnee enely ss caperakee ocstrare on the ire and an equivalent
ovsoniie the was wiee..
welouryca
Calorker auorage lespts amployowers
crephayae woriing
sterage loagts employrnere ANGiounma yeont
apuiddn Addtiinally
dasiloon join
ger two evengy
apple
Arony the teeagge cleens oho ose Google
teeagger relers to 13 <- 19, Evegle accennt cofers
To wive this task in the dasdeeso we need to comgaee the comterrof
muit conglies
scorant and Mussoo accere wniich groep of chees is now than the other?
mant like "G-ar greal cowr. Micesoot aoconse cofiee emaal like "g-a0 correttoo ove""
virwe is run formets, and oemor the wattr that ths higher samberr Adtmmnallyy on sout to applyy a benwoer and conerates
To uaker this sak in the dasibaar, we need allaa distect volaes
What are eninasittes with the most
most intertatioall sodeens refess MANISMIDDIIDE
Figet intronatio
crxtamn) 10000
one skoams jon two saMes, group resalts, orler sesals descending oxder ceooding to the son valees reuux aottroullyy duilng pitting paosentgze
colarae dirsk
sosls 100.
perceraage WULLIPPLIDIIEESSSS
To wive this task is the dasihose we need compane values is two ooturee and comeet the coinga seis reclt ino procesigg Addtionalyy we ned to jois twe tanko. apply caeo tare us Sermal of valac in one
ACTUAL SLASSED TIME- c 72 CRR, BLAPSED TIKEE)
What the percerage iluino whids landad
COUINTICokkk) 100) lesdal ai refore to DEST:
a Pat lough vare faske thas schokkzedt
Pinabugh refers to Descripiion wlach comsina Punbarhh
oolamn and caaee dast dhe values two coleazaa are not mull Morpovzz to en the peroovagy namthr we reed cet can solies isnn
Owte fron scheadeed totor to
ACTUALL 51 APSED TTMB CRRS 31 APSEDD TIMB:
narshe muliply this sumbher by 1801 and divade by the soal
Beteere 111202 and 41/2013, oever calls rodlr FAAR Miri
sorrage
"2017-01 2207-04-00's
11/2017 60/20/1 Daic
retore deegs
ahtsn from willes ove foverat, need Additceally
wladed
these
dasibacc
sorthess
saherings
roach thas ritre st March Daer: Chereod Saper Kings rofors
Ton satie this sak in the dacirons we reed to apply an cithovor oondrain,, son the aciecaad roalls socendine under and rracm the up cne resal.
When dal Charni Saper Kings play to tine muach?
to Iear, Naree on "Chazza Super Kinnd'
3is sorch refers to wircMircc Dune)
Which ohis "ellber percos"
shlp
shipeaddy PERAIN Litgrrurree
To this
dasibose, we need apply aquality
begerooom Additionall.
To wakee this
the dasilanc need odles one colares and
Closkee srouse of apporsace is porerange
Which ingrddent appeaed the rost is recipec?
COUN INTiaggndiee 143x calottriis MI MUETIII MIDVVIDE
COUN COGNNTIAAoRRooeeeen... All. CoUNTingrvass Alli. 150)
ingrediet oppeaed the most is recipes inters to MASI
cdlcakte wre persevage number join two tilles group tie selected cesats.
of madee ano flaa susbber axaligle thi samber by 100, and diske by
and oin the cesuits is desiending coder acconling to the siee of each group
Adtiinnllyy so cakedaas the peroerage wamber we reed tn-o the cours
tse aacdees ccers.
Table 14: Annotated demonstrations for COGS. All these examples are from the example bank of COGS.
Input Query
Annotated Skill Based Descriptions
Isahella liked that Elizaheth saw.
This sentence cvntains clause in which the verh 'saw' has object.
This sentence in the passive voice and has prepositional phrase f.e. 'to noun phrase") ) which descrihes the recipient of the verb.
A sandwich was fod to a giraffe.
Benjamin froze.
The verb 'froze' has no object.
This sentence is in the passive voice, has un object and has a prepositional phrase (i...' 'by uoun phrase') which describes the agent of the verb.
Sophia was given cookie by Emma.
Sophia liked a box on the cake
This sentence has a single object with a modification phrase.
This sentence has a direct object with a modification phrase and bas prepositional phrase (i.e.' 'to noun phrase") " which describes the recipient of th verb.
Fimma sold the drink beside road to zehra
A lion atc. :
The verb 'ate' has no object.
Eleanor was offered the ball.
This sentence in the passive voice and has an object.
A box was helped.
This sentence in the passive voice and has no object or prepositional phrase.
The fish dreamed to walk.
This sentence has an infinitive verb.
A cat lended a lawyer the cake.
This sentence has an indirect object and a direct object.
This sentence is in the passive voice and has two prepositional phrases: the first one fi.a. 'to noun phrase") describes the recipient describes the agent of the verb.
The hasket was handed to cat by Emma.
the second one fi.e., "by noun phrase")
Sofia thought that pancake rolled.
This scutence contains clause which the verb 'rolled' bas no object.
Liam hoped that the boy wanted to dance.
This sentence contains clause that has infinitive verb.
The cat gave Ethan a rose the table.
This scutence has indirect object and direct object with modification phrase.
The duke was passed shell table in the house by Emmaa.
This seutence in the passive voice. object with nested modification phrase.
and has prepositional plrase (i..., "by noun phrase") which deseribes the agent of the verb.


Reviews:
[{"review_id": "VTDNVyNuPp", "reviewer": "Reviewer_CEQW", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable questions and for recognizing the insights our work contributes!\nWe hope that our responses below have addressed your concerns and elucidated some misunderstandings.\n\n> W1: The authors designed two variants to identify the consistency and distinctiveness properties of candidate examples, of which the efficacy should be questioned.\n>\n> R1: There is a misunderstanding about the purpose of two variants:\n> they are designed to **alleviate the influence from prompt-order sensitivity in our LLM-based rewriting process**, rather than identifying some properties behind in-context examples.\n> In other words, we did not optimize the consistency and distinctiveness properties of in-context examples, but rather designed two approaches for aggregating samples of skill-based descriptions.\n>\n> Specifically, to alleviate the prompt-order sensitivity, we sampled several skill-based descriptions by changing the order of demonstrations, \n> then aggregated these samples from the view of consistency (i.e., averaging these samples) or distinctiveness (i.e., taking one sample that is most similar to the skill-based description for the other in-context example).\n\n> W2: There should be more experiments: using ChatGPT or GPT4, reproducing more fine-tuning-based methods, and evaluating more tasks beyond semantic parsing.\n> \n> R2: We have expanded our experimental results as follows and provided defenses for the adequacy of the initial experiments in our paper:\n> * **Skill-KNN can benefit ChatGPT and GPT4**:\n> It is importent to note that the four LLMs used in our main experiments belong to the GPT-3.5 and Codex series and are among the most powerful models available.\n> Consequently, we believe our main experimental results sufficiently demonstrate the effectiveness of Skill-KNN.\n> To further address your concerns, we have evaluated Skill-KNN with gpt-35-turbo and gpt-4 (Dataset: Spider, Variant: base), further demonstrating that Skill-KNN can benefit pinnacle LLMs.\n> \n>    | LLM               |Method           | Exec. Acc. (%) |\n>    |-------------------|----------------|--------------|\n>    | gpt-4             |Random           | 76.1           |\n>    | gpt-4             |KNN [9]| 76.7           |\n>    | gpt-4             |Din-SQL [1]| 74.2           |\n>    | gpt-4             |Skill-KNN (ours) | 81.3           |\n>    | gpt-35-turbo      |Random| 74.3           |\n>    | gpt-35-turbo      |KNN [9]| 73.7           |\n>    | gpt-35-turbo      |Skill-KNN (ours)| 76.2           |\n> * **Skill-KNN can perform comparably or even better than fine-tuning-based CEIL [3] and TST [4]**:\n> As our Skill-KNN is a training-free method, we initially compared it with a standard fine-tuning-based method (i.e., EPR [2]).\n> To further address your concerns, we reproduce two more fine-tuning-based methods, CEIL [3] and TST [4] (Dataset: Spider, LLM: text-chat-davinci-002, Variant: base).\n> These results further support that Skill-KNN can perform comparably or even better than fine-tuning-based methods.\n> \n>    | Method                   | Exec. Acc. (%) |\n>    |--------------------------|------------------|\n>    | EPR (w/ fine-tuning) [2] | 74.4 |\n>    | CEIL (w/ fine-tuning) [3]   | 75.0 |\n>    | TST (w/ fine-tuning) [4]   | 76.3 |\n>    | Skill-KNN (w/o fine-tuning, ours) | 76.8 |\n> * **Skill-KNN can be effective in the math reasoning task GSM8K [5]**:\n> Our study primarily evaluated the effectiveness of Skill-KNN in the semantic parsing/code generation field, an important area in NLP research (note that we selected the parsing track as the Submission Track 2).\n> To further examine the generalizability of Skill-KNN beyond semantic parsing, we have applied it to a challenging math reasoning task, GSM8K [5] (LLM: text-chat-davinci-002, Variant: base).\n> These results evidence that Skill-KNN can also be effective in tasks beyond semantic parsing.\n>\n>    | Method                   | Acc. (%) |\n>    |---|---|\n>    | Random | 69.1 |\n>    | KNN [9] | 69.9 |\n>    | Skill-KNN | 71.0 |\n\n> W3: The concept of training-free example selection is not novel.\n> There should be further discussion and comparison with existing methods.\n> \n> R3: We would like to clarify the novelty of this work: we are the first to **raise the research question of how to make training-free selection methods better fit the desired skills in downstream tasks**,\n> and we are also the first to **propose a rewriting-based solution following this research question**.\n> \n> Existing training-free few-shot selection methods can be generally categorized into two directions.\n> * Static selection (e.g., classic best-of-5 selection [6], k-center [7], and influence-based method [8]) which chooses one set of examples from a large bank to serve all inputs,\n> * Dynamic selection (e.g., KNN [9], DDP [10], and MMR [11]) which dynamically choose examples for different inputs.\n> \n> As our Skill-KNN is a dynamic selection method, our paper primarily compares it with strong baselines in dynamic selection methods,\n> such as KNN + OpenAI Embedding, representing pure similarity-based selections,\n> and MMR, representing diversity-based selections.\n> Below, we provide additional comparisons with both static and dynamic methods (Dataset: Spider, LLM: text-chat-davinci-002, Variant: base).\n> These results further demonstrate that we have compared with existing sota baselines in our paper.\n>    | Method                   | Exec. Acc. (%) |\n>    | --- | --- |\n>    | Random | 72.9 |\n>    | Best-of-5 (static) [6] | 73.7 |\n>    | K-Center (static) [7] | 73.4 |\n>    | Influence (static) [8] | 73.4 |\n>    | KNN (dynamic) [9] | 74.0 |\n>    | DDP (dynamic) [10] | 74.6 |\n>    | MMR (dynamic) [11] | 74.6 |\n>    | Skill-KNN (dynamic, ours) | 76.8 |\n\nReferences\n\n[1] Pourreza, Mohammadreza, and Davood Rafiei. \"Din-sql: Decomposed in-context learning of text-to-sql with self-correction.\" arXiv preprint arXiv:2304.11015 (2023).\n\n[2] Rubin, Ohad, Jonathan Herzig, and Jonathan Berant. \"Learning To Retrieve Prompts for In-Context Learning.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n[3] Ye, Jiacheng, et al. \"Compositional exemplars for in-context learning.\" arXiv preprint arXiv:2302.05698 (2023).\n\n[4] Poesia, Gabriel, et al. \"Synchromesh: Reliable Code Generation from Pre-trained Language Models.\" International Conference on Learning Representations. 2021.\n\n[5] Cobbe, Karl, et al. \"Training verifiers to solve math word problems.\" arXiv preprint arXiv:2110.14168 (2021).\n\n[6] Nakano, Reiichiro, et al. \"Webgpt: Browser-assisted question-answering with human feedback.\" arXiv preprint arXiv:2112.09332 (2021).\n\n[7] Sener, Ozan, and Silvio Savarese. \"Active Learning for Convolutional Neural Networks: A Core-Set Approach.\" International Conference on Learning Representations. 2018.\n\n[8] Nguyen, Tai, and Eric Wong. \"In-context Example Selection with Influences.\" arXiv preprint arXiv:2302.11042 (2023).\n\n[9] Liu, Jiachang, et al. \"What Makes Good In-Context Examples for GPT-3?.\" Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. 2022.\n\n[10] Chen, Laming, Guoxin Zhang, and Eric Zhou. \"Fast greedy map inference for determinantal point process to improve recommendation diversity.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[11] Ye, Xi, et al. \"Complementary explanations for effective in-context learning.\" arXiv preprint arXiv:2211.13892 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation is clear and the idea of allowing the model to induce skills to solve problems looks like sound. \n2. The proposed method does not require the training or fine-tuning of prompt selection method, making it more lightweight and generalizable.\n3. Experimental evaluation on several semantic parsing tasks with different large language models demonstate the potential effectivenss of the proposed methods.", "reasons_to_reject": "1. The authors mentioned that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; However, they still adopt the embedding to identify the consistency and distinctiveness of candidate examples and make final selections. It seems contradicted to the claim. Furthermore, I question the efficacy of this embedding-based matching approach in genuinely capturing the desired properties of consistency and distinctiveness especially using the pretrained embedding models. A more comprehensive analysis is warranted to clarify this matter.\n\n2. The experimental results presented lack robustness. Specifically, the evaluation is limited to a select number of semantic parsing tasks, which does not comprehensively demonstrate the enhanced in-context learning capabilities of large language models using the suggested method. Furthermore, for the fine-tuning-based baselines, the authors exclusively focus on the EPR approach, neglecting to consider recent works like CEIL [1]. Additionally, these large language models cannot be categorized as the pinnacle of model architectures. It would be highly beneficial to see evaluations involving state-of-the-art large language models, such as ChatGPT or GPT-4.\"\n\n3. The concept of training-free prompt selection is not novel. Several works have already explored this approach. It would be advantageous if the authors could provide an exhaustive discussion and empirical comparison of these existing methods.\n\n\n[1] Compositional Exemplars for In-context Learning", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "This paper proposed a prompt selection method called \"SKILL-KNN\" that extensively utilizes the in-context learning abilities of  Large Language Models (LLM) to  eliminate training of prompt retriever.   This is accomplished by initially enabling the LLM to deduce the skill analysis and generation from raw inputs, using a limited number of annotated instances. Subsequently, the model inputs the skill representations into a readily available embedding model to identify the most similar examples from the example bank.  Moreover, the authors present two extensions of \"SKILL-KNN\", considering both consistency and distinctiveness of candidate examples.  Experimental results  on several semantic parsing benchmarks demonstrate that the proposed prompt selection method can boost the in-context learning abilities of large language models compared on baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qDeISUanZP", "reviewer": "Reviewer_zEH1", "summary": "", "questions": "A. In line 003, \"Few-shot selection-selecting appropriate examples for each test instance separately—is important for in-context learning.\" Please give a detailed explanation.\n\nB. In line 097, \"We evaluate SKILL-KNN on five challenging semantic parsing tasks: \".\nThere are five datasets on one task: text-to-SQL.\n\nC. In line 123: \"3) our comprehensive experiments across various semantic parsing tasks and backbone models demonstrate the effectiveness of SKILL-KNN, and our analysis of annotated demonstrations provides further insights for better utilization of SKILL-KNN\".\nIn my opinion, the experiments are not your contribution.\n\nD. In line 194, \" Note that we annotated skills with natural language descriptions rather than rule-based constraints. Thus, these skills can better suit the off-the-shelf embedding model, and our annotated demonstrations can better generalize to unseen data.\" \nWhy do the annotated demonstrations can better generalize to unseen data when annotated skills with natural language descriptions?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and valuable questions!\nWe hope that our responses below have addressed your questions.\n\n> Q1: Explain why few-shot selection is important for in-context learning.\n> \n> A1: Few-shot selection is crucial for in-context learning due to the observation that **the performance of in-context learning can be highly variable depending on the choice of prompts** [1, 2].\n> This implies that for a given test case, in-context learning may perform well with certain few-shot examples, while failing with others.\n> As a result, identifying the appropriate few-shot examples to enhance in-context learning performance is a significant research question.\n> \n> Furthermore, numerous studies have demonstrated that **different test cases may prefer distinct few-shot examples** [3, 4, 5, 6].\n> Consequently, we define the few-shot selection as the process of \"selecting appropriate examples for each test instance separately.\"\n\n> Q2: This work evaluates five semantic parsing datasets rather than five tasks.\n> \n> A2: Thank you for bringing this to our attention and we will revise our wording accordingly.\n> Specifically, we will clarify that our work evaluates SKILL-KNN on five semantic parsing datasets, with four of these datasets belonging to the text-to-SQL task, and the COGS dataset pertaining to the thematic role parsing task.\n> \n> Moreover, beyond semantic parsing, we have also applied Skill-KNN to a challenging math reasoning task, GSM8K [5] (LLM: text-chat-davinci-002, Variant: base).\n> The following results evidence that Skill-KNN can also be effective in tasks beyond semantic parsing.\n>\n>    | Method                   | Acc. (%) |\n>    |---|---|\n>    | Random | 69.1 |\n>    | KNN | 69.9 |\n>    | Skill-KNN | 71.0 |\n\n> Q3: Experiments cannot be regarded as a contribution in the work.\n> \n> A3: We respectfully disagree with this perspective.\n> In a methodology-focused paper, the contributions should encompass both 1) **the novelty and insights behind proposing the method**,\n> and 2) **the comprehensive experiments that evaluate the effectiveness of the proposed method**.\n> As such, we believe it is appropriate to emphasize our experimental contributions within our paper.\n\n> Q4: Further clarification about the statement in line 194:\n> \"... we annotated skills with natural language descriptions rather than rule-based constraints. Thus, these skills can better suit the off-the-shelf embedding model, and our annotated demonstrations can better generalize to unseen data.\"\n> \n> A4: It is important to note that **off-the-shelf embedding models are primarily pre-trained on natural language (NL) data** and may not be well-suited for handling specifically designed structural constraints.\n> By annotating skills with NL descriptions, we can better align with the off-the-shelf embedding models, which in turn allows us to leverage their generalizability when encoding unannotated NL descriptions more effectively.\n\nReferences\n\n[1] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n[2] Zhao, Zihao, et al. \"Calibrate before use: Improving few-shot performance of language models.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Liu, Jiachang, et al. \"What Makes Good In-Context Examples for GPT-3?.\" Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures. 2022.\n\n[4] \"Learning To Retrieve Prompts for In-Context Learning.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n[5] Poesia, Gabriel, et al. \"Synchromesh: Reliable Code Generation from Pre-trained Language Models.\" International Conference on Learning Representations. 2021.\n\n[6] Zhang, Yiming, Shi Feng, and Chenhao Tan. \"Active Example Selection for In-Context Learning.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  The proposed method addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task.\n2. The proposed method does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. ", "reasons_to_reject": "The proposed method mainly relies on manually annotated skill-based representations to rewrite the input query, it is difficult to transfer it directly to other domains.", "questions_to_authors": "A. In line 003, \"Few-shot selection-selecting appropriate examples for each test instance separately—is important for in-context learning.\" Please give a detailed explanation.\n\nB. In line 097, \"We evaluate SKILL-KNN on five challenging semantic parsing tasks: \".\nThere are five datasets on one task: text-to-SQL.\n\nC. In line 123: \"3) our comprehensive experiments across various semantic parsing tasks and backbone models demonstrate the effectiveness of SKILL-KNN, and our analysis of annotated demonstrations provides further insights for better utilization of SKILL-KNN\".\nIn my opinion, the experiments are not your contribution.\n\nD. In line 194, \" Note that we annotated skills with natural language descriptions rather than rule-based constraints. Thus, these skills can better suit the off-the-shelf embedding model, and our annotated demonstrations can better generalize to unseen data.\" \nWhy do the annotated demonstrations can better generalize to unseen data when annotated skills with natural language descriptions?", "paper_topic_and_main_contributions": "Paper Topic：\nThis paper proposes a skill-based few-shot selection method for in-context learning and applies it in text-to-SQL task.\n\nMain Contributions:\n1. They propose a skill-based few-shot selection method SKILL-KNN, which leverages the power of prompting LLMs to generate skill-based representations.\n2. They provide some annotated skill-based representations for 16 examples of text-to-SQL task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vc7JL4dHAS", "reviewer": "Reviewer_khfp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review and encouraging feedback!\nWe hope that we have addressed your questions and concerns adequately below.\n\n> Q1: The purpose of Database Schema.\n> \n> A1: It serves to **eliminate ambiguity when generating skill descriptions for text2sql tasks**,\n> because a single natural language (NL) query may require different skills depending on the specific database structure. \n> \n> For example, consider the NL query “show me the student names who are younger than 12”.\n> If the database stores “student name” and “student age” in separate tables, a “join tables” skill would be necessary to search for the requested information.\n> Conversely, if both “student name” and “student age” are located within the same table, there would be no need to employ the join skill.\n\n> Q2: How many times should we change the order of demonstrations?\n> \n> A2: In our main experiments, we set *m*=5 times by default,\n> and **there is a balance between achieving optimal performance and minimizing computational costs**.\n> During our initial exploration, we had experimented with *m*=3/5/7/9 (Dataset: Spider, LLM: code-cushman-002, variant: consistency).\n> \n>    | *m* | 3 | 5 | 7 | 9 |\n>    |---|---|---|---|---|\n>    | Exec. Acc. (%) | 72.8 | 74.7 | 75.1 | 75.0 |\n> \n> As these results suggested that performance improves marginally when *m*>5, we decided to set *m*=5.\n\n> Q3: The motivation behind two variants and analysis.\n> \n> A3: This is a great point.\n> The motivation behind the two variants stems from our consideration of disturbances from the prompt-order sensitivity as additive noises to the ground-truth skills during the rewriting process.\n> The two variants are designed to address two types of noise:\n> \n> * **Zero-mean white noise** which frequently occurs in results and originates from a zero-mean distribution (e.g., zero-mean Gaussian distribution).\n> We assume its magnitude is relatively small compared to the ground truth. Zero-mean white noise can cause the loss or redundancy of partial information in the ground truth.\n> * **Spike noise** which occasionally occurs in results and has a much larger magnitude than the ground truth.\n> It strongly influences the information in the ground truth and causes outliers. \n> \n> **The consistency-based variant is more effective at addressing zero-mean white noise**,\n> as the averaging operation reduces the variance of zero-mean noise.\n> **The distinctiveness-based variant is better suited for handling spike noise**,\n> as it mitigates the influence of outliers.\n> The final results in our Table 2 indicate that both types of noise occur in our LLM-based rewriting,\n> as evidenced by the close win times of the two variants (19 vs 15).\n> \n> To further support that two variants are better at tackling two different types of noise,\n> we conduct additional analysis from two perspectives.\n> \n> * **Evaluation of selection performance.**\n> We examined the performance of each variant in selecting better examples from the example bank under the two noise patterns.\n> We construct some synthetic data for automated evaluation:\n> we take 1,000 unique embeddings of skill-based descriptions as ground truth,\n> then we add noise on each ground-truth embedding to construct the sample set,\n> and finally we assess whether each sample set could correctly select the original ground-truth embedding.\n> The accuracies are shown as follows.\n> \n>    | | Zero-Mean White Noise | Spike Noise |\n>    |---|---|---|\n>    | Consistency | 95.4% | 81.9% |\n>    | Distinctiveness | 90.4% | 87.9% |\n> \n> * **Human evaluation of rewriting.**\n> In Spider dev set, we first identified examples where one variant can always succeed with different LLMs while the other variant always failed.\n> Then, we check the frequency of spike noise occurrences in these examples through human evaluation.\n> We checked 23 examples and found that:\n> for examples where the consistency-based variant wins, the spike noise occurs in 0.4 sample/set;\n> for examples that distinctiveness-based variant wins, the spike noise occurs in 1.6 sample/set. \n> \n> The results of the above analysis further evidence that the consistency-based variant performs better than the distinctiveness-based variant under the zero-mean white noise but performs worse under the spike noise.\n\n> Q4: Revise some terms.\n> \n> A4: Thank you for pointing out the potential confusion regarding the terms used in our paper.\n> We will make the following changes in our revision to ensure clarity for our readers:\n> * We will replace the term \"skill-based representation\" with \"skill-based description\" to avoid the confusion with continuous representations.\n> * We will use the term \"rewriting model\" to refer to the LLM that generates the skill-based descriptions, and we will use the term \"embedding model\" to refer to the model that outputs a continuous embedding vector with respect to a given input.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting solutions for few-shot in-context sample selection for the task of semantic parsing and strong analysis.\n2. Detailed introduction and good writing.\n3. Good results.", "reasons_to_reject": "1. How is the “Database Schema”  col in Table 1 used for what purpose?\n2. In line 223, by changing the order of demonstrations we can generate several candidates and get a population of candidates w.r.t a given data sample in the example bank, how many times should we conduct this process and ablation studies can be available.\n3. The motivation of the proposed two variants in sample similarity measurement is not clear, experimental analysis about in what scenarios a consistency-based variant is better than the distinctiveness-based would be interesting.\n4. Not a big problem but there are some terms making the audiences confused. For example, an embedding model is a LLM to get a representation w.r.t an input; skill-based representations sound like a continuous representation instead of raw texts. \n\nI will raise my score if some concerns have been solved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a prompting-based rewriting method that sheds light on the skill-aware information for better few-shot in-context sample selection in the scenario of semantic parsing. The authors come up with an effective and reasonable method to alleviate LLMs’ sensitivities to in-context samples’ order with two solutions and proves that they are more appropriate than the other through experiments. With enough experiments and analysis, the proposed method is a contribution to LLMs-based semantic parsing.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a prompt selection method called “SKILL-KNN” that uses in-context learning ability of LLMs and power of prompting LLMs to generate skill based representations. 

Pros:

Motivation and writing is clear
 

Proposed method does not require training/finetuning
 
Experimental results demonstrate that potential effectiveness of the method 

Cons:

Better explanations as to why their proposed embedding methods (based on pretrained embeddings) are not biased by surface natural language features that are unimportant for the target task. 

Reliance on manually annotated demonstrations to prompt a frozen LLM in order to generate skill-based representations - The authos need to provide some more explanations as to how cognitively loaded this task can be and how easy it is to generalize across domains 

The original work will also benefit from incorporating the additional discussion points/experiments conducted during rebuttal.
Decision: Accept-Main