Fairness-guided Few-shot Prompting for Large Language Models
Authors: Huan Ma
Changqing Zhang
Yatao Bian
Lemao Liu
Zhirui Zhang
Peilin Zhao
Shu Zhang
Huazhu Fu
Qinghua Hu
Bingzhe Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: large language models
prompts
classification

Abstract: Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper,  we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner.

Fairness-guided guided Few-shot Prompting for Large Language Models
Huan Ma Changqing Zhang'; Yatao Bian', Lemao Liu', Zhirui Zhang",
Peilin Zhao?, Shu Zhang', Huazhu Fu', Qinghua Hu', Bingzhe 2u College of Intelligence and Computing. Tianjin University, Tianjin, China
2 Al Lab, Tencent, Shenzhen, China Institute of High Performance Computing (IHPC),
Agency for Science, Technology and Research (A STAR), Singapore
zhanchangginget ju. cn lbingzhewu@tencent comm
Abstract
Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by few input-output examples. However, prior research has shown that in-context learning stability due to variations in training examples, example order, and prompt formats. Therefore, the construction of appropriate prompt is essential for improving the performance of in-context learning. In this paper, we revisit this problem from the view of predictive bias. Specifically, we introduce metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-on state-of-the of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's 's in-context learning performance in an effective and interpretable manner. Code is available https://github..com Hoanaa
suffer from high in-
Introduction
Large language models (L..Ms,, such as GPT-3 1 and BLOOM 2 have demonstrated remarkable ability in performing in-context learning (ICL) downstream tasks. ICL refers to the process of conditioning an LLM to solve various downstream tasks using prompts constructed from a few demonstration input-output pairs (3 (i.e., few-shot prompting). Despite its impressive performance, prior research has shown that ICL suffers from high instability due to variations in the choice of in-context demonstrations, demonstration order, and prompt formats an appropriate prompt has been identified as critical factor for improving the performance of ICL
[) Therefore, constructing
(6).
Previous research studies this problem typically from two directions: (1) prompt tuning in the
embedding space [7 10 [1]. (2) prompt searching in the text space 4 12 13
[6]. The key idea of prompt tuning is to inject task-specific embedding into hidden layers and then tune these embeddings using gradient-based optimization [5). However, these methods require modify the original inference process of the model, which is impractical for the case of black-box LM services
15
"Coresspodiigg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

such as GPT3 and ChatGPT [7]. Furthermore, prompt tuning introduces additional computational and storage costs, which is typically expensive for LLM. A more feasible and efficient way is to optimize prompting via searching approximate demonstration samples and ordering in the original text space [ [5]. Bunch of works are presented to constructs prompts from either "global" or "local" views. On the one hand, global-view based methods typically optimize the different elements of the prompt as a whole, with the aim of achieving superior performance. For example, one approach, as described in (14, constructs a search procedure that leverages the overall diversity of demonstrations. Another approach 4 attempts to optimize the ordering of the entire set of demonstrations to achieve better performance. In contrast to the global view, local-view based methods optimize each individual demonstration by designing different heuristic selection criteria such as prior work KATE [15]. These methods have achieved impressive improvements of them still suffer from the following limitations: (1) Most of current research mainly focuses on searching prompts along a single dimension, such as example selection or order. However, the overall influence of various dimensions on the performance remains unclear. (2) These methods are typically based on heuristic criteria, and there is gap between them and actual performance. A unified view that explains how these methods work is needed. (3) More importantly, existing methods optimize prompts globally or locally, which may lead to suboptimal performance.
wide range of tasks. However, most
In this paper, we revisit this problem from the perspective of predictive bias. We find a key insight that the quality of given prompt depends its inherent bias. Based this insight, we propose a surrogate metric based on predictive bias for evaluating the quality of prompts. This metric allows us to evaluate prompt in single forward process without additional development set. Specifically, we apply a given prompt to a "content-free" input and expect the model output an uniform predictive distribution (a content-free input contains no useful information). Therefore, we employ the uniformity of the predictive distribution to characterize the bias of give prompt. This shares a similar idea to the prior work which uses this metric calibrate the model output [18]. In contrast to this work which mainly focus on using this metric for calibration when the prompt is fixed, we further explore its usage in automatically searching an approximate prompt. Moreover, through extensive experiments, we empirically validate the correlation between the inherent bias of a given prompt and its quality measured by the average task performance on a given test set (see Fig. 2).
Moreover, this bias- based metric allows us to build prompting optimization techniques in a "local-to- global" manner. We present two novel strategies for efficiently searching high-quality prompts in bias-guided way: (1) T-fair-Prompting (2) G-fair-Prompting. We focus on a general setting where a labeled set with size N is given. The goal of our strategies is perform combinatorial optimization over this set to find near-optimal prompts (i.e., select demonstrations and their orders). Specifically, T-fair-Prompting uses an intuitive way that first computes the bias of each single demonstration (i.e., one-shot prompting) and then select the top-k fair demonstrations to form the final prompts. This strategy can be efficiently done with a complexity of O(N). Note that T-fair- Prair-Prommtiig is based on the assumption that the optimal prompt is usually constructed from demonstrations with the smallest individual bias. However, this may not hold true in real situations and often leads to sub-optimal solutions. Therefore, we further introduce G-fair- Promptiromtgggg to improve the search quality. f--fir-Prompiie follows the normal procedure of the greedy search which finds the optimal solution by making locally optimal choices at each step. At each step of the algorithm, the selected demonstration is the one which makes the updated prompts achieves the best fairness score. This strategy trades off the quality of the search with the worst-case time complexity. By accepting higher worst-case time complexity of ONN)), the search quality is significantly improved. Note that C-fair-Prompting works from a local to global perspective, wherein bias of individual samples are considered in the early stages while the later stage focus on the reduction of global predictive bias.
To evaluate the effectiveness of our strategies, we conduct extensive experiments with current mainstream models, such as GPT-3 (.] on various downstream tasks. Our results indicate that our method can significantly enhance the model's 's in-context learning performance in an effective and interpretable manner. The overall contribution is summarized as follows:
We introduce to use the predictive bias to assess the quality of a given prompt in an efficient and development set independent way and the empirical effectiveness of this metric is comprehensively validated.
Based on the above idea, we propose two efficient and effective strategies, namely, T-fair- Prompting and G- G-fair-Promptie Prompting to optimize the prompts.
2

---

Selection
(b) Selection (cal)
(c)
Permutation
(d) Permutation (cal)
Figure I: ICL suffers from high instability due to high variations in demonstrations selection and order, even when post calibration is performed.
The effectiveness of these two strategies are validated on various LLMs ranging from GPT-series models to LLaMA family [19] released by Meta recently. Consistent relative improvements of over 10% have been observed over different downstream tasks in contrast SOTA methods.
Relation to calirration-before-sss Our paper shares a similar metric with cal-before-use 18 to asses the predictive bias of a given prompt. However, the prior approach aims to use this metric to calibrate the output, which can be still easily affected by the quality of the used prompt (more results be found in Table In contrast, space to improve the model's 's performance, without requiring any post-adjustment to the output of the model. Moreover, we have firstly empirically validated the connection between predictive bias and the final task performance as shown in Fig. 2 which has not been studied in [18]. Through experiments, we have discovered that, even without calibration, the prompt selected by our method can outperform a randomly selected prompt with calibration.
research aims to find near-optimal prompt the original
2 Related Work
In-context Learning Previous research, cited in [ 20). has demonstrated that Large Language Models can complete tasks with zero- or few-shot learning using in-context learning. LLMs perform well with an appropriate prompt. However, recent works 4 18| has shown that the performance of LLMs is affected by the prompt used. Therefore, determining the optimal prompt is crucial and fundamental research area.
Original space searching A more intuitive approach for determining the best prompt is to search in the original space by selecting or reordering the prompt sentences entered by users. The searching can be concluded in two perspective. Global view: A naive strategy is to enumerate all candidates to find the prompt that can achieve the best performance on validation set, but this strategy is computationally expensive since its complexity is Σm=1 C. kt considering to demonstration selection and order permutation, where & represents the number of demonstrations selected, and C signifies the combinatorial function. Zhang et al. [12] find that errors frequently fall into the same cluster, where each cluster contains similar questions, so they proposed ddiversiry-guide searching strategy to select diverse demonstrations. In addition to demonstrations selection, Lu et al. [4) have identified
the impact of the prompt order on the results. They found the best sequence which yields the most diverse prediction results on the probing set by generating probing set through LLMs. However, this method is also computationally expensive, and it may be difficult to ensure that the generated probing set is sufficiently balanced. Local view: Previous studies [33 show that reducing the model's ' uncertainty helps improve the model's performance, and [4] propose Active Prompting to select demonstrations according to the uncertainty of LLMs. KATE [5] selects the prompt based on the distance amongst embeddings, with the goal of selecting the closest example. However, this method ignores the influence of the order of the examples and requires access to sentence embeddings. 16 demonstrate that LLMs can be easily distracted by irrelevant context, accordingly they identify several approaches for filtering out irrelevant information in context.
In the realm of original space searching, most of the current methods tend to focus solely on the influence of a singular factor (highlighted above) on performance, utilizing heuristic metrics to select context demonstrations that perform well according to this criterion. While these investigations certainly bring benefits to the community, they lack comprehensive consideration of both local and
3

---

10
Aconcyy
Accurecy
20
Onall Anmonc 6i Poreli
65-
Sorted by Falmess
Sorted Faimess
Sorted by Fairnsss
AGNews (BLOOM 176B)
(b) AGNews (LLaMA 13B)
(c) AGNews (LLaMA 65B)
20 Ceciegt--
20
Zacon Decipt
Serted by falmess
Sorted by Fairness
Sorted by Falmnss
(d) TREC (BLOOM 176B)
(e) TREC (LLaMA 13B)
( TREC (LLaMA 65B)
Figure 2: Accuracy is highly consistency with faimess and greedy search can find good prompt, where "Random" " and "Oracle" indicate the average accuracy of all prompts and the upper-bound performance according to fairness.
global perspectives. The method proposed offers metric select context demonstrations from the perspective of predictive bias, which naturally facilitates transition from local view to global view.
3 Revisiting the Sensitivity across Demonstrations
In this section, we will clarify the notations and the templates used in this paper. Then, will demonstrate some brief empirical results to show how different demonstration construction factors (e.g., example selection and order) affect performance. We further introduce the definition of predictive bias/fairness of a given prompt and show its connection to the predictive performance on different downstream tasks.
3.1 Notations
We consider a training set consisting of N samples S = {(x,, y where T; is the sen- tence and y/ E y is the label of the 7th training sample, and y is the space of all labels for the task. We use template I'(-) to transform these sentences and labels into natural language space (i.e., prompt construction). Take an instance from the AGNews dataset [21 for exam- ple, we have
"Cubans Risking Life for Lure of America.", ", Y/ = "World". and T(x;, Ms) is "Article: Cubans Risking Life for Lure of America. Answer: World". We concatenate these demon- strations to form prompt p, which by default is p T(œ1, V1) @ ... ① I( Vn). where @ indicates sentences combination option. At test time, we append the prompt p with T = "Article: <test sentence>. Answer: and feed it to large language model M. The predicted class given by:
M(yle arg arg plypp r), p(v)pTT) M(ylpsT)
(I)
where M(ylp 7) indicates the probability predicted by LLM, and the probability normalized to fit the task. We denote the predictive distribution by P(x) == {p(v| @ r)ly E )/. In this paper, focus on evaluating the instability caused by demonstrations, and fix the prompt template following prior work 18].
3.2 Stability of Few-shot Prompting
As demonstrated by prior research, the few-shot prompting technique is highly susceptible to variety of factors, including the selection and order of demonstrations [4] [8 In this study, we delve deeper into the stability of sho-shot prompting, specifically focusing on the recently released LLaMA family

---

by Meta [9]. Additionally, we evaluate the stability of LLaMA models calibrated using the current Sfate-of-the-arr method [2 T5I.
To elucidate the impact of demonstration selection, we select four demonstrations for each different seed and randomly sample order for each combination. Subsequently, we present the performance on AGNews in the form of a boxplot, which displays the data distribution based on a five-number summary (minimum, first quartile [QI], median, third quartile [Q3], and maximum). As shown in Fig [a))) the accuracy demonstrates significant variability across various demonstrations. The detailed settings please refer to Appendix A.7
To investigate the inftuence of permutations, we examine all possible permutations of four fixed demonstrations, resulting in 4! distinct candidates. (d) also reveals high degree of variance. While post-calibration contributes to mitigating instability, it is essential to note that the model remains sensitive even after post-calibration. This finding underscores the importance of meticulous demonstration selection. In subsequent experiments, we discover that our approach can be employed further enhance the performance of the calibrated model.
3.3 Predictive Bias of ICL
As demonstrated in the preceding discussion, the performance of ICL is significantly impacted by various factors such as demonstration, permutation, and selection (refer to Appendix [A4 for additional information). Consequently, devising an efficient method for constructing an appropriate prompt with near-optimal performance is crucial step in deploying LLMs for diverse downstream tasks. As outlined the introduction, numerous studies aim to optimize prompts in ICL. This paper further investigates this issue through the lens of predictive bias, which refers to the discrepancy between targeted classes.
To achieve this, we initially introduce an efficient technique to assess the inherent predictive bias of a given prompt, drawing inspiration from previous work [8]. We construct a training set-independent metric to measure predictive bias as follows: first, we merge the provided prompt with semannticfree test sample information (e.g., "[N/A]", denoted by 77) and obtain the LLM's predictive distribution for this sample. Ideally, the predictive distribution should closely resemble uniform distribution, as the test sample lacks semantic information. In this paper, we employ entropy as measure of predictive bias, defined as:
fair(p) = Σ plyle @ n) log log BEY
Previous studies have utilized this metric to calibrate the model's 's output. In this paper, we conduct comprehensive examination of the relationship between predictive bias and overall performance. Specifically, in a scenario with four training samples (due to the time-consuming nature of enumerat- ing all prompt cases for a larger number), demonstrations for various datasets and LLMs. Subsequently, we arrange all candidates in descending order based fairness, where an "index 0" denotes the prompt with the highest fairness. We perform experiments using five different seeds, resulting in training sets comprising distinct demonstrations while maintaining the test samples with seed O. Fig. 2 displays the results for different models, revealing strong correlation between the model's performance and fairness score (i.e., fairer prompts yield better performance). The red star, referred as the "Oracle" represents the optimal average performance, which consistently correlates with higher fairness. This observation prompts us to enhance the ICL performance by identifying the fairest prompt.
enumerate all possible combinations and permutations of
Nevertheless, discovering the fairest demonstration combination proves to be a formidable challenge, given the existence of Σm Ckkk! distinct candidates. As the size of the training set increases, this task becomes intractable. In order to tackle this problem, we propose two efficient strategies for approximating the most suitable demonstrations in the subsequent section.
"This notion differs slightly from the concept of social bias, which concentrates specific feature attributes rather than labels. Our approach be naturally extended to mitigate social bias in various settings.
5

---

Prompipnn
Demonstrations
G-juir-Prompting
Review: great piece of
fair T
Review: great piece of
Sentiment: positive
Sentiment: positive
Unique Evaluation
2 Review: ... fails on its own...
Review: ... fails on its own....
Sentiment: negutive
fair
Sentiment: negative
Review: ... great piece
Sentiment: positive
Fairness Runking
<Demonstratin
fair 1
+
<Demonstration >>
Demonnstration
<Deoonstration 1>
Figure 3: Overview of Most-fair Prompting.
Fairest Prompt Search
Drawing upon the aforementioned observations, we propose two strategies aimed at identifying the most fair prompt, which have been empirically demonstrated achieve superior performance, Let us consider training set S comprising N samples; the goal of these search strategies is to select subset of samples from the training set and construct the context specific order so as to optimize the fairness criterion in Eq. 2
In an ideal scenario, we would consider the factors of demonstration selection and order permutation by examining Σm= C distinct candidates, which enumerates all possible situations. However, evaluating every candidate is infeasible, as demonstrated when N yielding over 10 candidates. In this paper, we introduce two search strategies to reduce computational cost: Praai-Prroppiing and G-fair G-fairProomptin. The T-fair-Pa T-fair-Prompp Prompting strategy decreases complexity from e Ck to (NN), but its performance hinges on the selection of and may be unstable when an unsuitable value of k is chosen. As result, propose additional greedy search strategy, termed which lowers complexity to O(N2) and offers superior approximation of the oracle solution. Fig. visualizes the computational costs over different training set size.
4.1 T-fair-Prompting
The central idea of TT-air-Prompping (top-k) is founded the heuristic understanding that the fairest prompt usually consists of demonstration samples with reduced individual biases. Consequently, T-fair-Prompting constructs the prompt through two-stage process. Initially, the prediction bias is assessed when the prompt formulated using individual demonstrations. Subsequently. the top-k fairest demonstrations are chosen and employed to prompt the LLM. It is important to note that fairer demonstrations are likely to be situated towards the end of the sequence, as the generation is more accordance with prior research [18]. A comprehensive
influenced by proximate demonstrations, description of the process is presented in Algorithm [ while visual representation can be found in Fig. 3 Specifically, when & is equivalent the size of the training set, the method degrade to search for the optimal order of demonstrations. Nevertheless, T-fair- f-fai-Prromptin is heavily reliant on chosen value of More crucially, Prompipto addresses this issue through purely local perspective, thereby neglecting considerations from global standpoint, which typically results in sub-optimal outcomes. a result, we subsequently introduce the G-fair-Promen Prait-Promptin method, which operates in a local-to-globe fashion, described below.

---

4.2 G-fair-Prompting
The fair--Prompting (greedy search) algorithm adheres to the standard procedure of greedy search, which seeks the optimal solution by making locally optimal choices at each stage. In each step of the algorithm, the chosen demonstration is the one that allows the updated prompts achieve the highest fairesss score. This strategy balances the quality of the search with the worst-case time complexity. By accepting an increased worst-case time complexity of O(N2), the search quality significantly enhanced. It is important to note that the G-fair-Prompting algorithm operates from local to global perspective as shown by Algorithm 2 During the initial stages, the bias of individual samples is taken into account, while the later stages focus reducing global predictive bias. Specifically, at each step, we insert new demonstration I( y:) from the remaining demonstration set S" (ensuring demonstrations are not repeated) at the beginning of the current context p and select the demonstration that maximizes the fairness improvement. Formally, at step 9 in Algorithm 2 the inserted demonstration should satisfy the following criterion:
arg max fair(I (Z,, 91) + p) EES
s.t. (I U.) B fair(p).
(3)
Algorithm T-fair-Promptinn
Algorithm 2 G-fair- Promptit
1: Given: training set ((r, y)) N pre- trained LLM M, transformation template
Given: training set S {( ys) N. pretrained
LLM M, transformation template r(.), and context-free input ?)
T()), and context-free input y
2: Initial prompt
2: Initial prompt
3: for (I,, vi) in do
while is not null do
4: Inference P + (1:,/T(a; M6)
@
4: for (æi, yi) in S do
y)ly E )) via M
5:
Pimp + [xxi, M/) + Inference P +- (p(ulpmmp @ m)ly E }} via
5: Calculate the fair(T( 9:)) according
6:
Eq.2 2
M
6: end for
7:
Calculate the fair (Pmp)) according to Eq.
7: Sort fair; .N (I (1)) in descend-
end for
ing order
9: Insert the demonstration that can improve fair- ness best and remove it from S
8: for in d,
do
9: Insert the most d fair demonstration at 10: Stop searching when fairness can't be im- proved
the head of p
10: end for return P
11: end while
12: return
5
Experiments
5.1 Experimental Setup
including open-source models and black-box cloud API. Recently, Meta has released their powerful pretrained LLMs, LLaMA. LLaMA models with 13B parameters
Models. There large number of available LLMs (Appendix A.2
achieve comparable performance in contrast BLOOM and GPT-3 with much larger model size. this paper, we evaluate the effectiveness of our method on BLOOM (176B) and LLaMA models of different sizes. We have opted to employ LLaMA (65B) as substitute for GPT-3 in our experiments, since oepnai strictly restricts the API access to certain areas.
Datasets. We conducted experiments various text classification datasets (11, namely SST-2, AGNews, CoLA, TREC, and RTE. Furthermore, the maximum input length of LLaMA is 512, and the sentences in RTE are too long for LLaMA. The task descriptions and statistics available in Table 6 in Appendix.
5.2 Results
We conducted experiments different settings and reported the results of five runs. We compared our method with the diversity-guut guided searching strategy proposed by Zhang et al. [T2) (Global view) and the similarity-guide searching strategy proposed by Liu et al. [15] (Local view). Note that methods
7

---

Table Accuracy for different prompting strategies (averaged on 5 .4) different seeds, where Top- k and Greedy indicate T-fair- Promppin with & demonstrations and G-fair-Proon Srir-Prompting respectively).
Ours
Model
Dataset
Random
Diversity
Similarity
Top-2
Top-4
Greedy
SST2
92.72.3
95.00.9
94.0p.9
94.60.5
9
91.24.0
AGNews
73.95.9
70.210.1
74.8a.8
75.42.2
74.82.3
79.61.4
BLOOM (176B)
TREC
47.914.6
46.0g.7
31.4g.1
55.413.3
39.219.3
66.82.5
RTE
62.44.2
69.21.9
67.23.5
55.61.0
57.61.9
63.02.1
CoLA
68.44.
71.0s.7
69.82.5
66.48.6
66.83.7
68. 68.26.2
SST2
82.511.8
90.02.7
72.84.4
82.0
80.012.2
85.68.2
AGNews
75.23.0
75.0g.1
75.02.4
73.23.9
69.84.4
76.44.6
LLaMA (33B)
TREC
68.111.1
68.24.7
60.63.4
71.41.1
57.817.3
80.2s.8
CoLA
66.911.
68.8a.s
72.82.0
63.813.3
69.83.9
70.64.2
SST2
90.07.7
90.8g.0
87.43.
88.28.6
95.81.6
87.8p.u
AGNews
76.8s.0
78.2s.1
78.21.8
77.03.4
76.2..9
76.0
LLaMA (65B)
TREC
63.614.2
65.210.9
64.05.5
65.813.0
57.419.9
74.012.2
CoLAA
66.2g.8
62.6g.6
59.2
67.611.7
62.6g.5
72.04.5
based on local view are time-consuming since they require searching different demonstrations for every test example. Table [ shows the performance of the different strategies, where "Random" indicates the average accuracy for enumerating all situations, "Diversity" and "Simillarity" indicate demonstrations are selected according to diversity and similarity (details please refer to Appendix (A66). respectively. For each dataset, we set the size of the training set to 4. "Diversity" and "Similarity" select 4 from demonstrations, as they need more candidates. The baseline is expensive to compute since enumerating all candidates for 4 demonstrations in RTE on BLOOM will take more than 120 NVIDIA A100 GPU hours. We enumerate all candidates for the training set with 4 demonstrations different models, shown in Fig. 2 The results models whose parameters less than 13B shown in Table (i.e. GPT2-XL (1.5B), LLaMA (7B), and LLaMA (13B)).
G-fair-Prompting can reach a close approximation of enumeration. To evaluate whether the G-fair-Prompping (Greedy) method can approximate the best performance enumerating all candidates, we marked the performance of E-ffir-Prompping with a green star (representing the closest value to averaged accuracy of P-oaii-Prompiing on the line). We found that G-fair-Prompting can achieve very close approximation to enumeration. As shown in Fig. 2 most prompts searched by G-fair-Pron Prait-prommtingg achieved a top 20% ranking, and on BLOOM (176B), G-fair-Prompting almost found the most fair prompt.
E-fair-Prompting outperforms T-fair-Promptinng As shown in Table T although T-fair- Prompting achieves better performance compared with random selection, G-fair-Prompting consis- tently outperforms Prompting. Furthermore, Top-2 significantly outperforms Top-4 in most cases (over 5%), indicating that the number of demonstrations selected is crucial. Overall, the results demonstrate that E-fair-Promptin achieves satisfactory performance with only a slight additional cost.
Compared with SOTA methods. We compared our methods with several State-of-the- (SOTA)
Ch
methods, including diversity-guided and similarity- guided techniques. We observed that our greedy approach outperforms most of these SOTA methods in most situations, and the improvements of over 10% are observed on dataset TRBC. The similarity- guided method, on the other hand, achieved the best per- formance on the topic classification task (AGNews). This is because it searches for unique prompt for every different test example based the dis-
Accuracy
8
Osclle Auceney:
20
0.9e-Punyying 48.
Jasdon Poaars 66A
10
40 Sorted by Fairness
20
50
60
Figure BLOOM not sensitive CoLA.
8

---

tance between the embeddings of the training samples and the test example. This strategy selects demonstrations with labels that are the same as the test samples, and Language Models (LLMs) tend to predict biased predictions toward the labels that always appear in the context. However, the similarity-guidd method may prove inadequate when applied to other tasks. Specifically, the sivillritt-guidee strategy exhibits lower performance compared to random selection in QC and acceptability tasks. Furthermore, the E-fair-Prompting approach may occasionally falter when the model's s sensitivity to the task is not immediately evident, as observed in the acceptability task on BLOOM (depicted in Fig. 4). Note that the training set size of compared methods is 4x larger than ours.
Comparison with Calibration Method. Post-calibration [18], can enhance the accuracy of a given prompt in most cases. However, when the selected prompt is of poor quality, the performance may remain inadequate even after calibration. We compared the performance of G-fair-Prompting with random selection with calibration (averaged on all candidates), and found that G-fair-Prompin can outperform random selection with calibration in most situations. For example, on the topic classification task, G-fair-Prompting achieves the best performance find that post calibration can harm the performance of the model and it occurs significantly times, so it is worthwhile to reconsider the influence of manipulating the model'ss probability directly.
most models. Moreover,
Table 2: Accuracy comparison after post calibration.
BLOOM (176B)
LLaMA (33B)
LLaMA (65B)
Dataset
Method
Average
Worst
Average
Worst
Average
Worst
Random (cal)
66.8g.0
57.2
69.2s.2
59.4
74.6g.7
66.2
TREC
Ours
66.82.5
64.0
80.25.3
75.0
74.012.2
50.0
Ours (cal)
77.01.1
75.0
76.65.1
70.0
72.812.6
48.0
Random (cal)
73.06.6
61.8
71.95.0
64.0
78.24.7
71.6
AGNews
Ours
79.61.4
77.0
76.44.6
69.0
76.04.o
71.0
Ours (cal)
77.41.4
76.0
76.04.4
68.0
76.43.6
70.0
Random (cal)
68.55.5
61.2
67.85.1
63.6
54.012.4
42.4
CoLA
Ours
68.26.2
57.0
70.6
64.0
72.04.5
66.0
Ours (cal)
68.05.2
58.0
70.43.8
65.0
72.04.5
66.0
Post calibration 18] can improve the accuracy of a certain prompt (in most cases), but when the selected prompt is very poor, the performance is still very poor even after calibration. We conducted experiments (Table 2] to compare the performance of G-fair- erfair-Prompting and random selection with calibration (" ("Average" and "Worst" indicate averaged accuracy and worst performance on all permutations of training examples), and observed that w-ffir-Prooppee outperforms random selection with calibration in most case. For instance, the CoLA, P-rair-Prommting exhibited superior performance on most models. Additionally, we find that post-calibration could negatively affect the model's performance in many scenarios while it sometimes can improve the performance significantly even on selected prompts, for example, an improvement by 10% is observed on BLOOM- TREC. For more detailed discussions, please refer to Appendix A.5 Hence, it is crucial reconsider the impact of directly manipulating the model's probability.
6 Conclusion
In this paper, we revisit the sensitivity of large language model across prompts, and analyse the issue from predictive bias perspective. Accordingly, we employ "content-free" strategy as metric termed as fairness to evaluate the predictive bias of fixed prompt and show that model's performance highly consistency with fairness. Then, we propose two strategy to search most fair prompt current famous LLMs, and validate the
in the original space. We conduct extensive experiments effectiveness of the proposed strategy. Moreover, in addition to fairness adopted in this paper, there would be more metrics for prompt searching in the future for different scenarios.

---

Acknowledgments
This work is supported by the National Natural Science Foundation of China (Grant No. 61976151 and 61925602), and Fu is supported by STAR Central Research Fund. The project was conducted during the internship in AI Lab, Tencent.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal. Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, volume 33, pages 1877-1901, 2020.
[2] Bloom: A 176b-parameter open-access multilingual language model.
https://
[3] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models knowledge bases? arXiv preprint 1rXiv:1909.01066, 2019.
[4] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2021.
[5] Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. Improving few-shot performance of language models via nearest neighbor calibration. arXiv preprint arXiv:2212.02226. 2022.
[6] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[7] Xiang Lisa Li and Percy Liang. Prefix- Prefix-tuning: Optimizing continuous prompts for generation, 2021.
[8] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021.
[9] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121, 2021.
[10] Guanghui Qin and Jason Eisner. Learning how ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv:2 2021.
[11] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2 10385, 2021.
[12] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03449) 2022.
[13] Claudio Gentile, Zhilei Wang. and Tong Zhang. Fast rates in pool-based batch active learning. arXiv preprint arXi::2202.0544 2022.
[14] Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302. /2246, 2023.
[15] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXi::1101.06804, 2021.
10

---

[16] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprini
2023.
/ttps://opeeaaa.com//looo chatppt
[18] Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
[19] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical Report, 2018.
[21] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0744 2018.
[22] Ben Wang and Aran Komatsuzaki. GPT-J66: A Billion Parameter Autoregressive Language Model. http://s/.ggtttooo.ooooooooooooo
May 2021.
[23] http:://mmngaaanee..eoeeeataate2
[24] OpenAl. Gpt-4 technical report, 2023.

---

A Appendix
A.1 Pretrained Large Language Models
Neural autoregressive language model (LMs) are designed for next token prediction to predict the probability distribution over the next token after sequence of tokens input, and pre-trained LMs show their superior performance since they are trained on various programming languages and a large-scale curated dataset. Training large natural LMs are very expansive and time-consuming process since they always have billions of parameters, which limits the development of LMs. Fortunately, many pre-trained LMs are open access limited access, which promotes researchers pool their time and makes the resources to collectively achieve a higher impact. EleutherAl makes the GPT-J [22] and GPT-Neox [23] public available Hugging Face. GPT-3 [ is limited access in OpenAl which be used by researchers for fee, and another large open-science open-access multilingual language model named Bloom 2 is provided by BigScience.
A.2 Open Access Models
Table 3: Pretrained language models
Model
Params
Provider
Access
GPT-2
124 M
Hugging Face
OPEN
GPT-Medium
335 M
Hugging Face
OPEN
GPT2-Large
774 M
Hugging Face
OPEN
GPT-XL
1.5 B
Hugging Face
OPEN
GPT-3 (ada)
350 M
OPENAI
LIMITED
GPT-3 (babbage)
1.3 B
OPENAI
LIMITED
GPT-3 (curie)
6.7 B
OPENAI
LIMITED
GPT-3 (davinci)
175 B
OPENAI
LIMITED
GPT-J
B
EleutherAI
OPEN
GPT-NeoX
20 B
EleutherAI
OPEN
Bloom
176 B
BigScience
OPEN
7 B
Meta
OPEN
13 B
Meta
OPEN
LLaMA
33 B
Meta
OPEN
65 B
Meta
OPEN
A.3 Additional Figures Different Settings
In additional to the Fig. 2 we shows the performance on different models for enumerating all candidates, note that the shadow indicates the half value of standard deviation for clear presentation since the variance is very high for LLMs.
A.4 Accuracy Varies with demonstrations
Accuracy Varies with Example Amount Demonstrations play an important role in imparting task-related information to language models through in-context learning. Then, the question arises does a larger number of demonstrations necessarily equate better performance? To answer this question, we evaluated performance in terms of accuracy by gradually increasing the number of demonstrations, We set p = [(11, 81 ) are erased with & decreasing from n to 1. Intuitively, accuracy would vary highly across different numbers of demonstrations, and the phenomenon is observed Fig. 6aa| To our surprise, however, erasing some demonstrations can result in a better performance. Removing some demonstrations can perform better and sometimes GPT-3 achieves best accuracy when there is only a few demonstrations remaining. This highlights the importance of considering the appropriate number of demonstrations.
+ T( yk),, where & 1.... n. and demonstrations
12

---

Siecle O mmuuiaii 202
-3
20
50
60
20
30
30
40
50
Sorted by Phirness
Sorted Farmess
Softed by FN mene
(a) AGNews (GPT2-XL 1.5B)
(b) TREC (GPT2-XL 1.5B)
(c) RTE (GPT2-XL 1.5B)
80
Severrtee 132
Zecapt
Serted by Falmess
Sorted by Faimness
Sorted by Falrnsss
(d) AGNews (LLaMA 33B)
TREC (LLaMA 33B)
(I) SST-2 (LLaMM 33B)
Figure 5: Accuracy is highly consistency with faimess and greedy search can find good prompt, where "Random" " and "Oracle" indicates the average accuracy of all prompts and the upper-bound performance according to fairness.
Accuracy Varies with Quantity
1990 Accuracy Varies with Permutation
Accuracy Varies with Selection
70
70
70
60
(a) Varying amount of examples
(b) Permutation
(c) Select different examples
Figure 6: ICL suffers from high instability due variations in example amount, example order, and example selection.
Example Order The performance of model is sensitive to the order of the demonstrations. as has been discussed in [4) Even when the demonstrations are the same, different permutations of the demonstrations can result in vastly different outcomes. As there are n! possible permutations, we introducing a strategy of permuting the demonstrations by circularly shifting the index of the demonstrations. The demonstration can be represented
[ Uk+11 ① ..... @ [[ yn) + T( v1) ① .... @ I (# yse) A.S shown in Fig. 6bb the accuracy varies highly with permutation which consistent with the observations in 4].
Example Selection In this paper, we find which demonstrations are selected is influence the model extremely. This scenario can be described as selecting k demonstrations in n training samples. In Fig. [c] we only select one example for demonstration ablate the impact of demonstrations order, and the accuracy also varies highly with different example selected. In this work, we only detail evaluate the proposed probing method on the erasing demonstrations and permutation, although our method improves by 20% in the setting of example selection on SST-2 (GPT2-XL), because selecting demonstrations on set with n training samples can't be regarded as k shot learning in the strict sense.
A.5 Relationship between with and without-calibrattin
G-fair-Prompting without post-calibration outperforms random demonstrations after post- calibration. Based on Table is apparent that E-fair-Promptie outperforms random selection prior to post-calibration. This leads to natural question: do prompts with better performance before cali- bration also indicate better performance after calibration proposed by Zhao al. (18) ? investigate the relationship between performance with- and vithout-calibration. we calculated the Pearson cor- relation coefficient between the accuracy with- and without-calibration Pearson wancaac., (accw /os aCGwith)) A positive coefficient value suggests that prompt with high accuracy before calibration has
13

---

Table 4: Accuracy for different prompting strategies (averaged on 5u.... different seeds).
Ours Top-4
Model
Dataset
Random
Diversity
Similarity
Top-2
Greedy
SST-2
61.16.1
60.811.4
65.8s.7
74.212.0
AGNews
38.911.4
45.212.5
37.211.2
46,4119
GPT2-XL (L.SB)
TREC
22.1
19.4s.9
28.2g.2
25.07.4
RTE
53.2g.9
54.07.5
53.65.9
56.42.2
AGNews
64.510.0
66.4g.1
66.011.7
69.2s.s
63.8s.7
LLaMA (7B)
TREC
49.510.4
51.49.6
48.410.5
38.615.2
61.318
CoLA
60.410.6
63.8s.7
58.27.8
61.66.5
36.43.6
AGNews
72.27.7
78.43.5
73.69.0
74.21.3
75.22.8
LLaMA (13B)
TREC
46.416.5
48.016.0
51.016.6
39.223.3
61.412.1
CoLA
67.72.9
67.22.4
67.02.0
67.21.6
67.02.0
Accuracy
85
with cal
w/o cal
80
75
0
10
20
30
40
50
60
Prompts
Figure Illustration of accuracy relationship between with- and without calibration when Pearson (-) is positive.
higher likelihood achieving higher accuracy after calibration than other prompts. We take the topic classification task on LLaMA(65BB) for illustration show the relationship between with- and without calibration when Pearson is positive 7 T the Pearson correlation coef- ficient on accuracy of permutation and G-fair- (G-far-Prommting after calibration. The majority of Pearson correlation coefficients were found to be positive, indicating that prompts with better performance before calibration have more potential perform well after calibration. Furthermore, our results the LLaMA family reveal that the larger the model, the stronger the correlation between performance with- and without-calibration. For instance, the value of the Pearson correlation coefficient increases from O to 0.7 as the model size increases.
Theorem A.1. Suppose the performance of the model under certain prompts with- and without- calibration is positively correlated, i.e., Pearson (acCw//o acCwith) O, if we can assure E Blaco ), then have EE (((aceSeteete Selected) EEacc ((accBandoom)
E (aceSelected
Table 5: Pearson's between the with and without-calibration..
BLOOM 176B
LLaMA
Dataset
13B
33B
65B
TREC
0.1274
0.1551
0.2959
0.3090
0.5151
AGNews
0.3875
-0.0471
0.3044
0.6953
0.7100
CoLA
0.4050
0.3592
0.5193
0.3611
0.8012
As analysed in Theorem A. higher likelihood of achieving higher accuracy after calibration than random selection. Our approach consistently identifies an appropriate prompt, evidenced by the results in Table [] Moreover, the
if we can find prompt with high accuracy before calibration, we have
14

---

performance of the model exhibits positive correlation with and without calibration under certain prompts, illustrated Table Therefore, our method is more likely to enhance calibration performance.
A.6 Diversity and Similarity
Diversity indicates the demonstrations are selected according to the diversity of the embeddings of all samples [12]. Specifically, we select the most diverse samples as demonstrations by k-means clustering on training set consists of 16 samples.
Similarity indicates the demonstrations are selected according to the similarity of the embeddings of all samples [15]. Specifically, we select the most similar samples as demonstrations according to Euclidean distance from the training set consists of 16 samples. Note that demonstrations for different test samples are different for best performance. larger training set may result in better performance, but the size of training set 16 for fair comparison.
Test sample
Diversity
Similarity
Figure 8: Illustration for Diversity and Similarity.
A.7 Details for experiments
In Fig. [ we show the high instability due to high variations in demonstrations selection and order for both with and without-calibration [18]. Specifically, for left two figures, sample different demonstrations randomly from dataset AGNews, and estimate the influence demonstration selection. On the other hand, the right two figures show the instable performance due to permutation when the demonstrations are fixed. Specifically, we randomly sample demonstrations and estimate the performance with all possible orders.
A.8 Complexity of different strategies
10669
Enumcrate
T-fair
G-fair (wie)
105553
1699)
50
Cost
Training Set Size N
Figure 9: Computational cost. T-fair and G-fair indicate T-fair- Srtir-Prompting and G-fair- Mair-Prompting respectively, and "w/c" indicates the worst case.
15

---

A.9 Limitations.
While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As result, may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.
16

---

B Clarifications After Rebuttal
@: Computational cost of using G-fair Prompting becomes non-trivial as N goes larger.
(I) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of 1/2 N2 is not common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [24), the number of demonstrations is fixed at 5. As result, the overall computation cost remains acceptable, especially when compared to other existing methods [T5] among the typical methods detailed in the related work.
(2) Moreover, our method does require re-selection of demonstrations for different test samples. Consequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with larger number of test samples.
(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore.
(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of O(n!). The smaller training set used in our experiments due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.
Q: One can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.
(1) The proposed simple version T-fair- frirt-rromptnng serves as a time-efficient alternative of G-fair- prompting, while still maintaining competitive performance.
(2) Compared with Pfair-prrompiie G-fair- aefoiiipromptigg offers the advantage of automatically deter- mining the number of demonstrations, providing more natural and flexible approach.
(3) Therefore, tradeoff between performance and time cost can be achieved by transitioning between Mrait-prompting and G-fair-pron prompting. Specifically, we can select the demonstrations order as same as frorpppoo and then decide the number of demonstrations by the strategy of (--farr-romppl The computation cost upper bound of this strategy will be the lower bound of G-fair-prompting while its fairness lower bound will be the upper bound of T-ffar-- Prirr-prompiing
Q: Under the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them.
(1) As shown Fig.2, we have tested different choices of few-shot examples (e-g.. [x1, x2]/(x3, x4]) and all possible orderings of them (e.g., [x1, x2, x4]/[x2, x4, x4)). In total, 64 different possible situations were considered when the training set size was 4.
(2) Notably, G-fair-Prompiic performs remarkably close to the best performance obtained through enumeration, highlighting the efficacy of our proposed method.
Q: How to choose the semantic-fae -ree token without any guidance?
(L) Following previous work [18. three different semantic- free prompts are selected, i.e., ['N/A', ","
'J. The fairness is then calculated as the average of them.
(2) These tokens were chosen based on their guaranteed performance in previous work 18 We will
provide further clarification this selection in the final version of our paper.
Q: Would be the proposed method helpful for long-form QA?
While the current version of our method is designed for classification tasks, we acknowledge the potential for its extension to long-form QA scenarios.
(1) The limitation preventing direct deployment in generation tasks lies in the absence of an appropri- ate method to measure the faimeess of sentence-level generation.
(2) Similarly, for measuring the calibration of large language models, such as ECE, can only be calculated for classification tasks 22).
17

---

(3) However, we believe that once the community develops techniques to measure the predictive bias of sentences, our proposed strategies can be naturally extended long sentence generation tasks. This presents an exciting avenue for future research.
Q: What does "total cost" in table "Dataset descriptions" refer to?
Table 6: Dataset descriptions.
Corpus
Task
Classes
Domain
Total Cost'
SST-2
sentiment
movic reviews
over 60 GPU hours
TREC
QA/QC
open domain
over 220 GPU hours
AGNews
topic
news
over 250 GPU hours
CoLA
acceptability
misc.
over 160 GPU hours
RTE2
NLI
news, Wikipedia
over 110 GPU hours
Total Cost=Hours x GPUs. Hardware: BLOOM=A100, LLaMA=V000, Not applicable LLaMA because of the maximum prompt token limit.
(1) "Total cost" indicates the overall computational cost each dataset including baselines and the proposed methods.
(2) We report the total computational cost on each dataset provide more comprehensive description of the datasets used this paper. We will clarify this the final version.
Q: The diversity baseline seems similarly competitive to the proposed approach.
Regarding performance, would like to clarify the following points:
(1) Our method consistently outperforms the diversity baseline across various scales of the LLaMA family datasets.
(2) The similarity in performance between the diversity method and method on the Bloom model attributed to the Bloom model's robustness to the transformations of the demonstration. Even random demonstrations can achieve competitive performance with the Bloom model.
(3) It is essential to note that the baselines, including the diversity method, are the latest methods proposed shortly before submitting this paper. They have demonstrated robust performance multiple tasks.
Q: Could the authors compare using a strategy such as the proposed one with large language model vs. fine-tuning with a smaller model?
**
Note that although the former strategy requires iterating over the entire dataset, the latter strategy only requires performing inference. It does not incur any space cost for storing gradients time cost for updating gradients, making this process highly efficient.
We would like clarify the advantages of utilizing a large language model and comparing it with fine-tuning a smaller model:
(1) Generalization ability of large models: Large language models (LLMs) accumulate significant prior knowledge during pre-trrining, resulting in better generalization abilities compared to small models fine-tuned for specific tasks. Even if the performance of a large model is not significantly better than that of dedicated small model on a specific task, the robustness the large model in open environments, such as out-o--distribution (OOD) scenarios, is significantly superior to that of dedicated small model (4
(2) Multiple tasks: A single large model can perform multiple tasks simultaneously. Compared to fine-tuning a small model for each task, a large model can seamlessly switch between various tasks and effortlessly incorporate new tasks.
(3) Time efficiency: Many open-source large models, such LLaMA, do not require modifications when encountering new tasks. Even equipped by the methods including baselines in this paper, only few additional inferences necessary, without any updates to the model or backpropagation of gradients.
Q: Explanation about the text in lines 293-296 and 300.
18

---

(I) Line 293-296 Explanation:
The sentence mentioned in lines 293-296 analyzes why the similarity-guiden method achieved the best performance on the topic classification task (AGNews). This method selects unique prompts for different test examples based the distance metric. Consequently, it chooses demonstrations with labels that are the same as the test samples. Large Language Models (LLMs) tend to predict biased outcomes towards the labels that frequently appear in the context of demonstrations. As a result, the similarity-guider method excels on the topic classification task but may perform poorly on other tasks, such natural language inference.
(2) Line 300 Explanation:
The Bloom model exhibits high robustness to various demonstrations, even random ones, leading to similar performance results between our method and the compared method. can be observed that the Bloom model's performance remains competitive regardless of the selected demonstrations.
Q: Discussion about limitations.
Due to space limitations, our discussion on limitations is included in Appendix (A.9 Limitations). While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities the next token. As result, may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.
Q: Figure 2 shows non-monotonic trend between the bias and predictive quality.
According to Figure 2, the relationship between predictive bias and predictive quality is not strictly monotonic. Instead, the overall trend shows a significant negative correlation. To avoid any misunder- standings, we will use clearer descriptions in the final version of the paper.
Q: The term fairness is slightly misleading.
The fairness defined in our work is defined from the concept of predictive bias, which may not align precisely with the fairness, such as social bias, that is commonly referred It is important to note the distinction between fairness in the context of predictive bias and fairness in the context of social bias. Moreover, we highlight that the proposed metric can be extended to analyze social bias, as social bias can also lead to predictive bias [4]
Q: Contribution and difference from 78
Our paper shares a similar metric with calibration -bbrratioo-beoore-usee [8] to access the predictive bias of a given prompt. However, the differences are significant and listed as follows:
(I) The prior approach aims to use the metric for calibrating the output, which can still be influenced by the quality of the used prompt, as evidenced by Table 3.
(2) In contrast, research focuses finding near-optimal prompt the original space to improve the model's performance, eliminating the need for any post-adjustment to the output.
(3) Additionally, our paper introduces empirical validation of the connection between predictive bias and final task performance, as depicted in Fig. 2, which is not explored in 24).
(4) The main contribution of our work lies in analyzing the relationship between predictive bias and performance through extensive experiments, and proposing a Greedy research strategy to solve the NP-hard problem.
(5) Our experiments have revealed that even without calibration, the prompt selected by our method outperforms randomly selected prompts, even with calibration.
(6) Moreover, we have observed that [24] can potentially harm the model's performance, which is practical concern worth considering when manipulating the model's probability directly.
Q: How sensitive are these strategies to the choice of hyperparameters?
In this paper, reduce the influence of hyperparameters by minimizing the need for manual hyperparameter tuning.
19

---

(1) In t-fair-prompting the number of demonstrations is automatically determined, indicating that no new demonstrations are added once it is determined that adding more demonstrations does not lead to improvement.
(2) As for the threshold for fairness improvement, fix it with O, indicating that new demonstrations are added to the prompt if they are deemed helpful.
(3) While these two aspects could be manually adjusted to make the method more flexible. To ensure the stability of the experimental results, we avoid deliberately setting these hyperparameters.
20


Reviews:
[{"review_id": "oVXl5Qtft0", "reviewer": "Reviewer_Q4AG", "summary": "Finding an (nearly) optimal prompt is crucial for successful in-context learning (ICL) of large language models (LLMs). It is known that performance of downstream tasks is sensitive to prompts themselves, few-shot examples, and their orders, often requiring large degree of non-trivial tuning. This paper presents a content-agnostic fairness metric that allows users to assess the quality of a given prompt purely in the perspective of predictive bias. Under this metric, the authors propose two methods for optimizing the prompts. Given the few-shot examples for ICL, T-fair Prompting computes the predictive bias of individual demonstrations by feeding each single shot. Then it chooses the user-specified number of examples with the lowest predictive biases. In contrast, G-fair Prompting promotes the greedy local search where it appends one additional shot that shows the best overall fairness. Users can choose one of the two proposed methods considering the tradeoff between algorithmic complexity and performance gain.", "questions": "(1)\tCan you provide computational cost for running T-fair and G-fair prompting over increasing N? \n\n(2)\tUnder the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them. Though such experiments would be expansive to navigate combinatorially many configurations, measuring the performance on a limited setting helps understanding the relative performance of T-fair and G-fair Prompting.\n\n(3)\tHow to choose the semantic-free token without any guidance? It is less clear that such token would exist universally across different LLMs across various downstream tasks. In any case, using just one manually-reserved tokens for fairness evaluation is not easily justifiable without sensitivity analysis.\n\n(4)\tPredictive bias would be more useful for solving classification tests by generation (as if the paper tested the sentiment analysis). Would be the proposed method helpful for long-form QA?", "limitations": "No specific points are described or probed.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe paper provides simple yet effective approaches for prompt tuning.\n\n(2)\tBoth T-fair and G-fair Prompting algorithms are natural and practical in the spirit of local individual search and greedy search.\n\n(3)\tThe proposed methods achieve highly competitive accuracies comparing to the current state-of-the-art methods.", "weaknesses": "(1)\tComputational cost of using G-fair Prompting becomes non-trivial as N goes larger even if it uses greedy approach for expanding local bias to global bias. \n\n(2)\tOne can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.\n\n(3)\tNeither guidance for how to choose semantic-free tokens nor sensitivity analysis for various semantic-free candidates are explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for the reviewers' valuable feedback. We have carefully reviewed the comments and offer the following responses to address each point:**\n\n----\n\n***Q1: Computational cost of using G-fair Prompting becomes non-trivial as N goes larger.***\n\n\n(1) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of 1/2*N^2 is not a common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [1], the number of demonstrations is fixed at 5. As a result, the overall computation cost remains acceptable, especially when compared to other existing methods [2,3,4] among the typical methods detailed in the related work.\n\n(2) Moreover, our method does not require re-selection of demonstrations for different test samples.\nConsequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with a larger number of test samples. \n\n(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search on a subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore. \n\n(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of O(n!). The smaller training set used in our experiments is due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.\n\n----\n\n***Q2: One can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.***\n\n\n(1) The proposed simple version T-fair-prompting serves as a time-efficient alternative of G-fair-prompting, while still maintaining competitive performance.\n\n(2) Compared with T-fair-prompting, G-fair-prompting offers the advantage of automatically determining the number of demonstrations, providing a more natural and flexible approach. \n\n(3) Therefore, a tradeoff between performance and time cost can be achieved by transitioning between T-fair-prompting and G-fair-prompting. Specifically, we can select the demonstrations order as same as T-fair-prompting and then decide the number of demonstrations by the strategy of G-fair-prompting. The computation cost upper bound of this strategy will be the lower bound of G-fair-prompting while its fairness lower bound will be the upper bound of T-fair-prompting.\n\n----\n\n***Q3: Under the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them.***\n\n \n(1) As shown in Fig.2, we have tested different choices of few-shot examples (e.g., [x1, x2]/[x3, x4]) and all possible orderings of them (e.g., [x1, x2, x4]/[x2, x4, x4]). In total, 64 different possible situations were considered when the training set size was 4.\n\n(2) Notably, G-fair-Prompting performs remarkably close to the best performance obtained through enumeration, highlighting the efficacy of our proposed method.\n\n----\n\n***Q4: How to choose the semantic-free token without any guidance?***\n\n \n(1) Following previous work [5], three different semantic-free prompts are selected, i.e., [‘N/A’, ‘’, ‘ ’]. The fairness is then calculated as the average of them. \n\n(2) These tokens were chosen based on their guaranteed performance in previous work [1]. We will provide further clarification on this selection in the final version of our paper.\n\n----\n\n***Q5: Would be the proposed method helpful for long-form QA?***\n\nWhile the current version of our method is designed for classification tasks, we acknowledge the potential for its extension to long-form QA scenarios. \n\n(1) The limitation preventing direct deployment in generation tasks lies in the absence of an appropriate method to measure the fairness of sentence-level generation.\n\n(2) Similarly, for measuring the calibration of large language models, such as ECE, it can only be calculated for classification tasks [1].\n\n(3) However, we believe that once the community develops techniques to measure the predictive bias of sentences, our proposed strategies can be naturally extended to long sentence generation tasks. This presents an exciting avenue for future research.\n\n---\n\n[1] OpenAI. GPT-4 Technical Report. URL: https://arxiv.org/pdf/2303.08774v3.pdf\n\n[2] Yao Lu. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2021. \n\n[3] Claudio Gentile. Fast rates in pool-based batch active learning. In ICML 2022.\n\n[4] Jiachang Liu. What makes good in-context examples for gpt-3? In ACL, 2022.\n\n[5] Tony Z. Zhao. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.\n"}, {"review_id": "ogZ77lIxgg", "reviewer": "Reviewer_pNzj", "summary": "The paper shows that the accuracy of large language models on various text classification tasks for a given prompt tends to be correlated with the entropy of the output prediction when using that prompt for a context free input: in general, the more uniform the output probability distribution for a given prompt (made up of demonstrations from the dataset), the higher the accuracy of that prompting strategy for the dataset. The authors use this insight to select the demonstrations and show that greedily selecting demonstrations one-by-one can yield competitive performance to other strategies.", "questions": "1. What does “Total Cost” in Table 1 refer to? What are the time complexities of the G-fair-prompting approach on the models and datasets tested?\n2. The diversity baseline seems similarly competitive to the proposed approach. When would the authors suggest using one over the other?\n3. Could the authors compare using a strategy such as the proposed one with a large language model vs. fine-tuning with a smaller model? Especially since the former requires iterating over the entire dataset anyway, albeit differently than in fine-tuning, when is the former advantageous over the latter?\n4. Could the authors explain the text in lines 293-296 and 300 (as per comment #2 in weaknesses)?", "limitations": "The authors do not directly address limitations, which would be worth mentioning briefly. I did not factor this absence into my review. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors point out that empirical observation that higher entropy in the context-free case is correlated with higher accuracy was first introduced in Zhao et al. 2021. However, showing this correlation holds in the context of prompting is, to my knowledge, novel. \n2. The proposed prompting search strategies are simple to implement and competitive.\n3. The experiments seem pretty extensive, both in datasets and models tested.", "weaknesses": "1. Given the time complexity of the G-fair-prompting algorithm (i.e. O(N^2) where N is the size of the training dataset), the proposed approach does not seem very practical.\n2. The discussion of experimental results is vague at times. For instance, the explanation provided for why the similarity-guided method is best for AGNews (lines 293-296) seems as if it would apply to other tasks as well; a meaningful explanation would be able to differentiate between these settings. Also, it is not clear what \"the model's sensitivity to the task is not immediately evident\" means (line 300). In addition, practical discussion of the proposed approach relative to baselines would be helpful (e.g. when one is preferred over the other). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: The time complexity of G-fair-prompting.***\n\n(1) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of 1/2*N^2 is not a common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [1], the number of demonstrations is fixed at 5. As a result, the overall computation cost remains acceptable, especially when compared to other existing methods [2,3,4].\n\n(2) Moreover, our method does not require re-selection of demonstrations for different test samples.\nConsequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with a larger number of test samples. \n\n(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search on a subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore. \n\n(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of O(n!). The smaller training set used in our experiments is due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.\n\n---\n\n***Q2: What does “total cost” in table 1 refer to?***\n\n(1) “Total cost” indicates the overall computational cost on each dataset including baselines and the proposed methods.\n\n(2) We report the total computational cost on each dataset to provide a more comprehensive description of the datasets used in this paper. We will clarify this in the final version.\n\n---\n***Q3: The diversity baseline seems similarly competitive to the proposed approach.***\n\nRegarding performance, we would like to clarify the following points:\n\n(1) Our method consistently outperforms the diversity baseline across various scales of the LLaMA family datasets.\n\n(2) The similarity in performance between the diversity method and our method on the Bloom model is attributed to the Bloom model's robustness to the transformations of the demonstration. Even random demonstrations can achieve competitive performance with the Bloom model.\n\n(3) It is essential to note that the baselines, including the diversity method, are the latest methods proposed shortly before submitting this paper. They have demonstrated robust performance on multiple tasks.\n\n---\n***Q4: Could the authors compare using a strategy such as the proposed one with a large language model vs. fine-tuning with a smaller model?*** \n\n** Note that although the former strategy requires iterating over the entire dataset, the latter strategy only requires performing inference. It does not incur any space cost for storing gradients nor time cost for updating gradients, making this process highly efficient.\n\nWe would like to clarify the advantages of utilizing a large language model and comparing it with fine-tuning a smaller model:\n\n(1) Generalization ability of large models: Large language models (LLMs) accumulate significant prior knowledge during pre-training, resulting in better generalization abilities compared to small models fine-tuned for specific tasks. Even if the performance of a large model is not significantly better than that of a dedicated small model on a specific task, the robustness of the large model in open environments, such as out-of-distribution (OOD) scenarios, is significantly superior to that of a dedicated small model [1].\n\n(2) Multiple tasks: A single large model can perform multiple tasks simultaneously. Compared to fine-tuning a small model for each task, a large model can seamlessly switch between various tasks and effortlessly incorporate new tasks.\n\n(3) Time efficiency: Many open-source large models, such as LLaMA, do not require modifications when encountering new tasks. Even equipped by the methods including baselines in this paper, only a few additional inferences are necessary, without any updates to the model or backpropagation of gradients.\n\n---\n***Q5: Explanation about the text in lines 293-296 and 300.***\n\n(1) Line 293-296 Explanation:\n\nThe sentence mentioned in lines 293-296 analyzes why the similarity-guided method achieved the best performance on the topic classification task (AGNews). This method selects unique prompts for different test examples based on the distance metric. Consequently, it chooses demonstrations with labels that are the same as the test samples. Large Language Models (LLMs) tend to predict biased outcomes towards the labels that frequently appear in the context of demonstrations .\nAs a result, the similarity-guided method excels on the topic classification task but may perform poorly on other tasks, such as natural language inference.\n\n(2) Line 300 Explanation:\n\nThe Bloom model exhibits high robustness to various demonstrations, even random ones, leading to similar performance results between our method and the compared method. It can be observed that the Bloom model's performance remains competitive regardless of the selected demonstrations.\n\n---\n***Q6: Discussion about limitations.***\n\nDue to space limitations, our discussion on limitations is included in Appendix (A.9 Limitations). While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As a result, it may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.\n\n---\n\n[1] OpenAI. GPT-4 Technical Report.\n\n[2] Yao Lu. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2021. \n\n[3] Claudio Gentile. Fast rates in pool-based batch active learning. In ICML 2022.\n\n[4] Jiachang Liu. What makes good in-context examples for gpt-3? In ACL, 2022.\n"}, {"review_id": "J2yKqqDNbI", "reviewer": "Reviewer_WoAP", "summary": "The paper proposes two new procedures to select examples to be used for few-shot learning (in-context learning) for LLMs. The key idea for example selection is to select those examples that lead to a distribution over classes that is as close to uniform as possible . The two proposed variants (a) select the top k examples which each individually have the least ‘bias’ and (b) greedily add examples that reduce ‘bias’ the most until any additional example would increase it. The authors evaluate the proposed methods on a variety of open source LLMs on multiple tasks, where the proposed methods perform well. ", "questions": "I would love to see the authors address the usage of their method to other tasks that are not easily cast into a multi-class classification task.\nI would also like the authors to address the use of the term ‘fairness’ and why that is necessary for the paper at hand.", "limitations": "The authors did not address limitations and other impact. Especially, given the term fairness in the title, I would expect the authors to at least discuss the use of the proposed method to address fairness issues and also the potential misuse to introduce unwanted biases. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "In-context learning, and in particular the selection of effective demonstrations is an important area of study and thus of interest to the research community. \nThe paper is well written and the proposed methods are clearly presented. \nThe evaluation is well chosen to demonstrate the effectiveness of the proposed methods. \nThe proposed methods are a clever idea that is simple yet effective. ", "weaknesses": "The proposed method is limited to multi-class classification tasks, which places a significant limit on the practical use-cases of the proposed method, i.e., it does not apply to summarization, translation, open ended generation etc. \nSome of the claims in the paper are too strong and not supported. For example in the abstract in lines 9-11 the authors claim that “Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality.” Which is clearly not true and not supported by their findings in Figure 2, which shows a non-monotonic trend.\nWhile the writing is generally good, there are a few mistakes across the paper, e.g., line 34 to construct \n\nI think the use of the term fairness is slightly misleading in this paper. This the term ‘fairness’ is associated with many different meanings, but mostly with social aspects, the authors could directly use entropy as that would be a more technical description of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for taking the time to provide valuable suggestions and have a positive attitude towards our work.**\n\n---\n\n***Q1: The proposed method is limited to multi-class classification tasks.***\n\n(1) We acknowledge the limitation that the proposed method is currently applicable only to classification tasks.\n\n(2) The primary reason behind this limitation is the lack of an appropriate method to measure the fairness of sentence-level generation in the current context.\n\n(3) Additionally, the calibration measurement, such as Expected Calibration Error (ECE), is feasible only for classification tasks [1].\n\n(4) However, we believe that as the research community develops methods to measure predictive bias in sentences, the strategies proposed in our paper can be naturally extended to long sentence generation tasks.\n\n---\n***Q2: Figure 2 shows a non-monotonic trend between the bias and predictive quality.***\n\nAccording to Figure 2, the relationship between predictive bias and predictive quality is not strictly monotonic. Instead, the overall trend shows a significant negative correlation. To avoid any misunderstandings, we will use clearer descriptions in the final version of the paper.\n\n---\n***Q3: The term fairness is slightly misleading.***\n\nThe fairness defined in our work is defined from the concept of predictive bias, which may not align precisely with the fairness, such as social bias, that is commonly referred to. \nIt is important to note the distinction between fairness in the context of predictive bias and fairness in the context of social bias. \nMoreover, we highlight that the proposed metric can be extended to analyze social bias, as social bias can also lead to predictive bias [1].\n\n---\n[1] OpenAI. GPT-4 Technical Report. URL: https://arxiv.org/pdf/2303.08774v3.pdf\n"}, {"review_id": "1JVJszaIzQ", "reviewer": "Reviewer_35hW", "summary": "The paper examines the sensitivity of large language models to prompts and analyzes the issue from a predictive bias standpoint. They utilize a fairness metric called a content-free strategy to evaluate the predictive bias of a fixed prompt. Methodologically, the paper proposes two strategies, T-fair-Prompting and G-fair-Prompting, to search for the fairest prompt and validates their effectiveness through extensive experiments on large-language models.", "questions": "Please provide an in-depth response regarding the weaknesses mentioned above.", "limitations": "The role of few-shot prompting in in-context learning is significant. Therefore, I believe that providing a new insight is beneficial for the in-context learning community. However, I would like to request a sufficient response that adequately addresses the novelty of the proposed work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper exploits the correlation between predictive bias and overall performance and proposes two strategies based on the finding to approximately estimate the fairest demonstration combination.\n2. It is intriguing to observe that fairness-guided selection, rather than diversity or similarity-guided approach, often exhibits better performance.", "weaknesses": "1. The predictive bias-based fairness metric, which is the most key and primary idea, is adopted directly from [18]. The way of using the fairness metric in each T-fair-Prompting and G-fair-Prompting is also straight-forward. I think that the methodological novelty of this paper is limited. \n2. I can't find an ablation study regarding the proposed fairness metric. I am curious about the results and analysis when using other semantic-free prompts, apart from N/A.\n3. When looking at Table 2, there are quite a few results where diversity-guided selection demonstrates higher performance. The overall performance is not particularly impressive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your time and valuable suggestions. Next, we will answer the questions in detail point by point to address any concerns.**\n\n---\n***Q1: Contribution and difference from [1].***\n\nOur paper shares a similar metric with calibration-before-use [1] to access the predictive bias of a given prompt. However, the differences are significant and listed as follows:\n\n(1) The prior approach aims to use the metric for calibrating the output, which can still be influenced by the quality of the used prompt, as evidenced by Table 3.\n\n(2) In contrast, our research focuses on finding a near-optimal prompt on the original space to improve the model's performance, eliminating the need for any post-adjustment to the output.\n\n(3) Additionally, our paper introduces an empirical validation of the connection between predictive bias and final task performance, as depicted in Fig. 2, which is not explored in [1].\n\n(4) The main contribution of our work lies in analyzing the relationship between predictive bias and performance through extensive experiments, and proposing a Greedy research strategy to solve the NP-hard problem.\n\n(5) Our experiments have revealed that even without calibration, the prompt selected by our method outperforms randomly selected prompts, even with calibration.\n\n(6) Moreover, we have observed that [1] can potentially harm the model's performance, which is a practical concern worth considering when manipulating the model's probability directly.\n\n---\n\n***Q2: About semantic-free prompts.***\n\n(1) Following previous work [1], three different semantic-free prompts are selected, i.e., [‘N/A’, ‘’, ‘ ’], and the fairness is calculated as the average of them. \n\n(2) We choose these tokens to represent semantic-free token as they demonstrate guaranteed performance in previous work [1].\n\n---\n***Q3: The overall performance is not particularly impressive.***\n\nRegarding performance, we would like to clarify the following points:\n\n(1) Our method consistently outperforms the diversity baseline across various scales of the LLaMA family datasets.\n\n(2) The similarity in performance between the diversity method and our method on the Bloom model is attributed to the Bloom model's robustness to the transformations of the demonstration. Even random demonstrations can achieve competitive performance with the Bloom model.\n\n(3) It is essential to note that the baselines, including the diversity method, are the latest methods proposed shortly before submitting this paper. They have demonstrated robust performance on multiple tasks.\n\n---\n\t\n[1] Tony Z. Zhao. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.\n"}, {"review_id": "VwQxHSFkff", "reviewer": "Reviewer_bbKr", "summary": "The paper addresses the problem of constructing an appropriate prompt for improving the performance of in-context learning in large language models (LLMs). It introduces a metric based on predictive bias to evaluate prompt quality and empirically demonstrates that prompts with higher bias lead to unsatisfactory predictive quality. Leveraging this observation, the paper proposes a novel search strategy, combining T-fair-Prompting and G-fair-Prompting, to identify near-optimal prompts for enhancing in-context learning performance. The proposed method is extensively evaluated on various downstream tasks using state-of-the-art LLMs, such as GPT-3 and LLaMA. The results show that the method effectively improves in-context learning performance in an interpretable manner.", "questions": "How generalizable is the proposed method beyond text classification tasks? Have you explored its applicability in other types of downstream tasks, such as language generation or sequence labeling?\n\nThe search strategies, T-fair-Prompting and G-fair-Prompting, effectively optimize prompts by considering bias and fairness. However, how sensitive are these strategies to the choice of hyperparameters, such as the number of demonstrations (k) or the threshold for fairness improvement? Have you conducted sensitivity analyses to investigate their impact on performance?", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper highlights the importance of prompt quality and introduces a novel metric based on predictive bias to evaluate prompt performance. The proposed method effectively searches for near-optimal prompts by addressing bias issues, leading to improved in-context learning performance.\n\nThe paper conducts comprehensive experiments on mainstream LLMs and various downstream tasks to validate the effectiveness of the proposed method. The results consistently demonstrate performance enhancements, surpassing state-of-the-art methods in most cases.\n\nThe paper presents two search strategies, T-fair-Prompting and G-fair-Prompting, which efficiently optimize prompts by considering bias and fairness. The strategies offer a balance between search quality and computational complexity, addressing the challenges of prompt construction.\n\nThe paper is well-written and effectively conveys the motivation, contributions, methodology, and experimental setup. The algorithms and search strategies are clearly described, allowing for easy understanding and reproducibility.", "weaknesses": "Scalability: The paper discusses the computational complexity of the search strategies and their limitations. While the proposed strategies show promising results, the complexity may become an issue when dealing with larger datasets or models. Further exploration of scalability and efficiency would be beneficial.\n\nGeneralizability: The paper mainly focuses on LLMs and downstream text classification tasks. Extending the evaluation to other types of tasks and models would enhance the generalizability of the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for carefully reviewing our work and valuable suggestions. We believe the following point-to-point response can address all the concerns:**\n\n---\n***Q1: The complexity of the search strategies.***\n\n(1) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of 1/2*N^2 is not a common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [1], the number of demonstrations is fixed at 5. As a result, the overall computation cost remains acceptable, especially when compared to other existing methods [2,3,4] among the typical methods detailed in the related work.\n\n(2) Moreover, our method does not require re-selection of demonstrations for different test samples.\nConsequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with a larger number of test samples. \n\n(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search on a subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore. \n\n(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of O(n!). The smaller training set used in our experiments is due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.\n\n---\n***Q2: Other types of tasks.***\n\nWhile the current version of our method is designed for classification tasks, we acknowledge the potential for its extension to other scenarios.\n\n(1) The limitation preventing direct deployment in generation tasks lies in the absence of an appropriate method to measure the fairness of sentence-level generation.\n\n(2) Similarly, for measuring the calibration of large language models, such as ECE, it can only be calculated for classification tasks [1].\n\n(3) However, we believe that once the community develops techniques to measure the predictive bias of sentences, our proposed strategies can be naturally extended to long sentence generation tasks. This presents an exciting avenue for future research.\n\n\n---\n***Q3: How sensitive are these strategies to the choice of hyperparameters?***\n\nIn this paper, we reduce the influence of hyperparameters by minimizing the need for manual hyperparameter tuning. \n\n(1) In G-fair-prompting, the number of demonstrations is automatically determined, indicating that no new demonstrations are added once it is determined that adding more demonstrations does not lead to improvement. \n\n(2) As for the threshold for fairness improvement, we fix it with 0, indicating that new demonstrations are added to the prompt if they are deemed helpful. \n\n(3) While these two aspects could be manually adjusted to make the method more flexible. To ensure the stability of the experimental results, we avoid deliberately setting these hyperparameters.\n\n---\n\n[1] OpenAI. GPT-4 Technical Report. URL: https://arxiv.org/pdf/2303.08774v3.pdf\n\n[2] Yao Lu. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2021. \n\n[3] Claudio Gentile. Fast rates in pool-based batch active learning. In ICML 2022.\n\n[4] Jiachang Liu. What makes good in-context examples for gpt-3? In ACL, 2022.\n"}]

Author rebuttal:

Meta-review: This work proposes two new algorithms to select examples to be used in few-shot learning. Example selection is quite important for few-shot learning and this work shows that selecting examples that lead to a distribution over classes that is as close to uniform as possible increases classification performance. The two proposed variants are to select the top k examples that deviate the least from the uniform distribution and to greedily add examples that reduce deviation the most. Good results are shown on open source LLMs. The word 'fairness' is misleading in this paper, it would be better to use a different word such as entropy.

Strengths:
- ICL is an important area and this work adds to the understanding of how examples selection affects ICL performance.
- The work is well written and presentation is good.
- The idea that higher entropy correlates with higher accuracy for prompting appears to be novel.
- The approach is simple to implement and yields good results.

Weaknesses:
- The term 'fairness' is used in a misleading way.
- The method is only applicable to multi-class classification tasks and cannot be generalized to generative tasks.
- Some claims in the paper are too strong.
- The G-fair prompting algorithm may be impractical.
- Discussion of experimental results is a bit vague.
Decision: Accept (poster)