Title: Fairness-guided Few-shot Prompting for Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language models

prompts

classification
Abstract: Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper,  we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner.

Fairness-guided Few-shot Prompting for Large Language Models

Abstract

Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper, we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner. Code is available at: https://github.com/MaHuanAAA.

1 Introduction

Large language models (LLMs), such as GPT-3 [1] and BLOOM [2], have demonstrated remarkable ability in performing in-context learning (ICL) on downstream tasks. ICL refers to the process of conditioning an LLM to solve various downstream tasks using prompts constructed from a few demonstration input-output pairs [3] (i.e., few-shot prompting). Despite its impressive performance, prior research has shown that ICL suffers from high instability due to variations in the choice of in-context demonstrations, demonstration order, and prompt formats [4, 5]. Therefore, constructing an appropriate prompt has been identified as a critical factor for improving the performance of ICL [6].

Previous research studies this problem typically from two directions: (1) prompt tuning in the embedding space [7, 8, 9, 10, 11], (2) prompt searching in the text space [4, 12, 13, 14, 15, 16]. The key idea of prompt tuning is to inject task-specific embedding into hidden layers and then tune these embeddings using gradient-based optimization [8, 15]. However, these methods require to modify the original inference process of the model, which is impractical for the case of black-box LM services

such as GPT3 and ChatGPT [17]. Furthermore, prompt tuning introduces additional computational and storage costs, which is typically expensive for LLM. A more feasible and efficient way is to optimize prompting via searching approximate demonstration samples and ordering in the original text space [4, 15]. Bunch of works are presented to constructs prompts from either "global" or "local" views. On the one hand, global-view based methods typically optimize the different elements of the prompt as a whole, with the aim of achieving superior performance. For example, one approach, as described in [14], constructs a search procedure that leverages the overall diversity of demonstrations. Another approach [4] attempts to optimize the ordering of the entire set of demonstrations to achieve better performance. In contrast to the global view, local-view based methods optimize each individual demonstration by designing different heuristic selection criteria such as prior work KATE [15]. These methods have achieved impressive improvements on a wide range of tasks. However, most of them still suffer from the following limitations: (1) Most of current research mainly focuses on searching prompts along a single dimension, such as example selection or order. However, the overall influence of various dimensions on the performance remains unclear. (2) These methods are typically based on heuristic criteria, and there is a gap between them and actual performance. A unified view that explains how these methods work is needed. (3) More importantly, existing methods optimize prompts globally or locally, which may lead to suboptimal performance.

In this paper, we revisit this problem from the perspective of predictive bias . We find a key insight that the quality of a given prompt depends on its inherent bias. Based on this insight, we propose a surrogate metric based on predictive bias for evaluating the quality of prompts. This metric allows us to evaluate a prompt in a single forward process without an additional development set. Specifically, we apply a given prompt to a "content-free" input and expect the model output an uniform predictive distribution (a content-free input contains no useful information). Therefore, we employ the uniformity of the predictive distribution to characterize the bias of a give prompt. This shares a similar idea to the prior work which uses this metric to calibrate the model output [18]. In contrast to this work which mainly focus on using this metric for calibration when the prompt is fixed, we further explore its usage in automatically searching an approximate prompt. Moreover, through extensive experiments, we empirically validate the correlation between the inherent bias of a given prompt and its quality measured by the average task performance on a given test set (see Fig. 2).

Moreover, this bias-based metric allows us to build prompting optimization techniques in a "local-toglobal" manner. We present two novel strategies for efficiently searching high-quality prompts in a bias-guided way: (1) T-fair-Prompting (2) G-fair-Prompting. We focus on a general setting where a labeled set with size N is given. The goal of our strategies is to perform combinatorial optimization over this set to find near-optimal prompts (i.e., select demonstrations and their orders). Specifically, T-fair-Prompting uses an intuitive way that first computes the bias of each single demonstration (i.e., one-shot prompting) and then select the top-k fair demonstrations to form the final prompts. This strategy can be efficiently done with a complexity of O ( N ) . Note that T-fair-Prompting is based on the assumption that the optimal prompt is usually constructed from demonstrations with the smallest individual bias. However, this may not hold true in real situations and often leads to sub-optimal solutions. Therefore, we further introduce G-fair-Prompting to improve the search quality. G-fair-Prompting follows the normal procedure of the greedy search which finds the optimal solution by making locally optimal choices at each step. At each step of the algorithm, the selected demonstration is the one which makes the updated prompts achieves the best fairness score.This strategy trades off the quality of the search with the worst-case time complexity. By accepting a higher worst-case time complexity of O ( N 2 ) , the search quality is significantly improved. Note that G-fair-Prompting works from a local to global perspective, wherein bias of individual samples are considered in the early stages while the later stage focus on the reduction of global predictive bias.

To evaluate the effectiveness of our strategies, we conduct extensive experiments with current mainstream models, such as GPT-3 [1], on various downstream tasks. Our results indicate that our method can significantly enhance the model's in-context learning performance in an effective and interpretable manner. The overall contribution is summarized as follows:

We introduce to use the predictive bias to assess the quality of a given prompt in an efficient and development set independent way and the empirical effectiveness of this metric is comprehensively validated.

Based on the above idea, we propose two efficient and effective strategies, namely, T-fairPrompting and G-fair-Prompting to optimize the prompts.

(a) Selection

(b) Selection (cal)

(c) Permutation

(d) Permutation (cal)

The effectiveness of these two strategies are validated on various LLMs ranging from GPT-series models to LLaMA family [19] released by Meta recently. Consistent relative improvements of over 10% have been observed over different downstream tasks in contrast to SOTA methods.

Relation to Calibration-before-use: Our paper shares a similar metric with cal-before-use [18] to asses the predictive bias of a given prompt. However, the prior approach aims to use this metric to calibrate the output, which can be still easily affected by the quality of the used prompt (more results can be found in Table 2). In contrast, our research aims to find a near-optimal prompt on the original space to improve the model's performance, without requiring any post-adjustment to the output of the model. Moreover, we have firstly empirically validated the connection between predictive bias and the final task performance as shown in Fig. 2, which has not been studied in [18]. Through experiments, we have discovered that, even without calibration, the prompt selected by our method can outperform a randomly selected prompt with calibration.

2 Related Work

In-context Learning Previous research, as cited in [1, 20], has demonstrated that Large Language Models can complete tasks with zero- or few-shot learning using in-context learning. LLMs perform well with an appropriate prompt. However, recent works [4, 18] has shown that the performance of LLMs is affected by the prompt used. Therefore, determining the optimal prompt is a crucial and fundamental research area.

Original space searching A more intuitive approach for determining the best prompt is to search in the original space by selecting or reordering the prompt sentences entered by users. The searching can be concluded in two perspective. · Global view : A naive strategy is to enumerate all candidates to find the prompt that can achieve the best performance on validation set, but this strategy is computationally expensive since its complexity is ∑ n k =1 C k n k ! considering to demonstration selection and order permutation, where k represents the number of demonstrations selected, and C signifies the combinatorial function. Zhang et al. [12] find that errors frequently fall into the same cluster, where each cluster contains similar questions, so they proposed a diversity-guided searching strategy to select diverse demonstrations. In addition to demonstrations selection, Lu et al. [4] have identified the impact of the prompt order on the results. They found the best sequence which yields the most diverse prediction results on the probing set by generating a probing set through LLMs. However, this method is also computationally expensive, and it may be difficult to ensure that the generated probing set is sufficiently balanced. · Local view : Previous studies [13] show that reducing the model's uncertainty helps improve the model's performance, and [14] propose Active Prompting to select demonstrations according to the uncertainty of LLMs. KATE [15] selects the prompt based on the distance amongst embeddings, with the goal of selecting the closest example. However, this method ignores the influence of the order of the examples and requires access to sentence embeddings. [16] demonstrate that LLMs can be easily distracted by irrelevant context, accordingly they identify several approaches for filtering out irrelevant information in context.

In the realm of original space searching, most of the current methods tend to focus solely on the influence of a singular factor (highlighted above) on performance, utilizing heuristic metrics to select context demonstrations that perform well according to this criterion. While these investigations certainly bring benefits to the community, they lack a comprehensive consideration of both local and

(a) AGNews (BLOOM 176B)

(b) AGNews (LLaMA 13B)

(c) AGNews (LLaMA 65B)

(d) TREC (BLOOM 176B)

(e) TREC (LLaMA 13B)

(f) TREC (LLaMA 65B)

global perspectives. The method proposed offers a metric to select context demonstrations from the perspective of predictive bias, which naturally facilitates a transition from local view to global view.

3 Revisiting the Sensitivity across Demonstrations

In this section, we will clarify the notations and the templates used in this paper. Then, we will demonstrate some brief empirical results to show how different demonstration construction factors (e.g., example selection and order) affect performance. We further introduce the definition of predictive bias/fairness of a given prompt and show its connection to the predictive performance on different downstream tasks.

3.1 Notations

We consider a training set consisting of N samples S = { ( x i , y i ) } N i =1 , where x i is the sentence and y i ∈ Y is the label of the i th training sample, and Y is the space of all labels for the task. We use a template Γ( · ) to transform these sentences and labels into natural language space (i.e., prompt construction). Take an instance from the AGNews dataset [21] for example, we have x i = "Cubans Risking Life for Lure of America." , y i = "World" , and Γ( x i , y i ) is "Article: Cubans Risking Life for Lure of America. Answer: World" . We concatenate these demonstrations to form a prompt ρ , which by default is ρ = Γ( x 1 , y 1 ) ⊕ · · · ⊕ Γ( x n , y n ) , where ⊕ indicates sentences combination option. At test time, we append the prompt ρ with τ = "Article: <test sentence>. Answer: " and feed it to a large language model M . The predicted class is given by:

where M ( y | ρ ⊕ τ ) indicates the probability predicted by LLM, and the probability is normalized to fit the task. We denote the predictive distribution by ˆ P ( x ) := { ˆ p ( y | ρ ⊕ τ ) | y ∈ Y} . In this paper, we focus on evaluating the instability caused by demonstrations, and we fix the prompt template following prior work [18].

3.2 Stability of Few-shot Prompting

As demonstrated by prior research, the few-shot prompting technique is highly susceptible to a variety of factors, including the selection and order of demonstrations [4, 18]. In this study, we delve deeper into the stability of few-shot prompting, specifically focusing on the recently released LLaMA family

by Meta [19]. Additionally, we evaluate the stability of LLaMA models calibrated using the current state-of-the-art method [12, 15].

To elucidate the impact of demonstration selection, we select four demonstrations for each different seed and randomly sample an order for each combination. Subsequently, we present the performance on AGNews in the form of a boxplot, which displays the data distribution based on a five-number summary (minimum, first quartile [Q1], median, third quartile [Q3], and maximum). As shown in Fig.1(a)(b), the accuracy demonstrates significant variability across various demonstrations. The detailed settings please refer to Appendix A.7.

To investigate the influence of permutations, we examine all possible permutations of four fixed demonstrations, resulting in 4! distinct candidates. Fig.1(c)(d) also reveals a high degree of variance. While post-calibration contributes to mitigating instability, it is essential to note that the model remains sensitive even after post-calibration. This finding underscores the importance of meticulous demonstration selection. In subsequent experiments, we discover that our approach can be employed to further enhance the performance of the calibrated model.

3.3 Predictive Bias of ICL

As demonstrated in the preceding discussion, the performance of ICL is significantly impacted by various factors such as demonstration, permutation, and selection (refer to Appendix A.4 for additional information). Consequently, devising an efficient method for constructing an appropriate prompt with near-optimal performance is a crucial step in deploying LLMs for diverse downstream tasks. As outlined in the introduction, numerous studies aim to optimize prompts in ICL. This paper further investigates this issue through the lens of predictive bias, which refers to the discrepancy between targeted classes. 2

To achieve this, we initially introduce an efficient technique to assess the inherent predictive bias of a given prompt, drawing inspiration from previous work [18]. We construct a training set-independent metric to measure predictive bias as follows: first, we merge the provided prompt with "semantic-free" test sample information (e.g., "[N/A]", denoted by η ) and obtain the LLM's predictive distribution for this sample. Ideally, the predictive distribution should closely resemble a uniform distribution, as the test sample lacks semantic information. In this paper, we employ entropy as a measure of predictive bias, defined as:

Previous studies have utilized this metric to calibrate the model's output. In this paper, we conduct a comprehensive examination of the relationship between predictive bias and overall performance. Specifically, in a scenario with four training samples (due to the time-consuming nature of enumerating all prompt cases for a larger number), we enumerate all possible combinations and permutations of demonstrations for various datasets and LLMs. Subsequently, we arrange all candidates in descending order based on fairness, where an "index 0" denotes the prompt with the highest fairness. We perform experiments using five different seeds, resulting in training sets comprising distinct demonstrations while maintaining the test samples with seed 0. Fig. 2 displays the results for different models, revealing a strong correlation between the model's performance and fairness score (i.e., fairer prompts yield better performance). The red star, referred to as the "Oracle" represents the optimal average performance, which consistently correlates with higher fairness. This observation prompts us to enhance the ICL performance by identifying the fairest prompt.

Nevertheless, discovering the fairest demonstration combination proves to be a formidable challenge, given the existence of ∑ N k =1 C k N k ! distinct candidates. As the size of the training set increases, this task becomes intractable. In order to tackle this problem, we propose two efficient strategies for approximating the most suitable demonstrations in the subsequent section.

1

Review: … great piece of

…

Sentiment: positive

1

2

2

1

Review: … fails on its own

…

Sentiment: negative

Review: … great piece of

…

Sentiment: positive

<

Demonstrations

>

<

Demonstration1

>

fair

↑

fair

↑

fair

↓

G-fair-Prompting

Demonstrations

⋮

⋮

1

Review: … great piece of

…

Sentiment: positive

1

Review: … fails on its own

…

Sentiment: negative

k

⋮

1

T-fair-Prompting

1

2

k

n

Fairness Ranking

Unique

⋮

Evaluation

s

⋮

1

<

Demonstrationk

>

<

Demonstration1

>

⋮

4 Fairest Prompt Search

Drawing upon the aforementioned observations, we propose two strategies aimed at identifying the most fair prompt, which have been empirically demonstrated to achieve superior performance. Let us consider a training set S comprising N samples; the goal of these search strategies is to select a subset of samples from the training set and construct the context in a specific order so as to optimize the fairness criterion in Eq. 2.

In an ideal scenario, we would consider the factors of demonstration selection and order permutation by examining ∑ N k =1 C k N k ! distinct candidates, which enumerates all possible situations. However, evaluating every candidate is infeasible, as demonstrated when N = 8 , yielding over 10 6 candidates. In this paper, we introduce two search strategies to reduce computational cost: T-fair-Prompting and G-fair-Prompting. The T-fair-Prompting strategy decreases complexity from Θ( ∑ N k =1 C k N k !) to Θ( N ) , but its performance hinges on the selection of k and may be unstable when an unsuitable value of k is chosen. As a result, we propose an additional greedy search strategy, termed G-fair-Prompting, which lowers complexity to O ( N 2 ) and offers a superior approximation of the oracle solution. Fig. 9 visualizes the computational costs over different training set size.

4.1 T-fair-Prompting

The central idea of T-fair-Prompting (top-k) is founded on the heuristic understanding that the fairest prompt usually consists of demonstration samples with reduced individual biases. Consequently, T-fair-Prompting constructs the prompt through a two-stage process. Initially, the prediction bias is assessed when the prompt is formulated using individual demonstrations. Subsequently, the topk fairest demonstrations are chosen and employed to prompt the LLM. It is important to note that fairer demonstrations are likely to be situated towards the end of the sequence, as the generation is more influenced by proximate demonstrations, in accordance with prior research [18]. A comprehensive description of the process is presented in Algorithm 1, while a visual representation can be found in Fig. 3. Specifically, when k is equivalent to the size of the training set, the method degrade to a search for the optimal order of demonstrations. Nevertheless, T-fair-Prompting is heavily reliant on the chosen value of k . More crucially, T-fair-Prompting addresses this issue through a purely local perspective, thereby neglecting considerations from a global standpoint, which typically results in sub-optimal outcomes. As a result, we subsequently introduce the G-fair-Prompting method, which operates in a local-to-global fashion, as described below.

4.2 G-fair-Prompting

The G-fair-Prompting (greedy search) algorithm adheres to the standard procedure of greedy search, which seeks the optimal solution by making locally optimal choices at each stage. In each step of the algorithm, the chosen demonstration is the one that allows the updated prompts to achieve the highest fairness score. This strategy balances the quality of the search with the worst-case time complexity. By accepting an increased worst-case time complexity of O ( N 2 ) , the search quality is significantly enhanced. It is important to note that the G-fair-Prompting algorithm operates from a local to global perspective as shown by Algorithm 2. During the initial stages, the bias of individual samples is taken into account, while the later stages focus on reducing global predictive bias. Specifically, at each step, we insert a new demonstration Γ( x i , y i ) from the remaining demonstration set S ′ (ensuring demonstrations are not repeated) at the beginning of the current context ρ and select the demonstration that maximizes the fairness improvement. Formally, at step 9 in Algorithm 2, the inserted demonstration should satisfy the following criterion:

Algorithm 1 T-fair-Prompting

1: Given: training set S = { ( x i , y i ) } N i , pretrained LLM M , transformation template Γ( · ) , and context-free input η

2: Initial prompt ρ

3: for ( x i , y i ) in S do

4: Inference ˆ P ← { ˆ p ( y | Γ( x i , y i ) ⊕ η ) | y ∈ Y} via M

5: Calculate the fair (Γ( x i , y i )) according to Eq. 2

6: end for

7: Sort fair i =1 , ··· ,N (Γ( x i , y i )) in descending order

8: for d in 1 , · · · , k do

9: Insert the most d fair demonstration at the head of ρ

10: end for

11: return ρ

5 Experiments

5.1 Experimental Setup

Models. There are a large number of available LLMs (Appendix A.2) including open-source models and black-box cloud API. Recently, Meta has released their powerful pretrained LLMs, LLaMA. LLaMA models with 13B parameters can achieve comparable performance in contrast to BLOOM and GPT-3 with much larger model size. In this paper, we evaluate the effectiveness of our method on BLOOM (176B) and LLaMA models of different sizes. We have opted to employ LLaMA (65B) as a substitute for GPT-3 in our experiments, since oepnai strictly restricts the API access to certain areas.

Datasets. We conducted experiments on various text classification datasets [21], namely SST-2, AGNews, CoLA, TREC, and RTE. Furthermore, the maximum input length of LLaMA is 512, and the sentences in RTE are too long for LLaMA. The task descriptions and statistics are available in Table 6 in Appendix.

5.2 Results

We conducted experiments on different settings and reported the results of five runs. We compared our method with the diversity-guided searching strategy proposed by Zhang et al.[12] (Global view) and the similarity-guided searching strategy proposed by Liu et al.[15] (Local view). Note that methods

Algorithm 2 G-fair-Prompting

1: Given: training set S = { ( x i , y i ) } N i , pretrained LLM M , transformation template Γ( · ) , and context-free input η

2: Initial prompt ρ

3: while S is not null do

4: for ( x i , y i ) in S do

5: ρ tmp ← Γ( x i , y i ) ⊕ ρ

6: Inference ˆ P ←{ ˆ p ( y | ρ tmp ⊕ η ) | y ∈ Y} via M

7: Calculate the fair ( ρ tmp ) according to Eq. 2

8: end for

9: Insert the demonstration that can improve fairness best and remove it from S

10: Stop searching when fairness can't be improved

11: end while

12: return ρ

based on local view are time-consuming since they require searching different demonstrations for every test example. Table 1 shows the performance of the different strategies, where "Random" indicates the average accuracy for enumerating all situations, "Diversity" and "Similarity" indicate demonstrations are selected according to diversity and similarity (details please refer to Appendix A.6), respectively. For each dataset, we set the size of the training set to 4. "Diversity" and "Similarity" select 4 from 16 demonstrations, as they need more candidates. The baseline is expensive to compute since enumerating all candidates for 4 demonstrations in RTE on BLOOM will take more than 120 NVIDIA A100 GPU hours. We enumerate all candidates for the training set with 4 demonstrations on different models, as shown in Fig. 2. The results on models whose parameters less than 13B are shown in Table 4 (i.e., GPT2-XL (1.5B), LLaMA (7B), and LLaMA (13B)).

G-fair-Prompting can reach a close approximation of enumeration. To evaluate whether the G-fair-Prompting (Greedy) method can approximate the best performance of enumerating all candidates, we marked the performance of G-fair-Prompting with a green star (representing the closest value to averaged accuracy of G-fair-Prompting on the line). We found that G-fair-Prompting can achieve a very close approximation to enumeration. As shown in Fig. 2, most prompts searched by G-fair-Prompting achieved a top 20% ranking, and on BLOOM (176B), G-fair-Prompting almost found the most fair prompt.

G-fair-Prompting outperforms T-fair-Prompting. As shown in Table 1, although T-fairPrompting achieves better performance compared with random selection, G-fair-Prompting consistently outperforms T-fair-Prompting. Furthermore, Top-2 significantly outperforms Top-4 in most cases (over 5% ), indicating that the number of demonstrations selected is crucial. Overall, the results demonstrate that G-fair-Prompting achieves satisfactory performance with only a slight additional cost.

Compared with SOTA methods. We compared our methods with several State-of-the-Art (SOTA) methods, including diversity-guided and similarityguided techniques. We observed that our greedy approach outperforms most of these SOTA methods in most situations, and the improvements of over 10% are observed on dataset TREC. The similarity-guided method, on the other hand, achieved the best performance on the topic classification task (AGNews).

This is because it searches for a unique prompt for every different test example based on the dis-

tance between the embeddings of the training samples and the test example. This strategy selects demonstrations with labels that are the same as the test samples, and Language Models (LLMs) tend to predict biased predictions toward the labels that always appear in the context. However, the similarity-guided method may prove inadequate when applied to other tasks. Specifically, the similarity-guided strategy exhibits lower performance compared to random selection in QC and acceptability tasks. Furthermore, the G-fair-Prompting approach may occasionally falter when the model's sensitivity to the task is not immediately evident, as observed in the acceptability task on BLOOM (depicted in Fig. 4). Note that the training set size of compared methods is 4 × larger than ours.

· Comparison with Calibration Method. Post-calibration [18], can enhance the accuracy of a given prompt in most cases. However, when the selected prompt is of poor quality, the performance may remain inadequate even after calibration. We compared the performance of G-fair-Prompting with random selection with calibration (averaged on all candidates), and found that G-fair-Prompting can outperform random selection with calibration in most situations. For example, on the topic classification task, G-fair-Prompting achieves the best performance on most models. Moreover, we find that post calibration can harm the performance of the model and it occurs significantly times, so it is worthwhile to reconsider the influence of manipulating the model's probability directly.

Post calibration [18] can improve the accuracy of a certain prompt (in most cases), but when the selected prompt is very poor, the performance is still very poor even after calibration. We conducted experiments (Table 2) to compare the performance of G-fair-Prompting and random selection with calibration ("Average" and "Worst" indicate averaged accuracy and worst performance on all permutations of training examples), and observed that G-fair-Prompting outperforms random selection with calibration in most case. For instance, on the CoLA, G-fair-Prompting exhibited superior performance on most models. Additionally, we find that post-calibration could negatively affect the model's performance in many scenarios while it sometimes can improve the performance significantly even on selected prompts, for example, an improvement by 10% is observed on BLOOMTREC. For more detailed discussions, please refer to Appendix A.5. Hence, it is crucial to reconsider the impact of directly manipulating the model's probability.

6 Conclusion

In this paper, we revisit the sensitivity of large language model across prompts, and analyse the issue from a predictive bias perspective. Accordingly, we employ a "content-free" strategy as a metric termed as fairness to evaluate the predictive bias of a fixed prompt and show that model's performance is highly consistency with fairness. Then, we propose two strategy to search the most fair prompt in the original space. We conduct extensive experiments on current famous LLMs, and validate the effectiveness of the proposed strategy. Moreover, in addition to fairness adopted in this paper, there would be more metrics for prompt searching in the future for different scenarios.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant No. 61976151 and 61925602), and Fu is supported by A*STAR Central Research Fund. The project was conducted during the internship in AI Lab, Tencent.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , volume 33, pages 1877-1901, 2020.

Bloom: A 176b-parameter open-access multilingual language model. https:// huggingface.co/bigscience/bloom .

Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 , 2019.

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL , 2021.

Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. Improving few-shot performance of language models via nearest neighbor calibration. arXiv preprint arXiv:2212.02216 , 2022.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation, 2021.

Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 , 2021.

Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. Warp: Word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121 , 2021.

Guanghui Qin and Jason Eisner. Learning how to ask: Querying lms with mixtures of soft prompts. arXiv preprint arXiv:2104.06599 , 2021.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10385 , 2021.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 , 2022.

Claudio Gentile, Zhilei Wang, and Tong Zhang. Fast rates in pool-based batch active learning. arXiv preprint arXiv:2202.05448 , 2022.

Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-ofthought for large language models. arXiv preprint arXiv:2302.12246 , 2023.

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt3 ? arXiv preprint arXiv:2101.06804 , 2021.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00093 , 2023.

https://openai.com/blog/chatgpt .

Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML , 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report , 2018.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax , May 2021.

https://huggingface.co/EleutherAI/gpt-neox-20b .

OpenAI. Gpt-4 technical report, 2023.

A Appendix

A.1 Pretrained Large Language Models

Neural autoregressive language model (LMs) are designed for next token prediction to predict the probability distribution over the next token after a sequence of tokens input, and pre-trained LMs show their superior performance since they are trained on various programming languages and a large-scale curated dataset. Training large natural LMs are very expansive and time-consuming process since they always have billions of parameters, which limits the development of LMs. Fortunately, many pre-trained LMs are open access or limited access, which promotes researchers to pool their time and makes the resources to collectively achieve a higher impact. EleutherAI makes the GPT-J [22] and GPT-Neox [23] public available on Hugging Face. GPT-3 [1] is limited access in OpenAI which can be used by researchers for a fee, and another large open-science open-access multilingual language model named Bloom [2] is provided by BigScience.

A.2 Open Access Models

A.3 Additional Figures on Different Settings

In additional to the Fig. 2, we shows the performance on different models for enumerating all candidates, note that the shadow indicates the half value of standard deviation for clear presentation since the variance is very high for LLMs.

A.4 Accuracy Varies with demonstrations

Accuracy Varies with Example Amount Demonstrations play an important role in imparting task-related information to language models through in-context learning. Then, the question arises does a larger number of demonstrations necessarily equate to better performance? To answer this question, we evaluated performance in terms of accuracy by gradually increasing the number of demonstrations. We set ρ = Γ( x 1 , y 1 ) ⊕··· ⊕ Γ( x k , y k ) , where k = 1 , · · · , n , and demonstrations are erased with k decreasing from n to 1 . Intuitively, accuracy would vary highly across different numbers of demonstrations, and the phenomenon is observed in Fig. 6a. To our surprise, however, erasing some demonstrations can result in a better performance. Removing some demonstrations can perform better and sometimes GPT-3 achieves best accuracy when there is only a few demonstrations remaining. This highlights the importance of considering the appropriate number of demonstrations.

(a) AGNews (GPT2-XL 1.5B)

(b) TREC (GPT2-XL 1.5B)

(c) RTE (GPT2-XL 1.5B)

(d) AGNews (LLaMA 33B)

(e) TREC (LLaMA 33B)

(f) SST-2 (LLaMA 33B)

(a) Varying amount of examples

(b) Permutation

(c) Select different examples

Example Order The performance of a model is sensitive to the order of the demonstrations, as has been discussed in [4]. Even when the demonstrations are the same, different permutations of the demonstrations can result in vastly different outcomes. As there are n ! possible permutations, we introducing a strategy of permuting the demonstrations by circularly shifting the index of the demonstrations. The demonstration can be represented as ρ = Γ( x k +1 , y k +1 ) ⊕··· ⊕ Γ( x n , y n ) ⊕ Γ( x 1 , y 1 ) ⊕··· ⊕ Γ( x k , y k ) .As shown in Fig. 6b, the accuracy varies highly with permutation which consistent with the observations in [4].

Example Selection In this paper, we find which demonstrations are selected is influence the model extremely. This scenario can be described as selecting k demonstrations in n training samples. In Fig. 6c, we only select one example for demonstration to ablate the impact of demonstrations order, and the accuracy also varies highly with different example selected. In this work, we only detail evaluate the proposed probing method on the erasing demonstrations and permutation, although our method improves by 20% in the setting of example selection on SST-2 (GPT2-XL), because selecting k demonstrations on a set with n training samples can't be regarded as k -shot learning in the strict sense.

A.5 Relationship between with- and without-calibration

· G-fair-Prompting without post-calibration outperforms random demonstrations after postcalibration. Based on Table 1, it is apparent that G-fair-Prompting outperforms random selection prior to post-calibration. This leads to a natural question: do prompts with better performance before calibration also indicate better performance after calibration proposed by Zhao et al. [18]? To investigate the relationship between performance with- and without-calibration, we calculated the Pearson correlation coefficient between the accuracy with- and without-calibration Pearson ( acc w/o , acc with ) . A positive coefficient value suggests that a prompt with high accuracy before calibration has a

Figure 7: Illustration of accuracy relationship between with- and without calibration when Pearson ( · ) is positive.

higher likelihood of achieving higher accuracy after calibration than other prompts. We take the topic classification task on LLaMA(65B) for illustration to show the relationship between with- and without calibration when Pearson is positive in Fig.7. Table 5 presents the Pearson correlation coefficient on accuracy of permutation and G-fair-Prompting after calibration. The majority of Pearson correlation coefficients were found to be positive, indicating that prompts with better performance before calibration have more potential to perform well after calibration. Furthermore, our results on the LLaMA family reveal that the larger the model, the stronger the correlation between performance with- and without-calibration. For instance, the value of the Pearson correlation coefficient increases from 0 to 0 . 7 as the model size increases.

Theorem A.1. Suppose the performance of the model under certain prompts with- and withoutcalibration is positively correlated, i.e., Pearson ( acc w/o , acc with ) > 0 , if we can assure E ( acc Selected w/o ) > E ( acc Random w/o ) , then we have E ( acc Selected with ) > E ( acc Random with ) .

As analysed in Theorem A.1, if we can find a prompt with high accuracy before calibration, we have a higher likelihood of achieving higher accuracy after calibration than random selection. Our approach consistently identifies an appropriate prompt, as evidenced by the results in Table 1. Moreover, the

performance of the model exhibits a positive correlation with and without calibration under certain prompts, as illustrated in Table 5. Therefore, our method is more likely to enhance calibration performance.

A.6 Diversity and Similarity

Diversity indicates the demonstrations are selected according to the diversity of the embeddings of all samples [12]. Specifically, we select the 4 most diverse samples as demonstrations by k-means clustering on training set consists of 16 samples.

Similarity indicates the demonstrations are selected according to the similarity of the embeddings of all samples [15]. Specifically, we select the 4 most similar samples as demonstrations according to Euclidean distance from the training set consists of 16 samples. Note that demonstrations for different test samples are different for best performance. A larger training set may result in a better performance, but we set the size of training set as 16 for a fair comparison.

C

Test sample

Diversity

Similarity

A.7 Details for experiments

In Fig. 1, we show the high instability due to high variations in demonstrations selection and order for both with- and without-calibration [18]. Specifically, for left two figures, we sample 4 different demonstrations randomly from dataset AGNews, and estimate the influence of demonstration selection. On the other hand, the right two figures show the instable performance due to permutation when the demonstrations are fixed. Specifically, we randomly sample 4 demonstrations and estimate the performance with all possible orders.

A.8 Complexity of different strategies

A.9 Limitations.

While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As a result, it may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.

B Clarifications After Rebuttal

Q: Computational cost of using G-fair Prompting becomes non-trivial as N goes larger.

(1) We acknowledge the concern regarding the computational cost of G-fair Prompting as N increases. However, we would like to emphasize that the worst-case computation cost of 1 / 2 ∗ N 2 is not a common scenario. In practice, only a few demonstrations are required. For example, in GPT-4 Technical Report [24], the number of demonstrations is fixed at 5. As a result, the overall computation cost remains acceptable, especially when compared to other existing methods [15] among the typical methods detailed in the related work.

(2) Moreover, our method does not require re-selection of demonstrations for different test samples. Consequently, the inference cost for selecting demonstrations is minimized, particularly when dealing with a larger number of test samples.

(3) In the worst-case scenario, even with an infinitely large training set, our model can efficiently perform Greedy search on a subset, consistently outperforming random demonstration. The additional computational cost depends on the size of the search space that users want to explore.

(4) In contrary, the naive strategy, i.e., enumerating all possible situations, has an exponential time complexity of O(n!). The smaller training set used in our experiments is due to considering the computational cost of the comparative methods in the report, rather than any limitations of the proposed method.

Q: One can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.

(1) The proposed simple version T-fair-prompting serves as a time-efficient alternative of G-fairprompting, while still maintaining competitive performance.

(2) Compared with T-fair-prompting, G-fair-prompting offers the advantage of automatically determining the number of demonstrations, providing a more natural and flexible approach.

(3) Therefore, a tradeoff between performance and time cost can be achieved by transitioning between T-fair-prompting and G-fair-prompting. Specifically, we can select the demonstrations order as same as T-fair-prompting and then decide the number of demonstrations by the strategy of G-fair-prompting. The computation cost upper bound of this strategy will be the lower bound of G-fair-prompting while its fairness lower bound will be the upper bound of T-fair-prompting.

Q: Under the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them.

(1) As shown in Fig.2, we have tested different choices of few-shot examples (e.g., [x1, x2]/[x3, x4]) and all possible orderings of them (e.g., [x1, x2, x4]/[x2, x4, x4]). In total, 64 different possible situations were considered when the training set size was 4.

(2) Notably, G-fair-Prompting performs remarkably close to the best performance obtained through enumeration, highlighting the efficacy of our proposed method.

Q: How to choose the semantic-free token without any guidance?

(1) Following previous work [18], three different semantic-free prompts are selected, i.e., ['N/A', '', ' ']. The fairness is then calculated as the average of them.

(2) These tokens were chosen based on their guaranteed performance in previous work [18]. We will provide further clarification on this selection in the final version of our paper.

Q: Would be the proposed method helpful for long-form QA?

While the current version of our method is designed for classification tasks, we acknowledge the potential for its extension to long-form QA scenarios.

(1) The limitation preventing direct deployment in generation tasks lies in the absence of an appropriate method to measure the fairness of sentence-level generation.

(2) Similarly, for measuring the calibration of large language models, such as ECE, it can only be calculated for classification tasks [24].

However, we believe that once the community develops techniques to measure the predictive bias of sentences, our proposed strategies can be naturally extended to long sentence generation tasks. This presents an exciting avenue for future research.

Q: What does 'total cost' in table 'Dataset descriptions' refer to?

'Total cost' indicates the overall computational cost on each dataset including baselines and the proposed methods.

We report the total computational cost on each dataset to provide a more comprehensive description of the datasets used in this paper. We will clarify this in the final version.

Q: The diversity baseline seems similarly competitive to the proposed approach.

Regarding performance, we would like to clarify the following points:

Our method consistently outperforms the diversity baseline across various scales of the LLaMA family datasets.

The similarity in performance between the diversity method and our method on the Bloom model is attributed to the Bloom model's robustness to the transformations of the demonstration. Even random demonstrations can achieve competitive performance with the Bloom model.

It is essential to note that the baselines, including the diversity method, are the latest methods proposed shortly before submitting this paper. They have demonstrated robust performance on multiple tasks.

Q: Could the authors compare using a strategy such as the proposed one with a large language model vs. fine-tuning with a smaller model?

** Note that although the former strategy requires iterating over the entire dataset, the latter strategy only requires performing inference. It does not incur any space cost for storing gradients nor time cost for updating gradients, making this process highly efficient.

We would like to clarify the advantages of utilizing a large language model and comparing it with fine-tuning a smaller model:

Generalization ability of large models: Large language models (LLMs) accumulate significant prior knowledge during pre-training, resulting in better generalization abilities compared to small models fine-tuned for specific tasks. Even if the performance of a large model is not significantly better than that of a dedicated small model on a specific task, the robustness of the large model in open environments, such as out-of-distribution (OOD) scenarios, is significantly superior to that of a dedicated small model [24].

Multiple tasks: A single large model can perform multiple tasks simultaneously. Compared to fine-tuning a small model for each task, a large model can seamlessly switch between various tasks and effortlessly incorporate new tasks.

Time efficiency: Many open-source large models, such as LLaMA, do not require modifications when encountering new tasks. Even equipped by the methods including baselines in this paper, only a few additional inferences are necessary, without any updates to the model or backpropagation of gradients.

Q: Explanation about the text in lines 293-296 and 300.

(1) Line 293-296 Explanation:

The sentence mentioned in lines 293-296 analyzes why the similarity-guided method achieved the best performance on the topic classification task (AGNews). This method selects unique prompts for different test examples based on the distance metric. Consequently, it chooses demonstrations with labels that are the same as the test samples. Large Language Models (LLMs) tend to predict biased outcomes towards the labels that frequently appear in the context of demonstrations . As a result, the similarity-guided method excels on the topic classification task but may perform poorly on other tasks, such as natural language inference.

(2) Line 300 Explanation:

The Bloom model exhibits high robustness to various demonstrations, even random ones, leading to similar performance results between our method and the compared method. It can be observed that the Bloom model's performance remains competitive regardless of the selected demonstrations.

Q: Discussion about limitations.

Due to space limitations, our discussion on limitations is included in Appendix (A.9 Limitations). While the proposed strategy does not require modification of the original inference process of the Language Model (LM), it still necessitates the logits or probabilities of the next token. As a result, it may be necessary to approximate the probabilities in some completely black-box LM services, such as GPT-4.

Q: Figure 2 shows a non-monotonic trend between the bias and predictive quality.

According to Figure 2, the relationship between predictive bias and predictive quality is not strictly monotonic. Instead, the overall trend shows a significant negative correlation. To avoid any misunderstandings, we will use clearer descriptions in the final version of the paper.

Q: The term fairness is slightly misleading.

The fairness defined in our work is defined from the concept of predictive bias, which may not align precisely with the fairness, such as social bias, that is commonly referred to. It is important to note the distinction between fairness in the context of predictive bias and fairness in the context of social bias. Moreover, we highlight that the proposed metric can be extended to analyze social bias, as social bias can also lead to predictive bias [24].

Q: Contribution and difference from [18].

Our paper shares a similar metric with calibration-before-use [18] to access the predictive bias of a given prompt. However, the differences are significant and listed as follows:

(1) The prior approach aims to use the metric for calibrating the output, which can still be influenced by the quality of the used prompt, as evidenced by Table 3.

(2) In contrast, our research focuses on finding a near-optimal prompt on the original space to improve the model's performance, eliminating the need for any post-adjustment to the output.

(3) Additionally, our paper introduces an empirical validation of the connection between predictive bias and final task performance, as depicted in Fig. 2, which is not explored in [24].

(4) The main contribution of our work lies in analyzing the relationship between predictive bias and performance through extensive experiments, and proposing a Greedy research strategy to solve the NP-hard problem.

(5) Our experiments have revealed that even without calibration, the prompt selected by our method outperforms randomly selected prompts, even with calibration.

(6) Moreover, we have observed that [24] can potentially harm the model's performance, which is a practical concern worth considering when manipulating the model's probability directly.

Q: How sensitive are these strategies to the choice of hyperparameters?

In this paper, we reduce the influence of hyperparameters by minimizing the need for manual hyperparameter tuning.

(1) In G-fair-prompting, the number of demonstrations is automatically determined, indicating that no new demonstrations are added once it is determined that adding more demonstrations does not lead to improvement.

(2) As for the threshold for fairness improvement, we fix it with 0, indicating that new demonstrations are added to the prompt if they are deemed helpful.

(3) While these two aspects could be manually adjusted to make the method more flexible. To ensure the stability of the experimental results, we avoid deliberately setting these hyperparameters.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of constructing an appropriate prompt for improving the performance of in-context learning in large language models (LLMs). It introduces a metric based on predictive bias to evaluate prompt quality and empirically demonstrates that prompts with higher bias lead to unsatisfactory predictive quality. Leveraging this observation, the paper proposes a novel search strategy, combining T-fair-Prompting and G-fair-Prompting, to identify near-optimal prompts for enhancing in-context learning performance. The proposed method is extensively evaluated on various downstream tasks using state-of-the-art LLMs, such as GPT-3 and LLaMA. The results show that the method effectively improves in-context learning performance in an interpretable manner.\n\nSTRENGTHS:\nThe paper highlights the importance of prompt quality and introduces a novel metric based on predictive bias to evaluate prompt performance. The proposed method effectively searches for near-optimal prompts by addressing bias issues, leading to improved in-context learning performance.\n\nThe paper conducts comprehensive experiments on mainstream LLMs and various downstream tasks to validate the effectiveness of the proposed method. The results consistently demonstrate performance enhancements, surpassing state-of-the-art methods in most cases.\n\nThe paper presents two search strategies, T-fair-Prompting and G-fair-Prompting, which efficiently optimize prompts by considering bias and fairness. The strategies offer a balance between search quality and computational complexity, addressing the challenges of prompt construction.\n\nThe paper is well-written and effectively conveys the motivation, contributions, methodology, and experimental setup. The algorithms and search strategies are clearly described, allowing for easy understanding and reproducibility.\n\nWEAKNESSES:\nScalability: The paper discusses the computational complexity of the search strategies and their limitations. While the proposed strategies show promising results, the complexity may become an issue when dealing with larger datasets or models. Further exploration of scalability and efficiency would be beneficial.\n\nGeneralizability: The paper mainly focuses on LLMs and downstream text classification tasks. Extending the evaluation to other types of tasks and models would enhance the generalizability of the proposed method.\n\nQUESTIONS:\nHow generalizable is the proposed method beyond text classification tasks? Have you explored its applicability in other types of downstream tasks, such as language generation or sequence labeling?\n\nThe search strategies, T-fair-Prompting and G-fair-Prompting, effectively optimize prompts by considering bias and fairness. However, how sensitive are these strategies to the choice of hyperparameters, such as the number of demonstrations (k) or the threshold for fairness improvement? Have you conducted sensitivity analyses to investigate their impact on performance?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the sensitivity of large language models to prompts and analyzes the issue from a predictive bias standpoint. They utilize a fairness metric called a content-free strategy to evaluate the predictive bias of a fixed prompt. Methodologically, the paper proposes two strategies, T-fair-Prompting and G-fair-Prompting, to search for the fairest prompt and validates their effectiveness through extensive experiments on large-language models.\n\nSTRENGTHS:\n1. The paper exploits the correlation between predictive bias and overall performance and proposes two strategies based on the finding to approximately estimate the fairest demonstration combination.\n2. It is intriguing to observe that fairness-guided selection, rather than diversity or similarity-guided approach, often exhibits better performance.\n\nWEAKNESSES:\n1. The predictive bias-based fairness metric, which is the most key and primary idea, is adopted directly from [18]. The way of using the fairness metric in each T-fair-Prompting and G-fair-Prompting is also straight-forward. I think that the methodological novelty of this paper is limited. \n2. I can't find an ablation study regarding the proposed fairness metric. I am curious about the results and analysis when using other semantic-free prompts, apart from N/A.\n3. When looking at Table 2, there are quite a few results where diversity-guided selection demonstrates higher performance. The overall performance is not particularly impressive.\n\nQUESTIONS:\nPlease provide an in-depth response regarding the weaknesses mentioned above.\n\nLIMITATIONS:\nThe role of few-shot prompting in in-context learning is significant. Therefore, I believe that providing a new insight is beneficial for the in-context learning community. However, I would like to request a sufficient response that adequately addresses the novelty of the proposed work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two new procedures to select examples to be used for few-shot learning (in-context learning) for LLMs. The key idea for example selection is to select those examples that lead to a distribution over classes that is as close to uniform as possible . The two proposed variants (a) select the top k examples which each individually have the least ‘bias’ and (b) greedily add examples that reduce ‘bias’ the most until any additional example would increase it. The authors evaluate the proposed methods on a variety of open source LLMs on multiple tasks, where the proposed methods perform well.\n\nSTRENGTHS:\nIn-context learning, and in particular the selection of effective demonstrations is an important area of study and thus of interest to the research community. \nThe paper is well written and the proposed methods are clearly presented. \nThe evaluation is well chosen to demonstrate the effectiveness of the proposed methods. \nThe proposed methods are a clever idea that is simple yet effective.\n\nWEAKNESSES:\nThe proposed method is limited to multi-class classification tasks, which places a significant limit on the practical use-cases of the proposed method, i.e., it does not apply to summarization, translation, open ended generation etc. \nSome of the claims in the paper are too strong and not supported. For example in the abstract in lines 9-11 the authors claim that “Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality.” Which is clearly not true and not supported by their findings in Figure 2, which shows a non-monotonic trend.\nWhile the writing is generally good, there are a few mistakes across the paper, e.g., line 34 to construct \n\nI think the use of the term fairness is slightly misleading in this paper. This the term ‘fairness’ is associated with many different meanings, but mostly with social aspects, the authors could directly use entropy as that would be a more technical description of their method.\n\nQUESTIONS:\nI would love to see the authors address the usage of their method to other tasks that are not easily cast into a multi-class classification task.\nI would also like the authors to address the use of the term ‘fairness’ and why that is necessary for the paper at hand.\n\nLIMITATIONS:\nThe authors did not address limitations and other impact. Especially, given the term fairness in the title, I would expect the authors to at least discuss the use of the proposed method to address fairness issues and also the potential misuse to introduce unwanted biases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that the accuracy of large language models on various text classification tasks for a given prompt tends to be correlated with the entropy of the output prediction when using that prompt for a context free input: in general, the more uniform the output probability distribution for a given prompt (made up of demonstrations from the dataset), the higher the accuracy of that prompting strategy for the dataset. The authors use this insight to select the demonstrations and show that greedily selecting demonstrations one-by-one can yield competitive performance to other strategies.\n\nSTRENGTHS:\n1. The authors point out that empirical observation that higher entropy in the context-free case is correlated with higher accuracy was first introduced in Zhao et al. 2021. However, showing this correlation holds in the context of prompting is, to my knowledge, novel. \n2. The proposed prompting search strategies are simple to implement and competitive.\n3. The experiments seem pretty extensive, both in datasets and models tested.\n\nWEAKNESSES:\n1. Given the time complexity of the G-fair-prompting algorithm (i.e. O(N^2) where N is the size of the training dataset), the proposed approach does not seem very practical.\n2. The discussion of experimental results is vague at times. For instance, the explanation provided for why the similarity-guided method is best for AGNews (lines 293-296) seems as if it would apply to other tasks as well; a meaningful explanation would be able to differentiate between these settings. Also, it is not clear what \"the model's sensitivity to the task is not immediately evident\" means (line 300). In addition, practical discussion of the proposed approach relative to baselines would be helpful (e.g. when one is preferred over the other).\n\nQUESTIONS:\n1. What does “Total Cost” in Table 1 refer to? What are the time complexities of the G-fair-prompting approach on the models and datasets tested?\n2. The diversity baseline seems similarly competitive to the proposed approach. When would the authors suggest using one over the other?\n3. Could the authors compare using a strategy such as the proposed one with a large language model vs. fine-tuning with a smaller model? Especially since the former requires iterating over the entire dataset anyway, albeit differently than in fine-tuning, when is the former advantageous over the latter?\n4. Could the authors explain the text in lines 293-296 and 300 (as per comment #2 in weaknesses)?\n\nLIMITATIONS:\nThe authors do not directly address limitations, which would be worth mentioning briefly. I did not factor this absence into my review.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFinding an (nearly) optimal prompt is crucial for successful in-context learning (ICL) of large language models (LLMs). It is known that performance of downstream tasks is sensitive to prompts themselves, few-shot examples, and their orders, often requiring large degree of non-trivial tuning. This paper presents a content-agnostic fairness metric that allows users to assess the quality of a given prompt purely in the perspective of predictive bias. Under this metric, the authors propose two methods for optimizing the prompts. Given the few-shot examples for ICL, T-fair Prompting computes the predictive bias of individual demonstrations by feeding each single shot. Then it chooses the user-specified number of examples with the lowest predictive biases. In contrast, G-fair Prompting promotes the greedy local search where it appends one additional shot that shows the best overall fairness. Users can choose one of the two proposed methods considering the tradeoff between algorithmic complexity and performance gain.\n\nSTRENGTHS:\n(1)\tThe paper provides simple yet effective approaches for prompt tuning.\n\n(2)\tBoth T-fair and G-fair Prompting algorithms are natural and practical in the spirit of local individual search and greedy search.\n\n(3)\tThe proposed methods achieve highly competitive accuracies comparing to the current state-of-the-art methods.\n\nWEAKNESSES:\n(1)\tComputational cost of using G-fair Prompting becomes non-trivial as N goes larger even if it uses greedy approach for expanding local bias to global bias. \n\n(2)\tOne can think of even better algorithm that can show benchmark performance that provides upper bound against G-Prompting.\n\n(3)\tNeither guidance for how to choose semantic-free tokens nor sensitivity analysis for various semantic-free candidates are explored.\n\nQUESTIONS:\n(1)\tCan you provide computational cost for running T-fair and G-fair prompting over increasing N? \n\n(2)\tUnder the proposed problem formulation, there will be a natural extension to test both different choices of few-shot examples and all possible ordering of them. Though such experiments would be expansive to navigate combinatorially many configurations, measuring the performance on a limited setting helps understanding the relative performance of T-fair and G-fair Prompting.\n\n(3)\tHow to choose the semantic-free token without any guidance? It is less clear that such token would exist universally across different LLMs across various downstream tasks. In any case, using just one manually-reserved tokens for fairness evaluation is not easily justifiable without sensitivity analysis.\n\n(4)\tPredictive bias would be more useful for solving classification tests by generation (as if the paper tested the sentiment analysis). Would be the proposed method helpful for long-form QA?\n\nLIMITATIONS:\nNo specific points are described or probed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-29"}