SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huan Ma $^{1,2}$ , Changqing Zhang $^{1*}$ , Yatao Bian $^{2}$ , Lemao Liu $^{2}$ , Zhirui Zhang $^{2}$ , Peilin Zhao $^{2}$ , Shu Zhang $^{2}$ , Huazhu Fu $^{3}$ , Qinghua Hu $^{1}$ , Bingzhe Wu $^{2*}$
<sup>1</sup> College of Intelligence and Computing, Tianjin University, Tianjin, China  
<sup>2</sup> AI Lab, Tencent, Shenzhen, China
<sup>3</sup> Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore  
<sup>1</sup> zhanchangqing@tju.edu.cn; <sup>2</sup> bingzhewu@tencent.com
Large language models have demonstrated surprising ability to perform in-context learning, i.e., these models can be directly applied to solve numerous downstream tasks by conditioning on a prompt constructed by a few input-output examples. However, prior research has shown that in-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. Therefore, the construction of an appropriate prompt is essential for improving the performance of in-context learning. In this paper, we revisit this problem from the view of predictive bias. Specifically, we introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. Then we empirically show that prompts with higher bias always lead to unsatisfactory predictive quality. Based on this observation, we propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning. We perform comprehensive experiments with state-of-the-art mainstream models such as GPT-3 on various downstream tasks. Our results indicate that our method can enhance the model's in-context learning performance in an effective and interpretable manner. Code is available at: https://github.com/MaHuanAAA.
Large language models (LLMs), such as GPT-3 [1] and BLOOM [2], have demonstrated remarkable ability in performing in-context learning (ICL) on downstream tasks. ICL refers to the process of conditioning an LLM to solve various downstream tasks using prompts constructed from a few demonstration input-output pairs [3] (i.e., few-shot prompting). Despite its impressive performance, prior research has shown that ICL suffers from high instability due to variations in the choice of in-context demonstrations, demonstration order, and prompt formats [4, 5]. Therefore, constructing an appropriate prompt has been identified as a critical factor for improving the performance of ICL [6].
Previous research studies this problem typically from two directions: (1) prompt tuning in the embedding space [7, 8, 9, 10, 11], (2) prompt searching in the text space [4, 12, 13, 14, 15, 16]. The key idea of prompt tuning is to inject task-specific embedding into hidden layers and then tune these embeddings using gradient-based optimization [8, 15]. However, these methods require to modify the original inference process of the model, which is impractical for the case of black-box LM services
such as GPT3 and ChatGPT [17]. Furthermore, prompt tuning introduces additional computational and storage costs, which is typically expensive for LLM. A more feasible and efficient way is to optimize prompting via searching approximate demonstration samples and ordering in the original text space [4, 15]. Bunch of works are presented to constructs prompts from either "global" or "local" views. On the one hand, global-view based methods typically optimize the different elements of the prompt as a whole, with the aim of achieving superior performance. For example, one approach, as described in [14], constructs a search procedure that leverages the overall diversity of demonstrations. Another approach [4] attempts to optimize the ordering of the entire set of demonstrations to achieve better performance. In contrast to the global view, local-view based methods optimize each individual demonstration by designing different heuristic selection criteria such as prior work KATE [15]. These methods have achieved impressive improvements on a wide range of tasks. However, most of them still suffer from the following limitations: (1) Most of current research mainly focuses on searching prompts along a single dimension, such as example selection or order. However, the overall influence of various dimensions on the performance remains unclear. (2) These methods are typically based on heuristic criteria, and there is a gap between them and actual performance. A unified view that explains how these methods work is needed. (3) More importantly, existing methods optimize prompts globally or locally, which may lead to suboptimal performance.
In this paper, we revisit this problem from the perspective of predictive bias. We find a key insight that the quality of a given prompt depends on its inherent bias. Based on this insight, we propose a surrogate metric based on predictive bias for evaluating the quality of prompts. This metric allows us to evaluate a prompt in a single forward process without an additional development set. Specifically, we apply a given prompt to a "content-free" input and expect the model output an uniform predictive distribution (a content-free input contains no useful information). Therefore, we employ the uniformity of the predictive distribution to characterize the bias of a give prompt. This shares a similar idea to the prior work which uses this metric to calibrate the model output [18]. In contrast to this work which mainly focus on using this metric for calibration when the prompt is fixed, we further explore its usage in automatically searching an approximate prompt. Moreover, through extensive experiments, we empirically validate the correlation between the inherent bias of a given prompt and its quality measured by the average task performance on a given test set (see Fig. 2).
Moreover, this bias-based metric allows us to build prompting optimization techniques in a "local-to-global" manner. We present two novel strategies for efficiently searching high-quality prompts in a bias-guided way: (1) T-fair-Prompting (2) G-fair-Prompting. We focus on a general setting where a labeled set with size  $N$  is given. The goal of our strategies is to perform combinatorial optimization over this set to find near-optimal prompts (i.e., select demonstrations and their orders). Specifically, T-fair-Prompting uses an intuitive way that first computes the bias of each single demonstration (i.e., one-shot prompting) and then select the top-k fair demonstrations to form the final prompts. This strategy can be efficiently done with a complexity of  $O(N)$ . Note that T-fair-Prompting is based on the assumption that the optimal prompt is usually constructed from demonstrations with the smallest individual bias. However, this may not hold true in real situations and often leads to sub-optimal solutions. Therefore, we further introduce G-fair-Prompting to improve the search quality. G-fair-Prompting follows the normal procedure of the greedy search which finds the optimal solution by making locally optimal choices at each step. At each step of the algorithm, the selected demonstration is the one which makes the updated prompts achieves the best fairness score. This strategy trades off the quality of the search with the worst-case time complexity. By accepting a higher worst-case time complexity of  $O(N^2)$ , the search quality is significantly improved. Note that G-fair-Prompting works from a local to global perspective, wherein bias of individual samples are considered in the early stages while the later stage focuses on the reduction of global predictive bias.
To evaluate the effectiveness of our strategies, we conduct extensive experiments with current mainstream models, such as GPT-3 [1], on various downstream tasks. Our results indicate that our method can significantly enhance the model's in-context learning performance in an effective and interpretable manner. The overall contribution is summarized as follows:
- We introduce to use the predictive bias to assess the quality of a given prompt in an efficient and development set independent way and the empirical effectiveness of this metric is comprehensively validated.
- Based on the above idea, we propose two efficient and effective strategies, namely, T-fair-Prompting and G-fair-Prompting to optimize the prompts.
(a) Selection
(b) Selection (cal)
Figure 1: ICL suffers from high instability due to high variations in demonstrations selection and order, even when post calibration is performed.
(c) Permutation
(d) Permutation (cal)
- The effectiveness of these two strategies are validated on various LLMs ranging from GPT-series models to LLaMA family [19] released by Meta recently. Consistent relative improvements of over  $10\%$  have been observed over different downstream tasks in contrast to SOTA methods.
Relation to Calibration-before-use: Our paper shares a similar metric with cal-before-use [18] to assess the predictive bias of a given prompt. However, the prior approach aims to use this metric to calibrate the output, which can be still easily affected by the quality of the used prompt (more results can be found in Table 2). In contrast, our research aims to find a near-optimal prompt on the original space to improve the model's performance, without requiring any post-adjustment to the output of the model. Moreover, we have firstly empirically validated the connection between predictive bias and the final task performance as shown in Fig. 2, which has not been studied in [18]. Through experiments, we have discovered that, even without calibration, the prompt selected by our method can outperform a randomly selected prompt with calibration.
In-context Learning Previous research, as cited in [1, 20], has demonstrated that Large Language Models can complete tasks with zero- or few-shot learning using in-context learning. LLMs perform well with an appropriate prompt. However, recent works [4, 18] has shown that the performance of LLMs is affected by the prompt used. Therefore, determining the optimal prompt is a crucial and fundamental research area.
Original space searching A more intuitive approach for determining the best prompt is to search in the original space by selecting or reordering the prompt sentences entered by users. The searching can be concluded in two perspective. ● Global view: A naive strategy is to enumerate all candidates to find the prompt that can achieve the best performance on validation set, but this strategy is computationally expensive since its complexity is  $\sum_{k=1}^{n} C_n^k k!$  considering to demonstration selection and order permutation, where  $k$  represents the number of demonstrations selected, and  $C$  signifies the combinatorial function. Zhang et al. [12] find that errors frequently fall into the same cluster, where each cluster contains similar questions, so they proposed a diversity-guided searching strategy to select diverse demonstrations. In addition to demonstrations selection, Lu et al. [4] have identified the impact of the prompt order on the results. They found the best sequence which yields the most diverse prediction results on the probing set by generating a probing set through LLMs. However, this method is also computationally expensive, and it may be difficult to ensure that the generated probing set is sufficiently balanced. ● Local view: Previous studies [13] show that reducing the model's uncertainty helps improve the model's performance, and [14] propose Active Prompting to select demonstrations according to the uncertainty of LLMs. KATE [15] selects the prompt based on the distance amongst embeddings, with the goal of selecting the closest example. However, this method ignores the influence of the order of the examples and requires access to sentence embeddings. [16] demonstrate that LLMs can be easily distracted by irrelevant context, accordingly they identify several approaches for filtering out irrelevant information in context.
In the realm of original space searching, most of the current methods tend to focus solely on the influence of a singular factor (highlighted above) on performance, utilizing heuristic metrics to select context demonstrations that perform well according to this criterion. While these investigations certainly bring benefits to the community, they lack a comprehensive consideration of both local and
(a) AGNews (BLOOM 176B)
(b) AGNews (LLaMA 13B)
(c) AGNews (LLaMA 65B)
(d) TREC (BLOOM 176B)
(e) TREC (LLaMA 13B)
Figure 2: Accuracy is highly consistency with fairness and greedy search can find a good prompt, where "Random" and "Oracle" indicate the average accuracy of all prompts and the upper-bound performance according to fairness.
(f) TREC (LLaMA 65B)
global perspectives. The method proposed offers a metric to select context demonstrations from the perspective of predictive bias, which naturally facilitates a transition from local view to global view.
3 Revisiting the Sensitivity across Demonstrations
In this section, we will clarify the notations and the templates used in this paper. Then, we will demonstrate some brief empirical results to show how different demonstration construction factors (e.g., example selection and order) affect performance. We further introduce the definition of predictive bias/fairness of a given prompt and show its connection to the predictive performance on different downstream tasks.
We consider a training set consisting of  $N$  samples  $S = \{(x_{i},y_{i})\}_{i = 1}^{N}$ , where  $x_{i}$  is the sentence and  $y_{i}\in \mathcal{V}$  is the label of the  $i^{th}$  training sample, and  $\mathcal{V}$  is the space of all labels for the task. We use a template  $\Gamma (\cdot)$  to transform these sentences and labels into natural language space (i.e., prompt construction). Take an instance from the AGNews dataset [21] for example, we have  $x_{i} = "Cubans\ Risking\ Life\ for\ Lure\ of\ America."$ $y_{i} = "World"$ , and  $\Gamma (x_i,y_i)$  is "Article: Cubans Risking Life for Lure of America. Answer: World". We concatenate these demonstrations to form a prompt  $\rho$ , which by default is  $\rho = \Gamma (x_1,y_1)\oplus \dots \oplus \Gamma (x_n,y_n)$ , where  $\oplus$  indicates sentences combination option. At test time, we append the prompt  $\rho$  with  $\tau = "Article:\langle test\ sentence\rangle$ . Answer: "and feed it to a large language model  $\mathcal{M}$ . The predicted class is given by:
$$
\hat {y} = \arg \max  _ {y \in \mathcal {Y}} \hat {p} (y | \rho \oplus \tau), \quad \hat {p} (y | \rho \oplus \tau) = \frac {\mathcal {M} (y | \rho \oplus \tau)}{\sum_ {y \in \mathcal {Y}} \mathcal {M} (y | \rho \oplus \tau)}, \tag {1}
$$
where  $\mathcal{M}(y|\rho \oplus \tau)$  indicates the probability predicted by LLM, and the probability is normalized to fit the task. We denote the predictive distribution by  $\hat{P} (x)\coloneqq \{\hat{p} (y|\rho \oplus \tau)|y\in \mathcal{V}\}$ . In this paper, we focus on evaluating the instability caused by demonstrations, and we fix the prompt template following prior work [18].
3.2 Stability of Few-shot Prompting
As demonstrated by prior research, the few-shot prompting technique is highly susceptible to a variety of factors, including the selection and order of demonstrations [4, 18]. In this study, we delve deeper into the stability of few-shot prompting, specifically focusing on the recently released LLaMA family
by Meta [19]. Additionally, we evaluate the stability of LLaMA models calibrated using the current state-of-the-art method [12, 15].
To elucidate the impact of demonstration selection, we select four demonstrations for each different seed and randomly sample an order for each combination. Subsequently, we present the performance on AGNews in the form of a boxplot, which displays the data distribution based on a five-number summary (minimum, first quartile [Q1], median, third quartile [Q3], and maximum). As shown in Fig.1(a)(b), the accuracy demonstrates significant variability across various demonstrations. The detailed settings please refer to Appendix A.7.
To investigate the influence of permutations, we examine all possible permutations of four fixed demonstrations, resulting in 4! distinct candidates. Fig.1(c)(d) also reveals a high degree of variance. While post-calibration contributes to mitigating instability, it is essential to note that the model remains sensitive even after post-calibration. This finding underscores the importance of meticulous demonstration selection. In subsequent experiments, we discover that our approach can be employed to further enhance the performance of the calibrated model.
3.3 Predictive Bias of ICL
As demonstrated in the preceding discussion, the performance of ICL is significantly impacted by various factors such as demonstration, permutation, and selection (refer to Appendix A.4 for additional information). Consequently, devising an efficient method for constructing an appropriate prompt with near-optimal performance is a crucial step in deploying LLMs for diverse downstream tasks. As outlined in the introduction, numerous studies aim to optimize prompts in ICL. This paper further investigates this issue through the lens of predictive bias, which refers to the discrepancy between targeted classes. $^2$
To achieve this, we initially introduce an efficient technique to assess the inherent predictive bias of a given prompt, drawing inspiration from previous work [18]. We construct a training set-independent metric to measure predictive bias as follows: first, we merge the provided prompt with "semantic-free" test sample information (e.g., "[N/A]", denoted by  $\eta$ ) and obtain the LLM's predictive distribution for this sample. Ideally, the predictive distribution should closely resemble a uniform distribution, as the test sample lacks semantic information. In this paper, we employ entropy as a measure of predictive bias, defined as:
$$
\operatorname {f a i r} (\rho) = - \sum_ {y \in \mathcal {Y}} p (y | \rho \oplus \eta) \log p (y | \rho \oplus \eta) \tag {2}
$$
Previous studies have utilized this metric to calibrate the model's output. In this paper, we conduct a comprehensive examination of the relationship between predictive bias and overall performance. Specifically, in a scenario with four training samples (due to the time-consuming nature of enumerating all prompt cases for a larger number), we enumerate all possible combinations and permutations of demonstrations for various datasets and LLMs. Subsequently, we arrange all candidates in descending order based on fairness, where an "index 0" denotes the prompt with the highest fairness. We perform experiments using five different seeds, resulting in training sets comprising distinct demonstrations while maintaining the test samples with seed 0. Fig. 2 displays the results for different models, revealing a strong correlation between the model's performance and fairness score (i.e., fairer prompts yield better performance). The red star, referred to as the "Oracle" represents the optimal average performance, which consistently correlates with higher fairness. This observation prompts us to enhance the ICL performance by identifying the fairest prompt.
Nevertheless, discovering the fairest demonstration combination proves to be a formidable challenge, given the existence of  $\sum_{k=1}^{N} C_N^k k!$  distinct candidates. As the size of the training set increases, this task becomes intractable. In order to tackle this problem, we propose two efficient strategies for approximating the most suitable demonstrations in the subsequent section.
2This notion differs slightly from the concept of social bias, which concentrates on specific feature attributes rather than labels. Our approach can be naturally extended to mitigate social bias in various settings.
Figure 3: Overview of Most-fair Prompting.
4 Fairest Prompt Search
Drawing upon the aforementioned observations, we propose two strategies aimed at identifying the most fair prompt, which have been empirically demonstrated to achieve superior performance. Let us consider a training set  $S$  comprising  $N$  samples; the goal of these search strategies is to select a subset of samples from the training set and construct the context in a specific order so as to optimize the fairness criterion in Eq. 2.
In an ideal scenario, we would consider the factors of demonstration selection and order permutation by examining  $\sum_{k=1}^{N} C_N^k k!$  distinct candidates, which enumerates all possible situations. However, evaluating every candidate is infeasible, as demonstrated when  $N = 8$ , yielding over  $10^6$  candidates. In this paper, we introduce two search strategies to reduce computational cost: T-fair-Prompting and G-fair-Prompting. The T-fair-Prompting strategy decreases complexity from  $\Theta \left( \sum_{k=1}^{N} C_N^k k! \right)$  to  $\Theta(N)$ , but its performance hinges on the selection of  $k$  and may be unstable when an unsuitable value of  $k$  is chosen. As a result, we propose an additional greedy search strategy, termed G-fair-Prompting, which lowers complexity to  $O(N^2)$  and offers a superior approximation of the oracle solution. Fig. 9 visualizes the computational costs over different training set size.
4.1 T-fair-Prompting
The central idea of T-fair-Prompting (top-k) is founded on the heuristic understanding that the fairest prompt usually consists of demonstration samples with reduced individual biases. Consequently, T-fair-Prompting constructs the prompt through a two-stage process. Initially, the prediction bias is assessed when the prompt is formulated using individual demonstrations. Subsequently, the top- $k$ -fairest demonstrations are chosen and employed to prompt the LLM. It is important to note that fairer demonstrations are likely to be situated towards the end of the sequence, as the generation is more influenced by proximate demonstrations, in accordance with prior research [18]. A comprehensive description of the process is presented in Algorithm 1, while a visual representation can be found in Fig. 3. Specifically, when  $k$  is equivalent to the size of the training set, the method degrades to a search for the optimal order of demonstrations. Nevertheless, T-fair-Prompting is heavily reliant on the chosen value of  $k$ . More crucially, T-fair-Prompting addresses this issue through a purely local perspective, thereby neglecting considerations from a global standpoint, which typically results in sub-optimal outcomes. As a result, we subsequently introduce the G-fair-Prompting method, which operates in a local-to-global fashion, as described below.
4.2 G-fair-Prompting
The G-fair-Prompting (greedy search) algorithm adheres to the standard procedure of greedy search, which seeks the optimal solution by making locally optimal choices at each stage. In each step of the algorithm, the chosen demonstration is the one that allows the updated prompts to achieve the highest fairness score. This strategy balances the quality of the search with the worst-case time complexity. By accepting an increased worst-case time complexity of  $O(N^2)$ , the search quality is significantly enhanced. It is important to note that the G-fair-Prompting algorithm operates from a local to global perspective as shown by Algorithm 2. During the initial stages, the bias of individual samples is taken into account, while the later stages focus on reducing global predictive bias. Specifically, at each step, we insert a new demonstration  $\Gamma(x_i, y_i)$  from the remaining demonstration set  $S'$  (ensuring demonstrations are not repeated) at the beginning of the current context  $\rho$  and select the demonstration that maximizes the fairness improvement. Formally, at step 9 in Algorithm 2, the inserted demonstration should satisfy the following criterion:
$$
\arg \max  _ {x _ {i} \in S ^ {\prime}} \operatorname {f a i r} \left(\Gamma \left(x _ {i}, y _ {i}\right) \oplus \rho\right) \quad \text {s . t .} \operatorname {f a i r} \left(\Gamma \left(x _ {i}, y _ {i}\right) \oplus \rho\right) > \operatorname {f a i r} (\rho). \tag {3}
$$
Algorithm 1 T-fair-Prompting