Active Prompting with Chain-of-Thought for Large Language Models (2024)

Abstract: The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning. It is known that the effective design of taskspecific prompts is critical for LLMs’ ability to produce high-quality answers. In particular, an effective approach for complex questionand-answering tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs. However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks. This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning). For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful to annotate from a pool of task-specific queries. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation. Experimental results demonstrate the superiority of our proposed method, achieving superior performance on eight complex reasoning tasks. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationships demonstrate the effectiveness of our method.1

heading: 1 Introduction
text: Large language models (LLMs) (Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022a; Tay et al., 2022; Scao et al., 2022; Zeng et al., 2022; Smith et al., 2022) have achieved great success in recent years. A typical way of applying LLMs is in-context learning (Brown et al.,
1Our code is available at https://github.com/ shizhediao/active-prompt.
2020) by providing a number of instructions and exemplars, which performs well on conventional language understanding and generation tasks but performs poorly on complex reasoning tasks (Rae et al., 2021; Liang et al., 2022; Wei et al., 2022a). Recent prompting studies (Wei et al., 2022b; Wang et al., 2022; Zhou et al., 2022) found that elaborating the reasoning steps in the exemplars endows LLMs with good reasoning abilities, namely chainof-thought (CoT) prompting. However, chain-ofthought prompting depends on human engineering: it requires humans to select a few informative questions and then annotate them with CoT and answers. The human-annotated exemplars (questions with annotated CoT and answers) are not necessarily the most effective for different tasks. For example, the original chain-of-thought prompting (Wei et al., 2022b) crafted exemplars for eight questions, which are either randomly selected from the training set or manually composed by humans. Due to there being a significant variance in the nature of reasoning tasks in terms of difficulty, scope, domain, and so on, we do not know what kind of question is the most worthy of annotating. It is also not clear whether a particular set of exemplars is the best to elicit the desired information. However, the good news is that annotating eight exemplars for different tasks is trivial. It costs little money and human effort. In light of this, we identify the key problem as how to determine which questions are the most important and helpful for annotation. We propose a solution to this problem by leveraging uncertainty and introducing a bit of human effort to annotate a small set of questions. The annotation budget is reasonable.
By borrowing ideas from the related problem of uncertainty-based active learning (Gentile et al., 2022), we introduce several metrics to characterize the uncertainty among the model’s predictions on each question. Therefore, we propose a new uncertainty-based annotation strategy that chooses
1330
a number of questions from the downstream dataset and involves humans annotating the rational chains, significantly improving the performance. Specifically, given a dataset D, we first ask the model to answer it k times. Then, we calculate the uncertainty u of this model based on k answers to each question. With u, we select the most uncertain n questions with the largest u and annotate these questions by the oracle to craft new exemplars E. Finally, we pre-pend E to each test question following the standard recipe of chain-of-thought prompting (Wei et al., 2022b). The schematics of our proposed approach are illustrated in Figure 1. There are several different ways for uncertainty estimation in the literature (Settles, 2009; Culotta and McCallum, 2005). In our main experiments, we characterize the uncertainty u by the disagreement and entropy of all predicted answers. In addition, we investigate other different uncertainty metrics, like variance and self-confidence. For selfconfidence, we re-organize the generated answer with the question using a new template and then ask the model’s confidence for such generation. In this scenario, u is defined as a categorical variable from {very confident, confident, not confident, wrong answer}. It is observed that the disagreement, entropy, and variance perform similarly well, while self-confidence is not working because LLMs are prone to be over-confident.
We conduct our experiments on eight datasets, spanning arithmetic reasoning, commonsense reasoning, and symbolic reasoning. Experimental results demonstrate the effectiveness of our proposed method by outperforming the competitive baseline models. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracy-uncertainty relationship display the benefits of each proposed module and reveal their effects. Our contributions are threefold: 1) We propose to judiciously select the most helpful and informative questions for annotation, reducing the human engineering workload. 2) We introduce an effective uncertainty-based question selection strategy with several different uncertainty metrics. 3) Our proposed method surpasses competitive baseline models by a large margin on multiple reasoning tasks. To the best of our knowledge, our work is the first to demonstrate the benefits of active question selection in chain-of-thought prompting for solving complex reasoning tasks.

heading: 2 Active-Prompt
text: The schematic illustrations of our proposed approach are illustrated in Figure 1. Given l unlabeled training data Dtr “ tq1, q2, ..., qlu and m test data Dte “ tp1, p2, ..., pmu with each q and p indicating the question without any answer or reasoning steps, our goal is to annotate
only n questions from Dtr as few-shot exemplars by constructing a new exemplar set E “ tpq1, c1, a1q, pq2, c2, a2q, ..., pqn, cn, anqu with reasoning steps c and the answer a. Then, we use E to prompt the test data Dte and obtain the predictions. In this section, we explain how to select the n most uncertain questions and annotate them.

heading: 2.1 Uncertainty Estimation
text: To select a few questions from a large dataset, we need an unsupervised method. Previous studies (Gentile et al., 2022) demonstrate that reducing the model’s uncertainty helps improve the model’s performance. Therefore, we introduce the uncertainty of LLMs as a metric to select data. In the chain-of-thought setting, we first forward the LLM k times to obtain k answers for each question. Then the uncertainty of a question could be measured in different ways. In our work, we consider four potential uncertainty metrics, described below.
Disagreement First, we consider measuring the uncertainty using the disagreement among k generated answers A “ ta1, a2, ..., aku. The disagreement is calculating the unique answers in the predictions. The implementation is simple. We first count the unique answers by a set operation to remove duplicate items, obtaining h unique items A “ ta1, a2, ..., ahu. Then, the disagreement is calculated by u “ h{k. Entropy The uncertainty could also be characterized by entropy, which is calculated by
u “ argmax i
´ kÿ
j“1 Pθpaj |qiq lnPθpaj |qiq, (1)
where Pθpaj |qiq is the frequency of a certain predicted answer among all predictions. A larger entropy denotes greater uncertainty in the system, and a smaller entropy denotes smaller uncertainty. Therefore, in complex reasoning, questions with a relatively large entropy will be selected as candidates.
Variance We further consider variance as a kind of uncertainty metric, which we hypothesize might be more suitable for arithmetic answers.
u “ argmax i
řk j“1paj ´ āq2
k ´ 1 ˇ̌ ˇ q“qi , (2)
where ā “ 1k řk
j“1 aj . It is observed that there is a huge variation in predicted answers. Some predicted answers are small numbers (e.g., 1), while
some are large numbers (e.g., 10000). To mitigate the domination issue of large numbers, we propose to normalize the predictions by all the mentioned numbers in the question. For example, given a question There are x1 people. Each person has x2 apples. How many apples are there altogether? and a predicted answer ŷ, we obtain ŷ{p|x1|` |x2|q after normalization.
We first conduct a pilot study and find that disagreement-, entropy- and variance-based metrics perform competitively well, significantly outperforming self-confidence (Details are shown in Section 5.1). Therefore, in our experiments, we mainly apply disagreement and entropy for our approach, which are simple to implement.

heading: 2.2 Selection and Annotation
text: After obtaining the uncertainty of each question, we can establish an uncertainty ranking according to the uncertainty of each question. Then, we will select the top-n uncertain questions for annotation. If there are more than n questions with the largest uncertainty, we will randomly select n questions from them. These n questions will be annotated with rationale chains and answers by human annotators to construct new exemplars E “ tpq1, c1, a1q, ..., pqn, cn, anqu. E will replace the initial Ê and we will use it for few-shot chainto-thought prompting.

heading: 2.3 Inference
text: With the new annotated exemplars E, we prompt each question with them in the inference stage. In addition, we apply self-consistency (Wang et al., 2022) to infer a question m times with a temperature T , and then select the most consistent answer.

heading: 3 Experimental Settings
text: In this section, we describe the details of the datasets and evaluation metrics, the baseline models, and the implementation in the following three subsections. More details are included in Appendix A.

heading: 3.1 Datasets and Evaluation Metrics
text: Following the standard evaluation settings in LLMs reasoning studies (Wei et al., 2022b), our experiments are conducted on three types of datasets: GSM8K (Cobbe et al., 2021), ASDiv (Miao et al., 2020), SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), SingleEq (KoncelKedziorski et al., 2016), CSQA (Talmor et al.,
2019), StrategyQA (Geva et al., 2021), and last letter concatenation (Wei et al., 2022b). For last letter concatenation, we test on an out-of-distribution setting, where the prompts are two letters while the test questions are four letters. The statistics of these datasets are reported in Table 6. We report the exact match accuracy as the evaluation metric.

heading: 3.2 Baselines
text: In our experiments, the following four methods serve as the main baselines: Chain-ofthought (CoT) (Wei et al., 2022b), Self-consistency (SC) (Wang et al., 2022), Auto-CoT (Zhang et al., 2022b), and Random-CoT. Random-CoT shares the same annotation process as Active-Prompt. The only difference is that it randomly samples questions from the training data for annotation instead of applying our proposed uncertainty metrics. Our experiments are mainly based on CodeX code-davinci-002 (Chen et al., 2021) for two reasons. First, it is the most capable model available at the time we were conducting our experiments, consistent with the observations in previous studies (Wei et al., 2022b; Wang et al., 2022; Miao et al., 2020). Second, it is free of charge in the initial limited beta period. In addition to code-davinci-002, we also test the performance with text-davinci-002, text-davinci-003 and gpt-3.5-turbo to verify our method’s effectiveness in the main experiment. We call the APIs from OpenAI’s services2.

heading: 3.3 Implementation
text: Hyperparameters In our implementation, the model could only access the training data D “ tXtr, Ytru before inference and is evaluated on the test data D “ tXte, Yteu. We apply the same number of exemplars as Wei et al. (2022b), which is 8 for GSM8K, ASDiv, SVAMP, and SingleEq, 7 for CSQA, 6 for StrategyQA, 4 for AQuA and Letter (4). Given that some datasets (i.e., ASDiv, SVAMP, and SingleEq) only have the test split, we adopt the annotation result of GSM8K and transfer it to these datasets for inference. The transfer details are in Table 6. In the inference stage, we set temperature T “ 0.7 and infer 40 times for each question. We then take the most consistent answer. Unless specified, the default version of gpt-3.5-turbo used is gpt-3.5-turbo-0613.
2https://openai.com/api/
Uncertainty Estimation At this stage, we start with a few manually annotated exemplars to help infer answers in the uncertainty estimation stage. These annotated exemplars are directly taken from Wei et al. (2022b). We call it the few-shot prompting trick to stabilize the prediction. However, our method is not dependent on few-shot prompting, other exemplar-free methods like zeroshot prompting (Kojima et al., 2022) could be applied, and we demonstrate that it works well in Section 5.1. In our experiments, we limit the size of candidate instances to 1,000. If the size of the original training data is larger than 1,000, we only randomly sample 1,000 instances from it and consider such a subset while estimating the uncertainty. If the size is smaller than 1,000, we will use the full data. We conducted the experiments with different pool sizes and found that 1,000 provides robust performance, and the performance gains of increasing the pool size would converge. k is set to 10 for all the datasets in our main experiments. The analysis of performance v.s. k is discussed in Section 5.1. The results show that with the increase in pool size, the performance continues to increase and will converge at k = 10. For the uncertainty metrics, we mainly report the performance of the disagreement-based (Active-Prompt (D)) and entropy-based (Active-Prompt (E)) methods. Due to it having been observed that StrategyQA often ties with the maximum disagreement to be 2/2 = 1, we also take the frequency into consideration for Active-Prompt (D).
Annotation Our approach needs human annotation for a few selected questions. The annotator is one of the co-authors and is familiar with machine learning and chain of thought prompting. Owing to the focus of our method being the example selection rather than the annotation, the annotator did not do trial and error and conduct the minimum human engineering, referring to the previous annotation practices (Wei et al., 2022b). Given a question, the annotator would mainly write the reasoning steps and give the true answer to it. The effect of different annotators and the separate effects of selection and annotation are discussed in Sections 5.1.

heading: 4 Experimental Results
text: The experimental results are displayed in Table 1. Overall, our model outperforms all baseline models by a large margin. Across eight
benchmark datasets, Active-Prompt (D) achieves superior results with an average of 7.0% and 1.8% improvement over self-consistency with text-davinci-002 and code-davinci-002, respectively. It demonstrates the effectiveness of our proposed active selection approach. In this section, we discuss the results of arithmetic reasoning, commonsense and symbolic reasoning.
Arithmetic Reasoning: Active-Prompt achieves the best performance compared with all baseline models, indicating the superiority of our method. Compared with the competitive baseline, self-consistency, Active-Prompt (D) outperforms it by an average of 2.1% with code-davinci-002. A larger improvement is observed with text-davinci-002, where ActivePrompt (D) improves over self-consistency by 7.2%. We notice that with code-davinci-002, the largest improvement is observed in GSM8K (4.2%) and AQuA (3.1%). One possible reason is that these two datasets do not require the transferability of CoT prompts because we can directly select and annotate the questions from their own training set.
However, ASDiv, SVAMP and SingleEq do not have training data, so we need to transfer the annotated CoT from GSM8K to them. This suggests that how to better transfer prompts from one task to another is considered an important future research direction.
Commonsense and Symbolic Reasoning: Consistent improvement is observed in commonsense reasoning and symbolic reasoning tasks. ActivePrompt outperforms self-consistency across all three tasks. Note that we test the out-of-distribution setting on Letter (4), which is more challenging, and Active-Prompt still achieves the best performance compared with all baseline models.

heading: 5 Analysis
text: In this section, we further conduct several additional experiments to disclose the effects of fewshot prompts, active selection, different annotators, uncertainty metrics, pool size, and prompt engineering. Finally, we analyze the relationship between uncertainty and accuracy, hoping to provide more explanation about how our method works.

heading: 5.1 Ablation Study
text: In this section, we reveal the impact of various modules in our proposed model design. First, we reported the performance under the zero-shot setting by removing the dependency of a few exemplars, then explored the contributions of our proposed active example selection strategy. In addition, we explore the effects of different annotators, different uncertainty metrics, and pool sizes. To verify their contributions, we ablate them one by one and evaluate three downstream tasks: GSM8K, ASDiv, and SingleEq. The results are shown in Table 2.
Effects of Few-Shot Prompts In our main experiments, we start with 4-8 manually annotated exemplars to help infer answers during the uncertainty estimation stage and demonstrate the effectiveness of our method. These annotated exemplars are directly taken from Wei et al. (2022b). However, our method is independent of the exemplars provided. In this section, we conduct further experiments with the assumption that we do not have access to them. Inspired by the recent research of Zero-ShotCoT (Kojima et al., 2022), we found it is possible to bypass the manual effort of writing the initial exemplars. Instead of using 4-8 human-written exemplars to generate k predictions, we simply add “Let’s think step by step." and let LLMs generate the reasoning steps and the final answer. The results are shown in Table 2 Zero-Shot-Active-Prompt, which performs competitively to Active-Prompt, demonstrating that our method is not necessarily
dependent on the few-shot exemplars.
Effects of Active Selection Our main contribution is the proposal of an effective example selection strategy (namely active selection). We replace the active selection with random selection by randomly selecting the same number of questions for annotation. The annotation process is exactly the same as Active-Prompt with the same annotation process and annotator. This model is called Random-CoT. The results are shown in Table 2. It is observed that Active-Prompt outperforms Random-CoT by a significant margin. Random-CoT only performs comparably to another baseline model self-consistency, illustrating that our applied annotation process has no advantages, and it is the active selection strategy that leads to performance gains. For example, on the GSM8K dataset, Random-CoT (78.6) slightly outperforms SC (78.0) while significantly underperforming Active-Prompt (82.2) by 3.6%. The full results of Random-CoT on all datasets are reported in Table 1 with a consistent performance drop compared with Active-Prompt.
Effects of Annotators In our main experiments, we asked the annotator not to do trial and error with minimum human engineering because the focus of our method is the question selection, rather than the best possible annotation. However, different annotators can still cause variations in the performance. In this section, we discuss the effects of different annotators. In addition to the annotator (annotator A), we directly use the human-annotated rationales from the GSM8K dataset (annotator B). The results are reported in Table 2. The results of annotators A and B are consistently better than baseline models, demonstrating the robustness of our proposed selection method. Surprisingly, we found that directly applying the solutions provided by GSM8K outperforms our annotated rationales, suggesting that the existing annotation of GSM8K is of high quality. In addition, we note that human prompt engineering has two complementary components: question selection and prompt template engineering. The method proposed in this work provides a good solution to the first problem. It is also possible to combine this technique with human-optimized prompt templates to further improve performance.
Effects of Uncertainty Metrics In our main experiments, we adopt disagreement and entropy as the uncertainty metric. In addition to those, other
uncertainty metrics can be incorporated. In this section, we mainly discuss four uncertainty metrics: disagreement, entropy, variance, and selfconfidence. The definitions of the first three metrics are illustrated in Section 2.1 and the definition of self-confidence can be found in Appendix D. First, we found that disagreement is not applicable to datasets with limited search space. For example, the StrategyQA has only two labels (yes or no), and the predictions often tie in the maximum disagreement 2/2=1. Therefore, we adopt entropy for StrategyQA. Second, the self-confidence-based method performs badly, so we did not conduct more experiments with it. We displayed an example of its prediction in Table 8. We conjecture that it is because GPT-3 is prone to be over-confident, which is consistent with previous observations (Si et al., 2022). Introducing an external well-trained discriminator to evaluate confidence is a practical way, and we leave it to future work. Last, the comparison between disagreement-, entropy- and variance-based methods are shown in Table 2. The results illustrate that they perform competitively well on ASDiv and SingleEq, while disagreement and entropy outperform variance in GSM8K. Therefore, we simply choose disagreement and entropy as the primary metrics in our main experiments.
Effects of Pool Size In the first step for uncertainty estimation, we generate k answers for each input question to construct a pool of predictions. Here, k affects the performance of estimating the uncertainty, further affecting the downstream task’s performance. To show the effect of the number of predicted answers, we plot the accuracy with respect to varying numbers of predicted answers (1, 5, 10, 15) in Figure 2 based on text-davinci-003. The results show that with the increase in pool size, the performance continues to increase and will converge at k “ 10. It is intuitive that a small k may confuse the selection process, leading to ties, while
a large k will lead to more accurate uncertainty estimation with better performance.

heading: 5.2 Uncertainty Analysis
text: The motivation of our proposed method is reducing the model’s uncertainty to help elicit the reasoning ability of LLMs, further improving the few-shot prompting performance. In this section, we display the relationship between uncertainty and accuracy. In Appendix A Figure 3, we report the uncertainty quantity and accuracy on GSM8K, ASDiv, and SingleEq. We observe that there is a highly negative correlation between uncertainty and accuracy. With the decrease in uncertainty, the accuracy increases, demonstrating that reducing the model’s uncertainty indeed helps improve the fewshot prompting-based predictions.

heading: 5.3 Transferability
text: In addressing the question of whether the uncertainty in selected exemplars is consistent across different models or if it originates from the specific task itself, an additional experiment was conducted. The experiment involves selecting exemplars using the code-davinci-002 model and then performing inference using both text-davinci-002 and text-davinci-003 models. The underlying hypothesis is that if the uncertainty is inherent to the task, then the exemplars identified by ActivePrompt would exhibit transferability across models. In other words, the active exemplars identified by one model would be applicable and effective when transferred to other models. From the results in Table 3, it is observed that all three selection-based methods perform effectively. The selected uncertain cases are related to tasks and can transfer to different models. It indicates that the uncertainty stems from the task, and the exemplars identified by Active-Prompt demonstrate good transferability. The results of this experiment provide insights into the nature of uncertainty in model predictions and its potential sources.

heading: 5.4 Performance of Weaker Models
text: Our main experiments are conducted based on powerful GPT-series models. One may wonder about the performance of weaker / smaller models, e.g., Llama-series models (Touvron et al., 2023a,b). In this section, we investigate the effectiveness of Active-Prompt with Llama-2 models and the results are shown in Table 4. It is observed that our proposed Active-Prompt outperforms CoT by a large margin, demonstrating this method is still
useful for weaker models. Note that we are using the instruction-tuned version of Llama2-70b in all our experiments (i.e., Llama2-70b-chat) because it is able to understand complex chain-of-thought prompting and follow human instructions.

heading: 5.5 Transferability between GPT and Llama Models
text: We also investigate the transferability between GPT and Llama models. Because smaller Llama models perform poorly on reasoning tasks, we conduct experiments with Llama2-70b-chat. We conduct two types of experiments: (1) select questions by gpt3.5-turbo and infer by Llama2-70b-chat (gpt-3.5turbo Ñ Llama2-70b-chat) and (2) select questions by Llama2-70b-chat and infer by gpt-3.5-turbo (Llama2-70b-chat Ñ gpt-3.5-turbo). Note that we are using the 0613 version of gpt-3.5-turbo. The results are shown in Table 5. The model before the arrow denotes the model for actively selecting questions, while the model after the arrow denotes the model for inference. The results demonstrate the feasibility of selecting questions with one model and then applying the selected questions to another model. In addition, selecting questions with larger models and applying them to smaller models results in better performance.

heading: 6 Related Work

heading: 6.1 Chain-of-thought Prompting
text: Chain-of-thought prompting elicits the reasoning abilities of large language models. The original idea proposed by Wei et al. (2022b) is to enrich the few-shot examples with reasoning steps, which greatly improve the performance on com-
plex tasks. Following Wei et al. (2022b), many studies improve standard CoTs in terms of selfconsistency (Wang et al., 2022), least-to-most prompting (Zhou et al., 2022), dynamic least-tomost prompting (Drozdov et al., 2022), bootstrapping (Zelikman et al., 2022), self-training (Huang et al., 2022), verifier (Li et al., 2022; Xu et al., 2024), prompt augmentation and selection (Shum et al., 2023), metaheuristics (Pan et al., 2023), and meta-graph prompting (Pan et al., 2024). These studies greatly improve the performance based on CoT on complex tasks while they are limited to a fixed set of exemplars. Compared with them, we propose annotating the most important task-specific questions for easy adaptation. Auto-CoT (Zhang et al., 2022b) clusters test questions according to the diversity and uses zero-shot prompting for answers. Unlike our method, it requires going through the test dataset, and our experiments show our superior performance over Auto-CoT. Note that both diversity and uncertainty are useful for selecting the most informative questions, and they are complementary. We consider the combination of diversity and uncertainty as a future direction.

heading: 6.2 Active Learning
text: Our work is also relevant to active learning (Cohn et al., 1996; Olsson, 2009; Settles, 2009; Rotman and Reichart, 2022; Lin et al., 2023), which aims to improve data labeling efficiency by finding the most helpful unlabeled data to annotate with reasonable budgets. Recent studies (Schröder et al., 2022; Köksal et al., 2022) demonstrate the benefits of active learning-based approaches for fine-tuning large language models for classification tasks. Following this, we incorporate max-entropy (Roy and McCallum, 2001), and least confidence (Culotta and McCallum, 2005) algorithms into in-context learning scenarios, and we verify the effectiveness of chain-of-thought prompting especially for complex reasoning tasks.

heading: 7 Conclusion
text: In this paper, we proposed Active-Prompt to elicit reasoning abilities in large language models (LLMs). Inspired by the idea of annotating reasoning steps to obtain effective exemplars, we aim to select the most helpful questions for annotation judiciously instead of arbitrarily. For this purpose, we propose an uncertainty-based active selection strategy to determine which questions are the most
important and helpful to annotate from a pool of task-specific questions. We introduce four different strategies of uncertainty estimation for ActivePrompt: disagreement, entropy, variance, and selfconfidence. These four strategies characterize uncertainty from different perspectives, and we primarily apply disagreement and entropy. Empirically, Active-Prompt achieved a promising performance on eight widely used datasets for arithmetic reasoning, commonsense reasoning, and symbolic reasoning. Further analyses of different uncertainty metrics, annotators, pool sizes, zero-shot learning, and an accuracy-uncertainty relationship demonstrate the effectiveness of our method.
Limitations
We have shown that Active-Prompt demonstrates superior performance over previous chain-ofthought prompting methods. While exciting, there are several limitations to the current work with future opportunities.
Experiments with more models. In our experiments, we present the complete results of text-davinci-002 and code-davinci-002 since code-davinci-002 is free of charge in the initial limited beta period. However, due to the high cost of text-davinci-002 and text-davinci-003, we were only able to carry out experiments with one of them. In addition, one promising direction is to experiment with more powerful models like GPT-4 (OpenAI, 2023). Unfortunately, conducting experiments with GPT-4 APIs is too costly. Furthermore, we did not conduct the experiments of self-consistency with gpt-3.5-turbo due to the cost. In the future, we plan to conduct experiments with GPT-4 and self-consistency experiments with gpt-3.5-turbo once we have more budgets.
Reproducibility In our experiments, we conduct most of the experiments with code-davinci-002 since it is free of charge in the initial limited beta period. The experiments of code-davinci-002 are finished before March 2023. However, OpenAI decided to shut off the access to code-davinci-002, which makes it difficult for researchers to reproduce our experiments. However, one can access it via OpenAI’s researcher access program 3 although the authors still do not have access to it.
3https://openai.com/form/ researcher-access-program

heading: A Experimental Settings
text: In this section, we describe the details of the datasets and evaluation metrics, the baseline models, and the implementation in the following three subsections.
A.1 Datasets and Evaluation Metrics
Following the standard evaluation settings in LLMs reasoning studies (Wei et al., 2022b), our experiments are conducted on three types of datasets: ‚ Arithmetic Reasoning: GSM8K (Cobbe et al.,
2021), ASDiv (Miao et al., 2020), and SVAMP (Patel et al., 2021), AQuA (Ling et al., 2017), and SingleEq (Koncel-Kedziorski et al., 2016). ‚ Commonsense Reasoning: CSQA (Talmor et al., 2019) and StrategyQA (Geva et al., 2021). ‚ Symbolic Reasoning: last letter concatenation (Wei et al., 2022b). This task evaluates the model’s ability to concatenate the last letters of the words in a name. The standard indistribution setting is trivial, and previous methods have achieved almost 100% accuracy (Wei et al., 2022b). We test on an out-of-distribution setting, where the prompts are two letters while the test questions are four letters. The statistics of these datasets are reported in Table 6. Note that in our experiments, we randomly sample 1000 data from the training set to reduce the computational cost. This may affect the performance of the uncertainty estimation. Intuitively, more training data will help capture the data distribution, leading to more precise uncertainty estimation. Given more financial support, the performance of our model will continue to increase. To make a fair comparison, we use the same test set as Wei et al. (2022b). We report the exact match accuracy as the evaluation metric.
A.2 Baselines
In our experiments, the following four methods serve as the main baselines: ‚ Chain-of-thought (CoT) (Wei et al., 2022b): stan-
dard chain-of-thought prompting which provides four to eight human-written exemplars consisting of a series of intermediate reasoning steps. ‚ Self-consistency (SC) (Wang et al., 2022): an improved version of CoT. Instead of greedy decoding, it samples a set of reasoning paths and chooses the most common answer. ‚ Auto-CoT (Zhang et al., 2022b): an automatic
exemplar construction method by clustering and generating rationales with zero-shot prompting (Kojima et al., 2022). ‚ Random-CoT: a baseline of Active-Prompt. It shares the same annotation process as ActivePrompt. The only difference is that it randomly samples questions from the training data for annotation instead of applying our proposed uncertainty metrics. Our experiments are mainly based on CodeX code-davinci-002 (Chen et al., 2021) for two reasons. First, it is the most capable model available at the time we were conducting our experiments, consistent with the observations in previous studies (Wei et al., 2022b; Wang et al., 2022; Miao et al., 2020). Second, it is free of charge in the initial limited beta period. In addition to code-davinci-002, we also test the performance with text-davinci-002 and text-davinci-003 to verify our method’s effectiveness in the main experiment. We call the APIs directly from OpenAI’s services4.
A.3 Implementation Hyperparameters In our implementation, the model could only access the training data D “ tXtr, Ytru before inference and is evaluated on the test data D “ tXte, Yteu. We apply the same number of exemplars as Wei et al. (2022b), which is 8 for GSM8K, ASDiv, SVAMP, and SingleEq, 7 for CSQA, 6 for StrategyQA, 4 for AQuA and Letter (4). Given that some datasets (i.e., ASDiv, SVAMP, and SingleEq) only have the test split, we adopt the annotation result of GSM8K and transfer it to these datasets for inference. The transfer details are in Table 6. In the inference stage, we set temperature T “ 0.7 and infer 40 times for each question. We then take the most consistent answer.
Uncertainty Estimation At this stage, we start with a few manually annotated exemplars to help infer answers in the uncertainty estimation stage. These annotated exemplars are directly taken from Wei et al. (2022b). We call it few-shot prompting trick to stabilize the prediction. However, our method is not dependent on few-shot prompting, other exemplar-free methods like zero-shot prompting (Kojima et al., 2022) could be applied, and we demonstrate that it works well in Section 5.1. For the uncertainty metrics, we mainly report the performance of the disagreement-based (Active-
4https://openai.com/api/
Prompt (D)) and entropy-based (Active-Prompt (E)) methods. Due to it having been observed that StrategyQA often ties with the maximum disagreement to be 2/2 = 1, we also take the frequency into consideration for Active-Prompt (D).
Annotation Our approach needs human annotation for a few selected questions. The annotator is one of the co-authors and is familiar with machine learning and chain of thought prompting. Owing to the focus of our method being the example selection, rather than the annotation, the annotator did not do trial and error and conduct the minimum human engineering, referring to the previous annotation practices (Wei et al., 2022b). Given a question, the annotator would mainly write the reasoning steps and give the true answer to it. The effect of different annotators and the separate effects of selection and annotation are discussed in Sections 5.1.

heading: B Uncertainty Analysis
text: Figure 3 shows the relation between accuracy and uncertainty.
C Variance Analysis
In our primary experiment, the step of uncertainty estimation necessitates querying each prompt in the training set k times to assess uncertainty. However, for datasets with a large number of instances — such as the GSM8K training set, which comprises 7,473 instances—to conserve resources — we randomly sample 1,000 instances to estimate uncertainty. To expose the inherent randomness in this sampling process, we repeated the random sampling three times, aiming to examine its vari-
ance. The results, as illustrated in Table 7, reveal that our method demonstrates robustness against the randomness of sampling. Sampling 1,000 instances proved to be sufficient for achieving stable and satisfactory results.

heading: D Self-confidence-based Uncertainty Estimation
text: Estimating the uncertainty can also be achieved by the LLMs themselves, namely self-confidence. It can be obtained by querying the model with a manually crafted template T like For the question q and the predicted answer a, report the confidence about the answer from choices. (a) very confident (b) confident (c) not confident (d) wrong answer. Then we select the least confident questions by:
u “ argmax i p1 ´ max j Pθpaj |qiqq “ argmin
i max j Pθpaj |qiq,
(3)
where Pθpaj |qiq is a categorical distribution from a set {very confident, confident, not confident, wrong answer}.

heading: E Logits-based Uncertainty Estimation
text: For models that provide logits, we can use the model’s output logits for uncertainty estimation. Therefore, we conduct further experiments to verify whether Active-Prompt still works. We first
conduct experiments with the logits returned by the gpt-3.5-turbo-0301 API. The results are shown in Table 9. As we can see, using logits, the ActivePrompt method outperforms the traditional Chain of Thought (CoT), and is slightly better than the Disagreement-based method.
Secondly, we also conducted experiments using the logits from Llama-2-70b, but we found that Llama tends to exhibit overconfidence, leading to poorer results when using its logits as a measure of uncertainty. The phenomenon of overconfidence in the logits of deep neural networks has been discussed in previous works (Guo et al., 2017; Kong et al., 2020; Chen et al., 2022), and our observations are consistent with theirs. In the future, we plan to explore more methods of calibration so that logits can be used as a measure of uncertainty for active learning.

heading: F Comparison with Diversity-based Methods
text: Auto-CoT (Zhang et al., 2022b) proposes a diversity-based method for question selection, and ours proposes an uncertainty-based method for it. In this section, we compare our method with AutoCoT to demonstrate their effectiveness and differences. Owing to Auto-CoT only reporting the results on GSM8K, MultiArith, and AddSub on code-davinci-002 without self-consistency, we first compare our method with it on these three datasets in the same setting. The results are shown in Table 10. It is observed that Active-Prompt outperforms Auto-CoT by a large margin. We attribute the improvement to uncertainty-based selection and human annotation. Note that both diversity and
uncertainty are useful for selecting the most informative questions, and they are complementary. We consider the combination of diversity and uncertainty as an important future direction.

heading: G Comparison with Complexity-based Methods
text: Complex-CoT (Fu et al., 2022) is a strong baseline which takes the complexity of prompts into consideration and proposes to select those complex prompts as exemplars. We find that ActivePrompt outperforms Complex-CoT, demonstrating the effectiveness of our proposed uncertainty-based methods. In addition, we can combine uncertainty and complexity to achieve better performance, and we leave this for future work.

heading: H Costs of Active-Prompt
text: Compared with selecting questions by humans, our proposed method is more efficient. For a new task, users need to do trials and errors a lot of times which costs a lot of human effort with unstable performance. Even so, the selected questions are still suboptimal. Second, as mentioned in Appendix A.3, we limit the size of candidate instances to 1,000 which greatly reduces the cost. 1,000 is a good balance between cost and performance. We verified that with more than 1,000 instances, the performance would converge. Doing uncertainty estimation 10 times with a pool of 1,000 questions is acceptable. The cost is smaller than self-consistency, which usually requires 40 times inference, although it is an orthogonal technique and can be complementary to ours. In addition, inspired by the new experimental results in Section 5.5, we are excited to find that questions selected by smaller models (e.g., Llama) perform well with larger models (e.g., gpt-3.5-turbo). Considering models like Llama are open-source which does not cause API cost, one may use it (with GPU) to replace black-box API.
For the annotation, using human annotation is costly. We believe that using some techniques like
zero-shot-CoT (Kojima et al., 2022) to replace manual annotation is a promising direction, and we will focus on exploring low-cost annotation methods in the future and integrate them with Active-Prompt.

heading: I Ablation Study of Longer CoT Annotations
text: Furthermore, we conduct an ablation study to differentiate the impacts of longer CoT annotations from our method. To explore this, we extended the length of the original CoT (Wei et al., 2022b) annotations to an average of 155 words, comparable to our average length of 160 words. The results are shown in Table 12. Our findings show that merely increasing the length of CoT annotations does not lead to improved performance, and in some cases, even reduces it. In contrast, our ActivePrompt method consistently demonstrates superior performance. This suggests that the selection of questions, rather than their length, contributes significantly to the improved results. Our approach effectively identifies and utilizes more informative examples for annotations.

heading: J Full Exemplars Generated by Active-Prompt
text: We display the full exemplars in Tables 13, 14, 15, 16, 17, 18.

heading: Future Work
text: , 2022) to replace manual annotation is a promising direction, and we will focus on exploring low-cost annotation methods in the future and integrate them with Active-Prompt