INFORM : Information eNtropy based multi-step reasoning FOR large language Models
Authors: Chuyue Zhou
WangJie You
Juntao Li
Jing Ye
Kehai Chen
Min Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Chain-of-Thoughts
Multi-Step Reasoning
Large Language Models
Prompting
In-context Learning

Abstract: Large language models (LLMs) have demonstrated exceptional performance in reasoning tasks with dedicated Chain-of-Thought (CoT) prompts. Further enhancing CoT prompts with exquisite exemplars can significantly improve reasoning performance.However, the effectiveness of CoT prompts may fluctuate dramatically with different choices of in-context examples. Additionally, manual construction of rationale steps can be time-consuming, presenting challenges for the widespread adoption of CoT prompting. In this work, we propose a novel approach by introducing information entropy (IE) as a criteria on for CoT prompt selection. We extend this criterion to the CoT generation and inference stages, automatically generating CoT prompts with higher information entropy scores and adaptively determining the number of samples. These three stages together form our proposed information- entropy-based multi-step reasoning for large language models, named INFORM. Our experiments across seven reasoning benchmarks utilizing two language models(GPT-3.5-Turbo and text-davinci-003) demonstrate the superiority of INFORM both in performance and efficiency.

INFORM
Information eNtropy based multi-step reasoning FOR large language Models
Chuyue Zhou'; Wangjie Juntao L Jing Ye', Kehai Chen', Min Zhang'
'Intitttee of Computer Science and Technology, Soochow University, China
2 Harbin Institute of Technology, Shenzhen {cyzhou, wjyouu, jyeyj]@stu. suda. edu. cn (ljt, minzhang)@suda. edu. cn chenkehai mrinkehai@hit... cn
Abstract
et al., 2022; Wang et al., 2023) have highlighted the importance of elaborating the reasoning steps in exemplars, Icading to the emergence of chain- of-thought (CoT) prompting. CoT has shown promising results in improving the reasoning abil- ities of LLMs. Various strategies like self-notes (Lanchantin et al., 2023), progressive-hin prompt- ing (Narang et al., 2023) and Least-to-Moss prompt- ing (Chowdhery et al., 2023), have been proposed to enhance CoT further.
Large language models (LLMs) have demon- strated exceptional performance in reasoning tasks with dedicated Chaon-of-Thought (CoT) prompts. Further enhancing CoT prompts with exquisite exemplars can significantly improve reasoning performance. However, the effec- tiveness of CoT prompts may fluctuate dramat- ically with different choices of in-context ex- amples. Additionally, manual construction of rationale steps can be time-come consuming, present- ing challenges for the widespread adoption of this work, we propose
While these works have focused on leveraging the inherent capabilities of models augment the original prompts, recent research has shown that the performance of CoT heavily relies on the choice of exemplars (Lu et al., 2022; Liu et al., 2022; Wei et al., 2022; Zhang et al., 2023; Fu et al., 2023). This has led to investigations into identifying exem- plars that maximize LLMs' reasoning abilitics. No- tably, complexity-based prompting (Fu et al., 2023) has introduced selection strategy based on the number of rationale steps in annotated data. They have observed that more complex CoT prompts effectively stimulate LLMs' multi-step reasoning capabilities. However, this approach is limited by its dependency on carefully annotated data. In the absence of annotations, it degrades to a querylen- based strategy, which is more fragile to input noise.
CoT prompting. novel approach by introducing information en- tropy (IE) as criteria on for CoT prompt selec- tion. extend this criterion to the CoT gener- ation and inference stages, automatically gen- erating CoT prompts with higher information entropy scores and adaptively determining number of samples. These three stages together form our proposed information entropy based multi-step reasoning for large language mod- els, named INFORM. Our experiments across seven reasoning benchmarks utilizing two lan- guage models(GPT-3.5-Tubb) and text-davinci- 003) demonstrate the superiority of INFORM both performance and efficiency.
1 Introduction
Large language models (LLMs) (Brown et al.,
This limitation highlights a fundamental draw- back of CoT prompting, i.e., the heavy reliance on human engineering and the scarcity of anno- tated datasets, which are time-consuming to create. To address this, previous works such as zero-shot- cot (Kojima al., 2022) and Auto-CoT (Zhang et al., 2023) attempted to alleviate the reliance on human effort in constructing CoT prompts. How- ever, they faced challenges such as low-quality gen- eration and high computational cost.
2020; Chowdhery et al., 2022; Thoppilan et al.,
2022: Le Scao al., 2022: Touvron al..
2023) have achieved great success in recent years. These models are commonly employed through in-context learning (Brown et al., 2020), where instructions and exemplars are provided to enhance language understanding and generation. However, the widely-used in-context learning methods might perform poorly for complex reasoning tasks (Liang al., 2022; Wei et al., 2022). Recent studies (Wei
To overcome the aforementioned issues, we pro- posed comprehensive CoT prompting process that bypasses the need for extensive human effort while improving the effectiveness of CoT. Our first
Equal Contribution
Juntao is corresponding author. htttps: / htwpttttttttteeee/eee//eeeeeee com/ htt///nnnnng///FFR

---

0.0
GPT 35 Turto
--- 35 Turbo
0.8
0.5
62
docin 003
0.15
0.06
INTORRATIION ENTIGEY SCORE
COMPLERSIUPS
Figure We compared the performance (ACC rate) of GSM8K dataset along with the increase of complex reasoning step and the information entropy scores two models: GPT-3.5-Ture and text-davinci-003.
and foremost objective to identify a pre-selection strategy that achieves comparable effectivenees to complexity-base prompting while remaining ap- plicable in all scenarios. Through extensive ex- periments, we have discovered that information entropy serve
effective CoT prompting framework con- sisting of three stages: question selection, CoT generation and information entropy self- consistency inference.
effective and accurate cri- terion. Figure 1 depicts the similar trends between performance and the number of rationale steps and information entropy scores queries, respectively. on the GSMSK dataset using GPr-o.5-Turbo and text-davinci-003. We consider information entropy to be a suitable criterion for several reasons: (1) Information entropy can be applied to queries, mak- ing it applicable to all datasets. (2) The information entropy score of query provides insights into its complexity, which is related the information con- tained in the annotated or generated prompt. (3) In- formation entropy well-suited for both the gener- ation and inference stages, enabling a coherent and unified framework, namely INFORM. Our experi- ments further validate the effectiveness of informa- tion entropy. Besides, we combined our informa- tion entropy criteria with self-consistency (Wang al., 2023), extending strategy from the input space (question selection) the output space (CoT generation and IE inference). In a nutshell, the contributions of our work are:
Experimental results demonstrate the effec- tiveness of INFORM on different reasoning tasks utilizing different models.
Related Work
2.1 In-context Learning
Recent studies have shown that with the ever- increasing scale of language models, they show remarkable ability to perform in-context learning (ICL) on downstream tasks (Brown et al., 2020; Kaplan al., 2020). ICL enables LLMs to tackle target task by using a few prompted examples as part the input, allowing them to solve uni- versal tasks without the need for gradient updates. However, it has been demonstrated that the perfor- mance of ICL is influenced by the prompts used (Liu et al., 2022, 2023). Therefore, determining the optimal prompt is crucial and fundamental area of research.
2.2 Chain-of-Thoughe Prompt
a novel prompting paradigm proposed by Wei et al. (2022), which involves series of rationale steps leading to the final answer. CoT has been shown to signifi- cantly enhance performance complex reasoning tasks such as arithmetic and commonsense reason- ing. This success has spurred several subsequent works that adapt different strategies to improve CoT, including self-consistency (Wang et al., 2023), Least-to- Most prompting (Chowdhery et al., 2023),
Chain-of-ThoughtiCCC
We introduce information entropy as a new cri- terion for CoT prompts selection, improving LLMs' performance on reasoning tasks.
We further apply information entropy criteria to the inference stage, generate reliable out- puts automatically and save computation costs in the meantime
We propose INFORM, a comprehensive and

---

self-notes (Lanchantin et al., 2023), Progressive- Hint Prompting (Narang al., 2023), and self- polish (Wei al., 2023).
2023) proposes an uncertainty-basee active selec- tion strategy to determine which questions are the most important and helpful to annotate from pool of task-specific queries.
Despite the remarkable success of CoT prompt- ing, previous studies have primarily focused on how to use CoT to achieve the best results while ignoring how to construct prompting examples. Re- search by Liu et al. (2023) and Lu et al. (2022) has shown that the effectiveness of CoT prompting can vary widely depending on the choice of CoT exam- ples. Given that data with manually annotated rea- soning chains are scarce, and it is time-consuming annotate rational chains manually in addition to the selection of questions and the accuracy of an- notations that need be considered, constructing appropriate prompts has been identified as a critical aspect of CoT prompting.
In the realm of rule-based selection, most current methods tend to focus on searching prompts along single dimension and either excessively rely on labeled data or require manual prompt construc- tion. Our work sits in CoT prompting and proposes a novel IE-based selection strategy that evaluates queries from multi-dimension and constructs the CoT prompt automatically.
Automated Generation is observed that data with annotated reasoning chains are scarce, and it's consumnng to annotate manually. Some researchers have committed to automating the gen- cration of CoT prompts for language models. Auto-CoT (Zhang et al., 2023) classify questions into different clusters, select representative ques- tions with diversity, and generate reasoning chains by zero-shot CoT prompt to construct demonstra- tions. Huang et al. (2022) demonstrate that LLM is also capable of self-improving with only unlabeled datasets. They use a pre-trained LLM to gener- ate high-confidence rationale-augmented answers for unlabeled questions using Chain-of Thought prompting and self-consistency and fine-tune the LLM using those self celf-generated solutions tar- get outputs. Automate-CoT (Shum et al., 2023) first augment rational chains from small labeled dataset, then prune low-quality chains to construct a candidate pool of machine-gencrated rationalc chains based on the labels, and finally apply a variance-reduced policy gradient strategy to esti- mate the gradients and optimize the latent variables with estimated gradients to select COT.
2.3 Example Construction For Prompting
Previous works on example construction for prompting can be viewed from two perspectives: rule-based selection and automated generation.
Rule-based Selection Rule-based selection methods employ various criteria to select optimal prompts from the original space. One intuitive strat- egy is to choose the most similar examples to the test question as its prompts. Nie et al. (2022) use nearest neighbor calibration to adjust the similarity scores between the input and the few-shot exam- ples. KATE (Liu et al., 2022) shared the same strat- egy but scored the similarity based on the distance among embeddings. However, similarity- -based strategies are usually computationally expensive. As an alternative, faimesss guided prompting (Ma et al., 2023) employs a content-free strategy that uses fairness as metric to evaluate the predictive bias of a fixed prompt and shows that model per- formance is highly consistent with faimess. For CoT prompting, fairness-based methods require reasoning chain annotations for the whole training set, which compromises their advantage of being few-shot. Fu et al. (2023) propose that the rea- soning ability of LLMs can be elicited by more complex CoT prompts, and the number of CoT steps can determine the complexity of the prompt. Their experiments demonstrate that selecting more complex CoT examples helps LLMs solve complex reasoning tasks. However, their complexity-bar based criteria heavily rely on labeled CoT data and degen- crate to querylen-based when there no labeled CoT data. In addition, Diao et al. (Diao et al.,
These methods have extremely dispensed hu- man efforts in generating CoT examples; however. most of them suffer from extra training costs, de- pendency on annotated data, and ignorance of the selection of questions. Our work proposes a com- prehensive process of CoT prompting, covering rule- based selection of query, automatical genera- tion of rationale steps and improvement of infer- ence results.
3 INFORM
The overall schematic of our proposed INFORM are illustrated in Figure 2. Our approach is devel- oped based on Information Entropy criteria, which we integrate into three different stages of standard

---

Unlabeled Questions
Input
bog, of only witte rottes palnned Hocch and yor hark Then aoomer Atwer Let stp systpp
Answer Lerrs
Ranking
know ttat there aoe 20 booc ou eorrect ansamr merriccoo
Ca Gucoton:: The max had spant, alliaiscs eutin apollaane EAC link ltysles.
B--Soome:5 5,30
Dasrion:Durm/wwa moezing lames
coranee in the rasonall, know Out there are brack musbrec in original
we
S-Scon WDD)
M0,, P(63
Diversity Pruning
baac then nut back
correc antweric 100)
Adjusts Number Samples
Assner
We know that there are 20 black nuttlo
nam senneent 1/20m
2015
sayo aoa Therebrr the bog numentte (oo mamtes noodl connet
Order Augment
Ruecton: Sarah souned neai tit gelled onto her cifine
Sell-Coneitencyy with Adjusted Number
bank when 20 marble Therefore,
know that
Cuestion: Bily cn't knew whatl to bought lea.
Wrong Antane
inower3
camenty holds 2000 marbles in 12t8.. Se the answer 2000
just
High
asoaet 6
Question Selection
CoT Generation
IE-SC Inference
Figure Overview of our proposed INFORM. Question Selection: Comprises three sub-steps: IE ranking, diversity
pruming, and order augmentation, aimed at selecting informative questions. CoT Generation: Automatically generates rationale steps for the selected questions and identifies the one with the highest IE score. IE-SC Inference: Adjusts the number of samples based on the IE score the query, followed by applying the original self-consistency method to conduct majority vote.
sew-shot CoT prompting. Specifically, our pro- posed approach consists of three sequential stages: question selection, CoT generation and IE-SC in- ference. We will introduce every stage in detail in the following sections.
selection. Specifically, our question selection strat- egy comprises three steps: IE ranking, diversity pruning and order augment.
IE Ranking Given set of unlabeled ques- tions Q (41 92: 23---nn, calculate their information entropy score denoted by H (h1, fız, hz...h,), using Equation 1. We then select the top-k questions with the highest scores, as we believe they will generate more informative ra- tionale steps. In our experiments, we set k to three times the number of candidates in the dataset.
3.1 Information Entropy
Information entropy describes the uncertainty or randomness of information. The defimition infor- mation entropy is as follows:
H(X) Ep(r) loga p(x;)
(1)
Diversity Pruning After obtaining set of ques- tions with high IE scores, consider incorporat- ing a filtering step to enhance diversity. Specifi- cally, we calculate the cosine-simit similar-t among the question set with:
The term p(x;) represents the probability of each outcome xy. Therefore, as the information entropy increases, the sentence becomes more informative. As demonstrated the complexity-base strategy, the more complex exemplars can significantly im- prove the performance of large language models.
(2g
1/1
cos(O)
(2)
(x:
Σ
(2:)²
21=1
3.2 Question Selection
For examples with high similarity (>80%), we only keep the one with the highest IE score, dis- carding the rest.
Previous research consistently demonstrates that selecting exemplars based on specific criteria can significantly improve the performance of L.LMs in steti-step reasoning tasks. We emphasize the im- portance of question selection instead of exemplar
Order Augment As widely recognized, in- context learning is sensitive to the order of demon- strations within prompts (Lu et al., 2022: Liu et al.,

---

2022). Intuitively, we sort the order of questions based on their information entropy score, effec- tively arranging them from easy to hard, aiming to exploit LLMs' rational reasoning ability gradually, which is similar to curriculum learning (Bengio et al., 2009) in spirit.
using fixed budget for the entire dataset. The ad- justment is based on the information entropy score of the query and can be expressed as follows:
Nmax Nuas (H (q:)
N₂ Nuin +
Hmin)
(3)
Hmaz
mmin
where H (q:) represents the IE-score of i-th query. H miin and H denote minimum and maximum IE scores, which set according to the dataset, typically to and 8. Nmi and Nmaz indicate the minimum and maximum number of samples, respectively, which we set to and 20 in our exper- iment. Our experiment demonstrates that IE-SC, in contrast to the original SC approach, well-suited for different datasets and different language mod- els, and effectively reduces the computational costs without compromising output quality.
3.3 CoT Generation
After employing our query selection strategy to acquire specific set of queries, we proceed to au- tomatically generate the corresponding rationale steps using the zero-shot-Co't technique (Kojima et al., 2022). Note that we generate rationale steps only when the data lacks annotated CoT; otherwise, we utilize the original rationale steps. Taking in- spiration from previous work that leveraged multi- sampling to enhance LLMs' performance during the inference stage (Wang et al., 2023), we extend this approach to our CoT generation stage. Specif- ically, given a selected query q, we append "Let's think step by step" to the end of the query and ask the model to answer our experiments. Once we obtain 10 distinct CoT sequences, we select the CoT's that have cor- rect answers based on the ground truth provided in the dataset. If the correct answer is not given, we adhere to the original self-consistency approach to employ majority voting mechanism. Finally, we select the CoT sequence with the highest infor- mation entropy score among the selected correct answers.
4 Experiments
4.1 Experimental Settings
times. We set k = 10
Datasets We evaluate our approach across seven
benchmarks, covering three categories of reasoning tasks: (i) Arithmetic Reasoning: GSM8K (Cobbe et al., 2021), MultiArith (Roy and Roth, 2015), Addsub (Hosseini et al., 2014), AQuA (Ling et al., 2017) (ii) Commonsense Reasoning: Com- monsenseQA (Talmor et al., 2019) and Strate- gyQA (Geva et al., 2021) (ili) Temporal Reasoning: Date-Undersn Satte-undest (Zhang et al., 2021). Among these datasets, GSM8K, AQuA, and Strategy provide their own rationale steps within their train- ing splits. Hence, for these datasets, we seam- lessly integrate our strategy into the query selec- tion and inference stages, leveraging the original rationales. For the remaining datasets, we first se- lect the queries and then automatically generate the Chains of Thought (CoT) prompts. By default, we employ the test split for evaluation. In cases where test sets are not provided, we assess performance on the validation set instead.
3.4 Information Entropy Celf-Consistency Inference
Wang et al. (2023) proposed Consisstss,,,, which improves performance for CoT reasoning by employing multiple diverse reasoning chains and aggregating their outputs using simple majority voting technique. However, this approach comes with a trade-off of the increased computational cost that the model must be prompted multiple times for the same question. As LLMs continue to grow in size, the cost becomes increasingly unacceptable. Our empirical observations revealed that having more candidates does not always yield better per- formance, and using fixed number of samples for every question in dataset can be inefficient, par- ticularly for straightforward questions. To address these issues, we introduce adaptive Information Entropy Self-consistency(F--SC)) Unlike the con- ventional approach, IE-SC dynamically adjusts the number of samples n. for each query instead of
We utilize the text-davinei-003 and GPT-3.5-Turbo version of the public ChatGPT model from the OpenAl API with 175 billion pa- rameters (Brown et al., 2020; Ouyang et al., 2022) for most of our experiments. We selected these two LLMs because they are the latest and most robust models available. For most datasets, queries are selected from the official training split. Given that some datasets only have the test split, we select queries from the validation split or from the test
Implementation

---

split itself, excluding them from subsequent experi- ments. Following Wei al. (2022), we set the num- ber of demonstrations k to 8, except for AQuA(4), StrategyQA (6), and CSQA (7). In datasets lack- ing annotated reasoning steps, we automatically generate CoTs and select the one with the highest IE score among the correct answers, Following Kojima et al. (2022), we add "Let's think step by step" before the reasoning chains for all prompting schemes improve the performance. During the standard decoding process, we set the temperature to and employ a greedy search algorithm to ob- tain results. Under the IE-SC setting, we set the temperature to 0.7 and determine the number of generations adaptively based on the IE score of the query, ranging from 5 to 20.
of our proposed approach. In the subsequent sec- tions, we calegorize the datasets based on the pres- ence of annotated rationale steps and provide a detailed discussion of our findings.
Annotated Dataset For datasets obtaining rea- soning annotations themselves, such as GSM8K, AQuA, and StrategyQA, we simply apply our strat- egy to the question selection and inference stage, utilizing the original rationale steps from the train- ing set. In comparison complexity prompt- ing, dedicated to selecting examples in the an- notated dataset, our approach outperforms it most datasets using two models. One exception is GSM8K dataset with text-davinci-003, where our method slightly lags behind the complexity- based strategy by 0.68%. Upon conducting thor- ough analysis of the reasons, we have discovered plausible explanation that the GSM8K dataset is meticulously annotated and the questions exhibit significant diversity, allowing the complexity- based selection strategy to achieve surprising re- sults. Furthermore, under the IE-SC setting, our method demonstrates additional improvements of 7.12%, 9.37% and 4.17%, respectively, with GPT- 3.5- Turbo. Compared to methods automatically generating the CoT prompt, like Auto-CoT, our ap- proach effectively leverages the labeled data, elim- inating the need for augmenting extra CoT exem- plars. In Appendix A, we supplement the experi- mental evidence that using original rationale steps leads to better results compared to generating it via LLMs, while also saving costs. Meanwhile, our strategy outperforms Auto-CoT by 3.03% and 2. 15% on average with these three datasets with GPT-3.5-Turbe and text-davinci-003, respectively. This discrepancy be attributed to strategy's selection of queries based on predefined rules and the original manually constructed rationale steps, which is more reliable compared to the zero-shot CoT generated in Auto-CoT.
Baselines We compare our IE-CoT with three baselines: manual-Con -oT, complex- -ool, and Auto- CОT. To ensure fairness between our method with other baselines, we use the same number of CoT ex- amples in all methods and datasets. We implement these methods on our own, following the original settings in the corresponding paper.
Manual-CoT (Wei et al., 2022) is widely used naive method for CoT prompting con- struction, which provides 4 to 8 manual- written exemplars consisting of series of maital-written intermediate reasoning steps.
Complex-CoT (Fu al., 2023) utilizes the number of reasoning steps as criteria se- lect more complex exemplars in the annotated data.
Auto-CoT (Zhang et al., 2023) is an automatic CoT exemplars construction method that clus- ters questions with diversity and generates CoT prompts by zero-shot-CoT.
4.2 Results
The main results are displayed in Table 1. Overall, our method consistently matches or exceeds the per- formance of baselines on most datasets, utilizing two different large language models. Our IE-CoT achieves average improvement of 3.28% and 2.87% compared to manual-CoT with GPT-3.5- Turbo and text-davinci-003, respectively. More- over, under the IE-SC settings, the improvement further increased to 7.38% and 6.51%. These re- sults unequivocally demonstrate the effectiveness
Unlabeled Dataset Our method consistently showcases improvements across the remaining unlabeled datasets, namely MultiArith, Addsub, CSQA, and Date-Understanding In contrast with the competitive bascline, Auto-CoT, which auto- matically generates CoT prompt using zero-shot- COT, the same as our approach, our approach sur- passes it on all datasets utilizing GPT-3.5-Turbo and text-davinei-003. The most notable advance- ments are observed in the Addsub and Date-

---

GSM8K
MultiArith
Addsub
AQuA
CSQA
StrategyQA
DATE
LaMDA-1377
Manual
14.30
44.90 75.70
51.90 63.50
20.60
57.90
65,40
26.80
Self-consistengy
27.70
26.80
63.10
67.80
PaL.M-500B
Manual
56.90
94.70
91.90
35.80
79.90
77.80
65.30
Self-consistency
74,40
99.30
93.70
48.30
80.70
81,60
GPT-3.5-Tuurb
Manual
79.59
97.33
89.11
55.51
71.08
60.21
77.60
Complex
80.89
97.66
92.65
56.29
73.46
64.73
75.20
Auto-Coll
76.62
96.28
88 22
54.53
72.84
62.28
79.50
IE-CoT(ours)
81.65
98.66
92.91
58.59
74.69
64.86
82.00
IE-SC
88.77
99.16
94.68
67.96
76.50
69.03
86.00
text-davinci-003
Manual
94.16
85.75
43.70
77.55
78,40
Complex 60.42
95.16
44.88
77.64
70.08
75.60
Auto-CoT
95.08
87.94
42.60
76.80
78.00
IE-CoT(ours)
59.74
96.00
89.10
45.66
78.10
70.95
84.80
IF-SC
65.20
97.33
91.13
51.90
80.09
74.58
89.60
Table I:
The overall performance of INFORM and the comparison against existing methods under different models on seven reasoning tasks. Manual-CoT, Complex and Auto-CoT denote thain-of-thought (Wei al., 2022). complexity-basee prompting (Fu et al., 2023) and Auto-CoT (Zhang et al., 2023), respectively. IE-SC denotes our performance under the IE-SC setting. Bold denotes the best performance in performed methods. Underline denotes the second performance. LaMDA (Thoppilan et al., 2022) and PaL. (Chowdhery et al., 2022) not accessible the public, so their numbers are from corresponding papers. The remaining methods are our own implementation.
understanding datasets, where our proposed strat- egy outperforms Auto-CoT by 3.85% and 3.98% on average with GPT-3.5-Turie and text-davinci-003. This can be attributed to the relatively simplified nature of queries in these two datascts. While au- tomatically generated CoT prompts may provide limited information, our strategy diligently selects the most informative queries and prompts during the selection and generation stages, resulting in a more effective prompting effect than Auto-CoT.
Datasets
GSM8K
AQuA
Random
79.59
55,51
IE-low
78.22
53,93
IE-high
80.42
56.69
IE-high + div
81.50
58.26
IE-high + div order
81.65
58.59
Table 2: Effect of every step in the query selection stage.
MultiA
Addsub
CSQA
Random
98.33
92.31
72.76
Noting that complexity-basee strategy degrades to querylen- based when handling unlabcled datasets, leading to the selection of highly similar questions. It lagged behind 2.32% and 2.82% on average by our approach with GPT-3.5- Turbbbbb and text-davinci- 003, respectively. This serves further evidence of the effectiveness and versatility of our approach.
IE-low
97.83
91.17
73.49
IE-high
98.66
92.91
74.69
Table 3: Effeet of information entropy score in CoT generation stage.
ranking, diversity pruming, and order augment. To evaluate the effectiveness each step, we con- ducted an ablation study by analyzing their in- dividual contributions. As presented in Table 2, our method consistently achieved improvements on two datasets. Specifically, IE-low and TE-high refer to the selection of queries with low informa- tion entropy scores and high information entropy scores, respectively. A substantial average gap of 2.48% is observed between these two extreme con- trasts. Additionally, IE-low lags behind randomly selected exemplars by an average of 1.47%, high-
5 Analysis
We conducted additional experiments to assess the effectiveness of INFORM and analyze the contri- butions of each component of INFORM. Due to the high cost associated with using the text-davinci- 003 API, we primarily utilized GPT-3.5-Ture for the additional experiments.
Effects of Question Selection As shown in Fig 2, the query selection stage involves three steps: IE

---

GSMI8K SC
AQUA
CSQA
SC
11-3
I1-C
76
89
75
87
20
10
11.8
Figure 3:
The performance of self self-coonistency with original self-consistency and our proposed IB-SC. The X-axis means the average number of candidates and the Y-axis means the exact accuracy.
Models
BLOOM-7b
OPT-6.7b
Alpaca-7b
Alpaca-138 13b
Davinci-002
Manual-CoTT
3.77
4.70
8.64
11.06
48.06
IF-CoT
4.25
5.48
9.70
17.21
50.49
Table 4: The performance of INFORM on five language models on GSM8K datasct.
lighting that queries with low information entropy lead to prompts lack of information, hampering the reasoning ability of LLMs. Moreover, the ap- plication of diversity pruning brings an additional 1.08% improvement for IE-high queries. This im- provement can be attributed to the decreased simi- larity between queries, which provides diverse ide- ology to LLMs, enabling them to handle differ- ent types of challenging tasks. As widely recog- nized, in-context learning is sensitive the order of demonstrations within prompts (Lu et al., 2022; Liu et al., 2022). Indeed, the result reveals that order- augment brings improvement; however, the mag- nitude of this improvement is very (light(0.24%) We leave further study on how to determine the op- timal ordering of exemplars within prompts to fully exploit the capabilities of LLMs in future work.
Effects of IE-SC We conducted further anal- ysis of our proposed Information Entropy Self- Consistency (IE-SC). As depicted Fig 3, while consiseoy undeniably yields significant im- provements for reasoning tasks, it is noteworthy that better performance does not necessarily rely on a larger candidate pool, as evidenced by the results the CSQA dataset. Our proposed IE- SC approach achieves remarkable 33% reduc- tion in computational cost (equivalent to reducing 6.7 instances per query) compared the original SC method, which samples 20 candidates. This reduction in cost comes with a minimal average decrease in accuracy of less than 0.2%. These findings demonstrate that our proposed approach strikes a favorable balance between effectiveness and efficiency.
Effects of IE in CoT generation After apply- ing our query selection strategy to obtain spe- cific set of queries, we proceed to automatically generate the corresponding rationale steps using the zero-shot-CoT technique (Kojima al., 2022). Noting that we generate rationale steps only when the data lacks annotated COT. Therefore, we select three datasets, namely MultiA, Addsub, and CSQA, which do not have labeled rationales, to evaluate the effectiveness of choosing CoT prompts with high information entropy scores. As depicted in Ta- ble 3, selecting high information entropy (IE) score prompts consistently outperforms the other two strategies: random selection and selecting prompts with low IE scores. This observation suggests that CoT prompts with high IE scores are more effective in stimulating LLMs' reasoning ability, leading to improved performance.
Robustness of INFORM We performed addi- tional experiments various large language mod- els to assess the robustness of our proposed IN- FORM approach. As illustrated in Table 4, method consistently achieves improvements across different models, including the previous version of GPT-3.5 ((ext-davinci-002) as well as smaller-scale language models such as Alpaca-7b and Alpaca- 13b. Since some larger-scale models are unavail- able, i.e., Palm and LaMDA we have not con- ducted experiments on them. Even so, these results have clearly demonstrated the versatility and effec- tiveness of INFORM, indicating its applicability to a wide range of language models. We further discuss the impact of linguistic characteristics on our approach and and provide some examples for further clarification in the appendix and C.

---

Acknowledgements
References
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international confer- ence on machine learning, pages 41-48.
This work is supported by the National Science Foundation of China (NSFC No. 62276077 and 62206194), Shenzhen College Stability Support Plan under Grants GXWD202208 1170358002 and BXWW222208111223100000
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
and the Natu- ral Science Foundation of Jiangsu Province (Grant
No. BK20220488).
6 Conclusion
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sehastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 22.3.1.0...
This paper presents INFORM, a novel framework for CoT prompting that consists of a comprehen- sive process encompassing query selection, CoT generation, and IE-SC inference. INFORM de- signed to be adaptable to various datasets and mod- els, effectively improving the performance of CoT prompts and mitigating the requirement for addi- tional human involvement. Our experimental re- sults demonstrate the efficacy and versatility of INFORM, showcasing its ability significantly enhance CoT performance in various scenarios.
Aakanksha Chowdhery, Sharan Narang, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Lc, Ed Chi, and Denny Zhou. 2023. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representa- tions.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 1466.
Limitations
Despite the promising results of our proposed INFORM, it comes with several limitations that should be addressed in future work:
Shizhe Diao. Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302.122:
While queries are typically manually con- structed, our query selection strategy may be susceptible noise when the queries contain irrelevant information. Integrating additional criteria could help mitigate this issue.
Yao Fu, Hao Peng. Ashish Sahharwal, Peter Clark, and Tushar Khot. 2023. Complexity-based prompting for multi-step reasoning. In International Conference Learning Representations.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
Our experiments focused solely on reason- ing tasks, showcasing the effectiveness of IN- FORM with CoT prompts. However, the per- formance of INFORM on non-reasoning tasks remains unknown.
Mohammad Javad Hosseini, Hannanch Hajishirzi. Oren Etzioni, and Nate Kushman, 2014. Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523-533.
Although we have evaluated the robustness of INFORM on several different LLMs of vary- ing sizes, there may still be models, such as very large-scale models like Palm, where IN- FORM may not be as effective.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu. Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.111//
These limitations highlight areas for further inves- tigation and refinement in order enhance the applicabiliiy and performance of INFORM in wider range of scenarios.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv e- prints, pages arXiv-200)

---

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In ICMI 2022 Workshop on Knowledge Retrieval and Language Models.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Jack Lanchantin, Shubham Toshniwal, Jason Weston, Arthur Szlam, and Sainbayar Sukhbaatar. 2023. Learning to reason and memorize with self-notes.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1743-1752, Lisbon, Portu- gal. Association for Computational Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model.
Freda Shi. Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay. Sebastian Ruder. Denny Zhou, et al. 2022. Language models are multilingual cfain-of-thought reasoners. In The Eleventh International Conference on Learning Representations.
Percy Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint 221::221.0091111
KaShun Shum, Shizhe Diao, and Tong Zhang. 2023. Automatic prompt augmentation and selection with tfain-of-thought from labeled data. arXiv preprint 2:Xii:2302.1282.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 157-167, Vancouver, Canada. Association for Computational Linguisties.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonseesseaa: A question answering challenge targeting commonsense knowl- edge. In Proceedings of NAACL-HLT, pages 4149- 4158.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (Dee- LIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-114.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.082
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):133..
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 8086-8098.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le. Fdd Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representa- tions.
Huan Ma. Changging Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. 2023. Faimess-- guided few-shot prompting for large language mod- cls. arXiv preprint arXiv:230..l 13217.
Jason Wei, Xuezhi Wang, Sharan Narang, Aakanksha Chowdhery, Dale Schuurmans, Quoe V Le, Ed H Chi, and Denny Zhou. 2023. Self-polish: Enhance reasoning in large language models via problem re- finement. In International Conference on Learning Representations.
Sharan Narang, Aakanksha Chowdhery, Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Fdd H Chi, and Denny Zhou, 2023. Progressive-hint prompting improves reasoning in large language models. In In- ternational Conference on Learning Representations.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning large language models. In Advances in Neural Information Processing Systems.
Feng Nie, Meixi Chen, Zhirui Zhang, and Xu Cheng. 2022. Improving few-shot performance language models via nearest neighbor calibration.

---

Zheng Zhang, Yuanhe Zhang, and Zhihong Zhou, 2021. Webgqa: A new benchmark for web-based multilin- gual question answering. In Proceedings of the 2021 Conference of the Association for Computational Lin- guistics.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic chain of thought prompting large language models. In International Confer- ence on Learning Representations.

---

A Compared With Original CoT
which implies that our method can generalize other languages. We have conducted some addi- tional experiments other languages; we built three Chinese CoT datasets that were translated from English. The results are as follows:
To elucidate the implications of disregarding the included CoT strings and relying on INFORM in- stead, we conducted supplementary experiments using GPT-3.5- Turbo. The experimental setup in- volved three conditions:
Withhut-ithout-- C which involved providing only the selected queries and their corresponding direct answers as examples without including the rationale steps; "INFORM- Generated-CoT, which entailed generating CoTs using INFORM for the selected questions, specif- ically through the CoT-generation operation; and which employed the original CoTs included in the datasets as per our original strategy. The obtained results are presented below:
MultiA
Strategyqu en
Addsub en
Manual-CoT IE-CoT
96.43
68.14
81.20
98.55
68.85
90.35
Table 6: Performance IE-CoT Chinese-CoT datasets.
SAIFORM--rrggnal-CCC
C Case Study
When compared to alternative methodologies, such as the selection of queries based on complexity, the utilization of query-len in datasets lacking ratio- nales may lead to the occurrence of invalid char- acter placeholders, exemplified by the inclusion of "Question: Shipment No of Defective
GSMSK
AQuA
StrategyQA
Without-Col 63.47
55.69
$2.14
Generated-CoT
75.18
56.71
62.58
Orginal-CoT 81.65
58.59
64.86
Table 5: Compare Generated-Col with Orginal-CoT.
Chips shipment Total Chips in shipment
S 1 3- 8,000
Based on the obtained results, it can be inferred that the Original-CoT strategy exhibited the most favorable outcome, indicating that manually con- structed CoT outperforms a zero-shot-Cov gen- erated by the model in the majority of scenarios. Nevertheless, when compared to examples with- CoT's, the automatically generated CoT still demonstrated significant improvement.
S
5- 12,00000
S 3-
6- 18,000000
S $4...-..44 16,00000
A computer chip manufacturer expects the ratio of the number of defective chips to the total num- ber of chips in all future shipments to equal the corresponding ratio for shipments S 1 2 S and 4 combined, as shown in the table above What s the expected number of defective chips in a shipment of 60,000 chips ?
B Linguistic Characteristics
It is important to notice that languageedisscuure- based reasoning, different from reasoning in gen- eral, may be affected by the linguistic character- istics of the language used. Our work has not ex- plicitly addressed the syntactic, semantic, and prag- matic aspects of the language. The focus of our work, however, on how select or construct better CoT examples through simple analysis of the data in the training set rather than how linguis- tic characteristics would affect the performance of large models on reasoning tasks. Our findings can serve as starting point for researchers interested in exploring the impact of sppccfg features on reasoning tasks.
Options: (A) 14 (B) 22 (C) 20 (D) 24 (E) 25 A: Let's think step by step. for total of 51000
chips ( adding $ 1 ,, S,, 4) total number of defective chips is 17 (adding defective chips of S
1 S S 3 S 4) so ratio 18 54000 or I every 3000 chips Keeping this ratio constant for 60000 chips number of defective chips will be ( 3000) * 60000 = 20 The answer is (C), 20."
For complex detection methods, special charac- ters are used to define the steps of rationale, it is sen- sitive to different characters for different datasets.
In addition, shi et al( 2022) have demonstrated that reasoning in English (EN-CoT) consistently achieves competitive or better performance than reasoning the native language of the question,


Reviews:
[{"review_id": "9IVw03xnJt", "reviewer": "Reviewer_ZqC6", "summary": "", "questions": "On what task has the evaluation on open-source models conducted, and why is gpt-3.5-Turbo listed as open-source?\nOn which open-source models do the authors evaluate their framework?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***We thank the reviewer for taking the time to evaluate our work and provide valuable feedback. Here, we address the main concerns that were brought up.***\n\n- ***The authors claim to evaluate their method on various open-source models but do not explain which models were evaluated, only the small alpaca models. It seems odd not to compare with available larger open-source language models such as OPT and BLOOM. The authors do not report on budget and CO2 emissions, which is an important aspect of research on LLMs***\n\n  - We appreciate the feedback regarding our evaluation of open-source models in our paper. We apologize for the confusion caused by our statement about evaluating on various open-source models, and a more accurate statement would be \"evaluating on various models\". While our main experiments were conducted on the 175B GPT-3.5 models, we believe it is valuable to further validate our approach on smaller-scale models, so we chose the small alpaca models. Regarding the OPT and BLOOM models, we conducted additional experiments on the GSM8K dataset and the results are as follows:\n\n    |            | BLOOM-7b | OPT-6.7b | OPT-13b |\n    | ---------- | -------- | -------- | ------- |\n    | Manual-CoT | 3.77     | 4.70     | 6.36    |\n    | IE-CoT     | 4.25     | 5.48     | 8.44    |\n\n   - We also understand the importance of considering budget and CO2 emissions in language model research. Unfortunately, **due to some policy restrictions**, we cannot provide a specific breakdown of our budget. However, we can estimate the cost based on the charging guidelines of the OpenAI API. Evaluating a 100-instance test set costs approximately 0.3-0.5 US dollars for greedy decoding (1 output chain). It's worth noting that the cost may vary depending on the number of tokens and the specific model used. In our analysis experiments, we primarily utilized the GPT-3.5-Turbo model, as it offers a more cost-effective solution compared to text-davinci-002. Regarding CO2 emissions, since our experiments primarily focus on the inference stage of LLMs and the GPT-3.5 models are closed-source, we lack specific information about the emissions. However, our IE-SC can effectively decrease the computational cost for self-consistency settings, as mentioned in our paper.\n\n- ***It is important to notice that language/discourse-based reasoning, different from reasoning in general, may be affected by the linguistic characteristics of the language used in the work. While the work solely relies on reasoning in English, the authors fail to acknowledge or consider how syntactic, semantic and pragmatic aspects of the language may have impacted their work and whether their results may generalise to other languages.***\n\n  - Thank you for bringing up these important points. It is indeed crucial the impact of linguistic characteristics on language/discourse-based reasoning. We acknowledge that our work has not explicitly addressed how syntactic, semantic, and pragmatic aspects of the language may have influenced our findings. The focus of this paper, however, is on how to select or construct better CoT examples through simple analysis of the data in the training set, rather than how linguistic characteristics would affect the performance of large models on reasoning tasks. Our findings can serve as a starting point for researchers interested in exploring the impact of language-specific features on reasoning tasks.\n\n  - In addition, shi et al.[1] have demonstrated that reasoning in English (EN-CoT) consistently achieves competitive or better performance than reasoning in the native language of the question, which implies that our method can generalize to other languages. We have conducted some additional experiments in other languages, we built three Chinese CoT datasets that were translated from English, the results are as follows:\n    |            | MultiA_cn | Strategyqa_cn | Addsub_cn |\n    | ---------- | --------- | ------------- | --------- |\n    | Manual-CoT | 96.43     | 68.14         | 81.20     |\n    | IE-CoT     | 98.55     | 68.85         | 90.35     |\n\n\n- ***Typos Grammar Style And Presentation Improvements:***\n\n   - We have corrected the other presentation/typos/grammatical errors listed in your comments. Thank you again for your patient and valuable proofreading.\n\n\n- ***On what task has the evaluation on open-source models conducted, and why is GPT-3.5-Turbo listed as open-source? On which open-source models do the authors evaluate their framework?***\n\n  - The evaluation was conducted on the GSM8K benchmark in the \"Robustness of INFORM\" section. Regarding the mention of GPT-3.5 models as open-source, we apologize for any confusion caused. Our intention was to highlight that we evaluated our framework on various models, including both open-source and close-source ones. We understand that this statement may have been misleading, and we apologize for the lack of clarity in our original paper, we will make the necessary corrections in the later version of the paper.\n\n    We have evaluated our framework on the alpaca-7b and alpaca-13b models. To further prove the robustness of INFORM, we have conducted additional experiments on OPT and BLOOM models and the results are as follows: \t\t\n\n    |     | BLOOM-7b | OPT-6.7b | OPT-13b | alpaca-7b | alpaca-13b |\n    | ---------- | -------- | -------- | ------- | --------- | ---------- |\n    | Manual-CoT | 3.77     | 4.70     | 6.36    | 8.64      | 11.06      |\n    | IE-CoT     | 4.25     | 5.48     | 8.44    | 9.70      | 17.21      |\n\n\n\n***We hope we have addressed all the questions and thank you again for your valuable feedback. Please let us know if there are any additional concerns.***\n\n[1] Shi F, Suzgun M, Freitag M, et al. Language models are multilingual chain-of-thought reasoners[C]//The Eleventh International Conference on Learning Representations. 2023.\n\n##", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I find the idea of using Information Entropy for the selection of prompts interesting in its own right as it can help investigate the information-theoretic connections of in-context learning, and the results obtained showing the entropy helps the model learn in complex tasks points to an interesting question on the theory of such large language models.\nAside from the measure itself, the authors propose a promising prompting framework for multi-step reasoning that could be implemented based on different other measures, even example complexity itself.\nThe paper is very clear, and the empirical validation of the proposal is quite thorough, in my opinion. ", "reasons_to_reject": "The authors claim to evaluate their method on various open-source models but do not explain which models were evaluated, only the small alpaca models. It seems odd not to compare with available larger open-source language models such as  OPT and BLOOM.\nThe authors do not report on budget and CO2 emissions, which is an important aspect of research on LLMs\n\nIt is important to notice that language/discourse-based reasoning, different from reasoning in general, may be affected by the linguistic characteristics of the language used in the work. While the work solely relies on reasoning in English, the authors fail to acknowledge or consider how syntactic, semantic and pragmatic aspects of the language may have impacted their work and whether their results may generalise to other languages.", "questions_to_authors": "On what task has the evaluation on open-source models conducted, and why is gpt-3.5-Turbo listed as open-source?\nOn which open-source models do the authors evaluate their framework?", "paper_topic_and_main_contributions": "The present paper proposes a method, for example, selection and automatic CoT prompt-generation for in-context learning in large language models based on the measure of information entropy.\nThe authors compare the notion of information entropy with the notion of complexity previously proposed by  Fu et al., showing that prompting based on information entropy approximates the behaviour of prompting based on complexity. The authors use this reasoning to propose information entropy as a measure for prompt selection in the training stage. \nFurther, the authors extend their idea and employ information entropy for the generation of CoT prompts for datasets which do not contain the rationale for the answers, pulling from the work on Auto-CoT of (Zhang et al. In this setting, the authors employ information entropy as a measure of quality for the selection of CoT prompts generated. Further, they propose the use of entropy at inference time to autonomously adjust the number of samples extracted from the model to answer the question in a self-consistency approach.\nThe authors perform an evaluation of their approach on four non-open LLM over seven different reasoning tasks, comparing their approach with previous methods such as prompting by complexity, auto-cot and manual prompts.\n\nThey perform analyses on the impact of entropy on the query selection stage and CoT generation. They evaluate the method on two small open-source language models (namely alpaca 7b and alpaca 13b), but it is not clear on which task.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "pg 2 line 109: we extend apply -> we extend?\npg 2 line 116: Cot generation -> CoT generation\n pg 7 Table 2: Rondom -> Random"}, "extra_text": {}}, {"review_id": "LXk6u1uLy4", "reviewer": "Reviewer_tD1V", "summary": "", "questions": "1. In the subsection Robustness of INFORM, which benchmark do you refer to?\n2. How do you derive the values for H_min, H_max, N_min, and N_max?\n3. Could you kindly specify the whole process of how you apply Equation 1 in the process of IE Ranking?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for taking the time to evaluate our work and provide valuable feedback. Here, we address the main concerns that were brought up.**\n\n- ***In the subsection Robustness of INFORM, which benchmark do you refer to?***\n\n  - The evaluation was conducted on the GSM8K benchmark in the \"Robustness of INFORM\" section. We understand that this was not explicitly stated, and we apologize for any confusion caused.\n\n- ***How do you derive the values for H_min, H_max, N_min, and N_max?***\n\n  - As mentioned after Equation 3 in our paper, H_min and H_max represent the minimum and maximum IE scores of the candidates, respectively. These values are calculated by equation 1 on different datasets. Typically, we have observed that the values are 3 and 8.\n\n  - Regarding N_min and N_max, they indicate the minimum and maximum numbers of samplings for self-consistency. In our experiment, we followed the approach proposed by Wang et al. [1] and selected 5 as the value for N_min. And the reason we selected 20 for N_max was based on our previous experiments, where we found that increasing the number of samplings beyond 20 did not yield significant further improvements in self-consistency. We believe that these values strike a balance between computational efficiency and achieving desirable results. We appreciate the reviewer's inquiry, and we hope this explanation clarifies the rationale behind our choices.\n\n    \n\n- ***Could you kindly specify the whole process of how you apply Equation 1 in the process of IE Ranking?***\n\n  - Equation 1 represents the calculation of the information entropy score.  We have experimented with several methods to calculate information entropy in our early experiments, and we ultimately chose the unigram method. Specifically, we preserved the original sentences without tokenizing or removing special characters since reasoning datasets often contain many Arabic numerals and mathematical symbols, which are important signals reflecting the amount of information. Then, we calculated the frequency of each word appearing in the sentence, denoted as p(xi), to obtain the final result of Equation 1. We consider this as the information entropy score of the sentence. We sorted all the queries in the training set based on their information entropy scores and selected the ones with higher scores, which we refer to as IE-ranking, indicating that they contain richer information.\n\n***We hope we have addressed all the questions and thank you again for your valuable feedback. Please let us know if there are any additional concerns.***\n\n\n[1] Wang X, Wei J, Schuurmans D, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models[C]//The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I like the idea of Question Selection: using information entropy ranking, diversity pruning, and order augmentation, aimed at selecting informative questions. In particular, I really like the idea of using information entropy in this case.\n2. The authors conducted an ablation study to evaluate the effectiveness of each component of the proposed framework.\n3. The proposed framework worked well across various reasoning tasks.", "reasons_to_reject": "1. In the subsection entitled \"Robustness of INFORM,\" the author neglects to specify the benchmark used in evaluating the performance of GPT-3.5 (text-davinci-002), Alpaca-7b, and Alpaca-13b.\n\n2. The author has not provided details regarding the derivation of the values for H_min, H_max, N_min, and N_max.\n\n3. It was not very clear how the authors derive p($x _i$) when applying Equation 1 in IE Ranking. How do you determine the probability of each outcome in this process?", "questions_to_authors": "1. In the subsection Robustness of INFORM, which benchmark do you refer to?\n2. How do you derive the values for H_min, H_max, N_min, and N_max?\n3. Could you kindly specify the whole process of how you apply Equation 1 in the process of IE Ranking?", "paper_topic_and_main_contributions": "Large Language Models (LLMs) have shown remarkable performance in reasoning tasks using Chain-of-Thought (CoT) prompts. The effectiveness of these prompts can be enhanced with specific examples but may vary widely, and manually constructing them is challenging. This work proposes a novel approach, named INFORM, which employs information entropy (IE) as a criterion for CoT prompt selection, extension, and inference. This method automatically generates CoT prompts with higher information entropy scores, and adaptively determines the number of samples, leading to a three-stage, information-entropy-based reasoning process. INFORM has been tested across seven reasoning benchmarks using two different models, GPT-3.5-Turbo and text-davinci-003, showing its superiority in both performance and efficiency. The key contributions include the introduction of information entropy to CoT prompts, its application to the inference stage, and the development of INFORM, a comprehensive framework for CoT prompting, which has proven effective in different reasoning tasks and models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A.", "Typos_Grammar_Style_and_Presentation_Improvements": "N.A."}, "extra_text": {}}, {"review_id": "Dc2uUbjk0b", "reviewer": "Reviewer_ZjNo", "summary": "", "questions": "1. Can you confirm that the entropy calculations are done on the unigrams? It's not really clear from the paper, but this is what I inferred from the code. Why not use the model logits for text-davinci-003 at least?\n\n2. How are the Manual results obtained for datasets without CoT examples included?\n\n3. The paper is written as though CoT was central, but actually four of seven the datasets have CoT strings included, and so all that is happening is that demonstrations are being selected and ranked. What happens if the include CoT strings are not used?\n\n4. Are the results all based on single runs? If so, what is the observed variation across runs and across choices for the many hyperparameters that are in play here? If these hyperparameters were tuned informally based on pilot experiments, what examples were used for that and how extensively was this done across all the approaches?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We thank the reviewer for taking the time to evaluate our work and provide valuable feedback. Here, we address the main concerns that were brought up.**\n\n- ***Can you confirm that the entropy calculations are done on the unigrams? It's not really clear from the paper, but this is what I inferred from the code. Why not use the model logits for text-davinci-003 at least?***\n\n  - Your understanding is correct. Actually, we have tried several methods to compute information entropy in our early experiments, including using model logits. However, we ultimately chose to use the unigram approach for the following reasons:\n\n    1. Our experiments primarily focused on reasoning datasets containing numerous Arabic numerals and mathematical symbols. Logits from models that were not specifically trained on such data may not accurately capture the desired information. On the other hand, using unigrams can better reflect the information since it is more directly based on the frequency distribution of individual words.\n\n    2. Model logits for short sentences may exhibit numerical irregularities. When the input sentence is short, the model's output logits may be very small, causing the values of the exponential function in the softmax operation to be extremely close to zero. In such cases, numerical overflow may occur during the computation of softmax. For instance, when we used the model logits of  GPT2 to calculate the information entropy for the MultiA dataset,  the information entropy score would be NAN, which means using model logits is not a universally applicable method.\n\n    3. It's worth noting that we extend our method to context generation and final result selection stages. Therefore, we opted for a more efficient approach to ensure feasibility within our overall framework. Using unigrams for entropy calculations could be a deliberate choice due to certain trade-offs.\n\n\n    Based on the aforementioned three points, we have chosen the unigram approach for computing information entropy. However, we acknowledge that model logits can yield comparable results in certain scenarios and there may be other potentially superior methods for calculating information entropy.\n\n\n - ***How are the Manual results obtained for datasets without CoT examples included?***\n\n   - The manual results for datasets without CoT examples included were obtained by randomly selecting examples from the training set and manually composing chains of thought for them as we demonstrated in the Baseline part in section 4.1 Experiment settings. This process was followed based on the approach described by Wei et al. [1]. There is an example in the MultiA dataset:\n\n     > Question: Tom bought 12 boxes of chocolate candy and gave 7 to his little brother. If each box has 6 pieces inside it, how many pieces did Tom still have?\n     >\n     > Answer: Let's think step by step.\n     > Tom had 12 boxes of candy originally. Then he gave 7 to his little brother, so he had 12 - 7 = 5 boxes of candy left. Each box has 6 pieces inside it, so Tom still had 5 x 6 = 30 pieces.\n     >\n     > Therefore, the answer (arabic numerals) is 30.\n\n    \n\n- ***The paper is written as though CoT was central, but actually four of seven the datasets have CoT strings included, and so all that is happening is that demonstrations are being selected and ranked. What happens if the include CoT strings are not used?***\n\n  - As we mentioned in section 3.3, when the data lacks annotated CoT,  we utilize our selection strategy to acquire a specific set of queries first. Following the approach outlined by Kojima et al. [2], rationale steps are generated through zero-shot CoT. We further integrated our method into the process of CoT generation. The effectiveness was shown in the ''Effects of IE in CoT generation\" part in section 5.\n    As for the case of not using CoT, there will be a significant collapse in the results, which has been extensively discussed and experimented in [1]. We also conducted experiments on GPT-3.5-Turbo and the results are as follows：\n\n    |              | Addsub | AQUA  | Strategyqa |\n    | ------------ | ------ | ----- | ---------- |\n    | one-shot     | 84.28  | 50.19 | 35.96      |\n    | one-shot-cot | 88.59  | 53.81 | 44.58    |\n\n    \n\n- ***Are the results all based on single runs? If so, what is the observed variation across runs and across choices for the many hyperparameters that are in play here? If these hyperparameters were tuned informally based on pilot experiments, what examples were used for that and how extensively was this done across all the approaches?***\n\n  - Yes, most of our experiments were conducted based on single runs. In the absence of self-consistency setting, we set the temperature of models to 0, and there was no additional randomness in our example selection stage, which means that for a specific dataset, our inputs remained the same for each run, resulting in deterministic outcomes. However, in the self-consistency setting, we still relied on single runs as we believe that the self-consistency setup itself partially eliminates randomness and also helps in computational efficiency.\n\n   - Regarding hyperparameters, we mostly followed the experimental settings of Wei et al.[1] and Wang et al.[3], such as the number of examples used for each dataset, temperature settings, and the minimum sampling count for self-consistency. These hyperparameters have been explored in their paper to understand their impact on the results. \n   - For the selection of H_min, H_max, N_min, and N_max. As mentioned after Equation 3 in our paper, H_min and H_max represent the minimum and maximum IE scores of the candidates, respectively. These values are calculated by equation 1 on different datasets. Typically, we have observed that the values are 3 and 8.\n\n      Regarding N_min and N_max, they indicate the minimum and maximum numbers of samplings for self-consistency. In our experiment, we followed the approach proposed by Wang et al. [3] and selected 5 as the value for N_min. The reason we selected 20 for N_max was based on our previous experiments, where we found that increasing the number of samplings beyond 20 did not yield significant further improvements in self-consistency. We believe that these values strike a balance between computational efficiency and achieving desirable results. We appreciate the reviewer's inquiry, and we hope this explanation clarifies the rationale behind our choices.\n   - Additionally, there are other hyperparameters such as the number of CoT generations and the threshold for similarity pruning, which were set empirically. However, the focus of this paper is not on tuning these hyperparameters in every stage for their subtle effects on the results, but rather on exploring the contributions of each component. Perhaps in future work, more in-depth research can be conducted on these hyperparameters.\n\n***We hope we have addressed all the questions and thank you again for your valuable feedback. Please let us know if there are any additional concerns***\n\n[1] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.\n\n[2] Kojima T, Gu S S, Reid M, et al. Large language models are zero-shot reasoners[J]. Advances in neural information processing systems, 2022, 35: 22199-22213.\n\n[3] Wang X, Wei J, Schuurmans D, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models[C]//The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a very lightweight addition to the toolkit for selecting demonstations and CoT strings. If I understand correctly, the entropy calculations are done entirely on the whitespace tokenized versions of the strings, so it doesn't even require access to model logits.\n\nThe reported results are very strong, and the paper does a good job of seeking to assign credit to the various components of the system.", "reasons_to_reject": "I do not have reasons to reject per se, but I do have come crucial question for the authors, and the answers might lead me to adjust my overall perspective.\n\nThe paper's evaluation should rest entirely on the results. In terms of scientific ideas: the system is just a pipeline of threshold-based decisions using a simply notion of entropy. This is great if it truly works.", "questions_to_authors": "1. Can you confirm that the entropy calculations are done on the unigrams? It's not really clear from the paper, but this is what I inferred from the code. Why not use the model logits for text-davinci-003 at least?\n\n2. How are the Manual results obtained for datasets without CoT examples included?\n\n3. The paper is written as though CoT was central, but actually four of seven the datasets have CoT strings included, and so all that is happening is that demonstrations are being selected and ranked. What happens if the include CoT strings are not used?\n\n4. Are the results all based on single runs? If so, what is the observed variation across runs and across choices for the many hyperparameters that are in play here? If these hyperparameters were tuned informally based on pilot experiments, what examples were used for that and how extensively was this done across all the approaches?", "paper_topic_and_main_contributions": "This paper develops a new method for selecting demonstrations and associated chain-of-thought elements. The method is based on the unigram-based entropy of these elements – more surprising elements are favored to be chosen, presumably because they have higher information content. The entropy of the test item is also used to sort the demonstations and to dynamically select the number of demonstrations, and CoT elements are filtered to those that get the correct answer. The results consistently strong along a number of QA datasets.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After spirited discussion, the reviewers agree that this is a sound paper with a surprising and interesting result: you can use entropy to select with chain-of-thought rationales are effective for QA tasks. Where there is less agreement among the reviewers is how excited we should be about this. While this is a useful tool, the method seems to be dependent on some important details not adequately specified in the submission (but addressed in the author response period) and thus hard to replicate, and this would be a stronger paper if there were more emphasis/discussion on why this works. However, many practitioners will be immensely interested in the "how", which may be enough.
Decision: Accept-Main