Venue: NAACL 2024

# Program-Aided Reasoners (_Better_) Know What They Know

Anonymous ACL submission

###### Abstract

Prior work shows that program-aided reasoning, in which large language models (LLMs) are combined with programs written in programming languages such as Python, can significantly improve accuracy on various reasoning tasks. However, while accuracy is essential, it is also important for such reasoners to "know what they know", which can be quantified through the _calibration_ of the model. In this paper, we compare the calibration of Program Aided Language Models (PAL) and text-based Chain-of-thought (CoT) prompting techniques over 5 datasets and 2 model types - LLaMA models and OpenAI models. Our results indicate that PAL leads to improved calibration in 75% of the instances. Our analysis uncovers that prompting styles that produce lesser diversity in generations also have more calibrated results, and thus we also experiment with inducing lower generation diversity using temperature scaling and find that for certain temperatures, PAL is not only more accurate but is also more calibrated than CoT. Overall, we demonstrate that, in the majority of cases, program-aided reasoners _better_ know what they know than text-based counterparts.1

Footnote 1: Anonymized code and data are available at [https://anonymous.4open.science/r/code-calibrates-A6ID/](https://anonymous.4open.science/r/code-calibrates-A6ID/).

001


In sum, the previous research has shown, as eloquently stated by Kadavath et al. (2022) "language models _(mostly)_ know what they know" -- LLMs are reasonably well calibrated, although some imperfections remain.

In this work, we examine the effect of program-aided reasoning on calibration. We consider five datasets that cover different reasoning tasks and evaluate the performance of both PaL and CoT style prompting for OpenAI models (OpenAI, 2023) and LLaMA models (Touvron et al., 2023) with respect to accuracy and calibration. We primarily explore three main research questions :

* **RQ 1:** Does program-aided reasoning result in significantly different calibration than text-based CoT?
* **RQ 2:** Are the observed trends different across OpenAI models and LLaMA models?
* **RQ 3:** Does the consistency of LLM generations affect calibration? We examine this by measuring generation diversity and answer space entropy.

Our results show that program-aided reasoners know what they know _even better_ than standard text-based reasoners with CoT. In particular, on OpenAI models, PaL exhibits not only superior accuracy but also a consistent enhancement in calibration of about 50%, over CoT. Interestingly, the consistent improvement of calibration is not observed in LLaMA models. Still, we find that adjusting the temperature of sampling (similar to a widely used method of Platt scaling (Platt et al., 1999), PaL improves with respect to accuracy and calibration. We also conduct a detailed analysis of these observations and find a correlation between the similarity of the generated chains-of-thoughts or programs and calibration, which might help explain these trends. Code and data available here under the Apache 2.0 license.

## 2 Preliminaries and Mathematical Formulation

### Measuring Calibration

Calibration refers to the alignment between the predicted probability estimates of a model and their actual correctness or accuracy (Guo et al., 2017b). Formally, a perfectly calibrated model can be expressed using the following equation, where \(X\) is the given input, \(Y\) is the true output, the model's output is \(\hat{Y}\) and \(P_{N}(\hat{Y}~{}|~{}X)=p\) is the probability, or "confidence", over the model's output.

\[P\left(\hat{Y}=Y~{}|~{}P_{N}(\hat{Y}~{}|~{}X)=p\right)=p,\forall p\in[0,1] \tag{1}\]

In essence, Equation 1 conveys that if a perfectly calibrated model makes 100 predictions, and the confidence of each prediction is 0.6, then we expect the accuracy to be also 0.6. Nevertheless, the model may exhibit varying confidence levels for each sample. Therefore, it is imperative to calculate calibration across all confidence scores. We estimate this probability by dividing the predictions into \(M\) separate and equally sized interval buckets based on their confidence levels.

We use the expected calibration error (ECE), a common measure of (lack of) calibration, a weighted average of the discrepancy between each bucket's accuracy and confidence. It is given in Equation 2

Here \(B_{m}\) is the _m_-th bucket that contains samples whose probabilities of predictions fall in the

Figure 2: Illustration of eliciting model confidence through self-consistency



interval \({\left(\frac{m-1}{M},\frac{m}{M}\right]}\), where \(\frac{|B_{m}|}{n}\) is \(B_{m}\)'s size relative to all the samples. \(\mathrm{acc}\left(B_{m}\right)\) is the average accuracy of the samples in the _m_-th bucket, and \(\mathrm{conf}\left(B_{m}\right)\) is the corresponding average confidence of the samples falling in the _m_-th bucket.

\[\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\mathrm{acc}\left(B_{m}\right)-\mathrm{ conf}\left(B_{m}\right)\right| \tag{2}\]

Consider a setup where we have buckets with a width of 0.1. All instances where a model assigns probabilities between 0.4 and 0.5 will be allocated to the bucket \(B_{5}\) or the bucket encompassing probabilities between 0.4 and 0.5. Subsequently, the average accuracy for the instances in these buckets and the average probability/confidence is computed. The absolute difference of the accuracy and confidence is multiplied by the proportion of total instances in a bucket. This process is repeated for every bucket, and the individual scores are summed up to calculate ECE.

### Self-consistency as a measure of confidence

Self-consistency Wang et al. (2022) is a natural language reasoning technique that uses chain-of-thought prompting to generate multiple paths for reasoning. This process aims to select the most consistent answer by sampling and marginalizing. Here, we use a latent variable \(Z\) to represent the reasoning chain/programs. \(Y\) is the answer that is either extracted in case of CoT or obtained after execution in case of PaL. We marginalize over \(Z\) by taking a majority vote over answers. Thus, we rely on majority voting over the answers to obtain confidence estimates for each sample.

\(K\) is a hyperparameter that controls the number of generations (referenced in equation 3). The higher the value of \(K\), the better our approximation of the probability of each sample. Figure 2 shows an overview of this process.

\[P(\hat{Y}_{0}|Z_{0})=\frac{1}{K}\sum_{i=0}^{K}\mathbbm{I}\left\{\hat{Y}_{i}= \hat{Y}_{0}\right\} \tag{3}\]

Wang et al. (2022) and Xiong et al. (2023a) suggest that self-consistency can be an effective way to elicit confidence from models. Hence, given the lack of per-token log probabilities in closed LMs like gpt-3.5-turbo and text-davinci-003, we adopt self-consistency as a proxy measure for calibration.

### Similarity and Answer Entropy

In addition to empirically evaluating the impact on accuracy and calibration, we conduct a qualitative analysis of the reasoning chains (the latent variable \(Z\) described previously). Here, we observe a consistent pattern, i.e. the correct answers corresponding to a question are often associated with similar generations.

We find that this observation aligns with the finding made by Li et al. (2022), that there are numerous ways in which solutions can be incorrect. In contrast, correct solutions tend to exhibit more uniform behaviour.

To empirically validate this observation, we employed sentence embeddings generated from the _all-MiniLM-v6_ model to compute the average similarity among the generations/reasoning chains, equivalent to calculating similarity over latent variables \(Z\).

Furthermore, to gain deeper insights into the relationship between similarity in generations and corresponding answers, we compute the entropy \(H(A)\) of the answer space where \(P(a_{i})\) refers to the probability of the \(i^{th}\) answer in K answers obtained by extraction or program execution for a given sample.

\[H(A)=-\sum_{i=1}^{K}P(a_{i})\cdot\log_{2}P(a_{i}) \tag{4}\]

This allowed us to investigate whether the observed similarity in the latent variable space \(Z\) leads to a lower entropy within the answer space.

## 3 Experimental Design

### Models

We compare the calibration and accuracy of two different prompting strategies - CoT and PaL on an equal number of closed-source and open-source models. The open source models used in experimentation are LLaMA2-13B, LLaMA2-70B Touvron et al. (2023). and the closed-source models are gpt-3.5-turbo, text-davinci-003 Brown et al. (2020). It should be noted that all models have received some form of supervision from code during pre-training OpenAI (2023; Touvron et al., 2023), in addition to being primarily trained on text. For LLaMA models, we leveraged vLLM Kwon et al. (2023) for distributed inference using A6000 GPU(s).

 



### Hyperparameters

For our experiments, we set temperature (T) as 1.0 and the probability (p) for nucleus sampling Holtzman et al. (2020) as 1.0 Selecting a temperature of 1.0 enables direct sampling from the model as no scaling of probabilities is involved, as seen from Equation 5. Here, \(z_{i}\) refers to the logit for the \(i\)th token generated, and \(N\) is the vocabulary size.

\[\sigma\left(z_{i}\right)=\frac{e^{\frac{z_{i}}{T}}}{\sum_{j=0}^{N}e^{\frac{z_{ j}}{T}}} \tag{5}\]

For use \(K=10\) generations per sample for all datasets. We set the maximum number of tokens (input + output) for each generation to 1024.

### Tasks

We examined reasoning tasks encompassing several challenges, including arithmetic, algorithmic, and symbolic reasoning. We use five datasets that cover these different kinds of reasoning tasks. The arithmetic reasoning datasets include _GSM8K_Cobbe et al. (2021) and _GSM8K Hard_Gao et al. (2022). The algorithmic reasoning tasks include _Object-Counting_Suzgun et al. (2022a) and _RepeatCopy_Suzgun et al. (2022a). We used _Date-Understanding_ as a Symbolic Reasoning Dataset Suzzgun et al. (2022a). Specific information about the datasets used can be found in Table 1.

### Prompt Design

We provide all models with natural language chain-of-thought (CoT) prompts and code-based Program-Aided Language Model (PaL) prompts. For datasets where CoT prompts are available in their original form, we use them as presented in the original paper Wei et al. (2022). We modify these prompts for other datasets to suit the specific task while maintaining their original format. For PaL prompts, we use and adapt the code prompts provided in Gao et al. (2022). The prompts are included in Appendix A.

## 4 Results

We investigate two model types: OpenAI models and LLaMA models along with the two different prompting strategies - PaL and CoT.

### Effect of prompting style on Calibration

In this section, we look at the first two _RQs_:

**RQ 1:**_Does one prompting style result in significantly better calibration than the other?_

**RQ 2:**_Are the observed calibration trends different across OpenAI models and LLaMA models?_

Table 2 shows results for OpenAI models, we observe that PaL prompting improves both calibration and accuracy across all datasets. We see approximately \(50\%\) relative reduction in calibration error and an average improvement of 18.42% in accuracy.

In Figure 3, we show reliability plots which illustrate improved calibration, with the reliability curves for PaL prompting consistently aligning closer to the ideal reliability curve as compared to CoT across datasets. While PAL shows a notable gain of 14.83% in accuracy across all datasets for LLaMA models, it shows better calibration in only half of our settings. Overall, for both OpenAI models and LLaMA models, we observe that PaL leads to better calibration than CoT for 75% of the settings. The reliability plots for all datasets for the models gpt-3.5-turbo and LLaMA2-7bB can be seen in Appendix Section E, D.

Effect of PaL on calibration controlling for accuracyOne reasonable hypothesis is that PaL is improving calibration because it achieves higher accuracy, and more accurate models can be better calibrated. To examine this hypothesis, we conduct statistical analysis using _mixed linear models_McLean et al. (1991), which allows us to consider

\begin{table}
\begin{tabular}{l l c p{162.3pt}} Dataset & Category & \# Samples & Example \\ GSMSK Cobbe et al. (2021) & Arithmetic & 1319 & Q: A probe takes 2 bolts of blue fiber and half that much white fiber. From many bolts in total does it take? A: 3 \\ GSMSK Hard Gao et al. (2022) & Arithmetic & 1319 & Q: A probe takes 2287720 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? A: 3431580 \\ Date Understanding Suzzgun et al. (2022a) & Symbolic & 360 & Q: Yesterday was April 30, 2021. \\  & & What is the date today in MM/DDYYYYY *: 050/12021 \\ Object Counting Suzzgun et al. (2022a) & Algorithmic & 250 & Q: I have three couches, a lamp, a stove, a table, a fridge, and a microwave. How many objects do I have? A: 8 \\ Repeat Copy Suzzgun et al. (2022a) & Algorithmic & 32 & Q: say python twice and data once, and then repeat all of this three times. A: python python data python python data python python data \\ \hline \end{tabular}
\end{table}
Table 1: Datasets with their examples and categories.



the significance of varying the prompting strategy while controlling for accuracy as a confounding factor.

Upon analyzing the results in Table 3, we observe that, when treating the prompting style as a fixed effect, PAL exhibits a negative coefficient of -0.103 (p=0.0) for OpenAI models, which is statistically significant with a threshold of p=0.05. This implies that PAL contributes to the reduction in ECE and has a positive impact on calibration. On the contrary, for LLaMA models, we did not find that PAL had a statistically significant effect on ECE after controlling for accuracy. Across LLaMA models and OpenAI models, PAL has a statistically significant (p=0.02) correlation of -0.067 with ECE, indicating that PAL helps increase calibration on the whole even when controlling for accuracy.

To summarize, we see that PAL prompting has better calibration than CoT prompting (_\(\mathbf{-RQI}\)_) . While PAL has improved calibration in all settings for OpenAI models, this trend is less consistent for LLAMA models (_\(\mathbf{-RQ2}\)_).

### Effect of generation diversity on calibration

In this section, we look at the third research question: _RQ 3: Does the consistency of LLM generations affect calibration?_

Qualitative analysis of the generations reveals that PAL generations adhere to a consistent structure that divides the problem-solving process into three distinct parts. This is depicted in Figure 4. In the first part, the model initializes the variables and sets up their initial values required for the calculation. This part is straightforward due to syntactic constraints and remains largely similar across generations. In the second part, the model generates the required logic by applying formulas and utilizing various operations to derive the desired result. Finally, in the third part, the model generates the answer by assigning the calculated result to a variable and returning it, which, again, doesn't vary much across generations. Hence, the diversity of the generation is mainly limited to the second part, making code more constrained in its generation space compared to text. Therefore, there is a **standardized structure in the code** generated by language models with PaL prompts.

Lower generation diversity and answer entropy observed in prompting strategy with better calibrationTo quantitatively analyze if code-based generations have lower generation diversity and lead to a narrower answer space, we computed aggregated cosine similarity scores for all the generations and entropy over the answer space. For OpenAI models, we note that the cosine similarity scores with PaL are higher than the corresponding scores for CoT. This observation suggests that code-based generations display a higher degree of similarity from a semantic perspective. Moreover, the answer entropy for PaL is lower than CoT. This implies that similar generations that cluster together in the semantic space Li et al. (2022a) also converge to the equivalent solution space. This

\begin{table}
\begin{tabular}{c|l l c c c c c c c c c} \hline Name & Score & Model & GSMK & Object-Counting & Repress-Copy & Date & Undertraining & GSMK Hard \\  & & & CGT & Pal. & CGT & Pal. & CGT & Pal. & CGT & Pal. & CGT & Pal. \\ \hline \multirow{8}{*}{LLoRa2-700} & ECE (J) & LLMA & 0.19 & 0.00 & 0.17 & 0.18 & 0.18 & 0.23 & 0.00 & 0.18 & 0.07 & 0.01 \\  & ACC (T) & LLMA & 59.2 & 0.93 & 9.60 & 9.20 & 0.46 & 72.18 & 86.66 & 70.18 & 21.45 & +0.82 \\  & SDI (J) & LLMA & 72.0 & 0.24 & 0.44 & 9.47 & 9.71 & 9.00 & 88.87 & 8.15 & 92.78 & 21.43 \\  & ENT (J) & LLMA & 2.24 & 1.92 & 1.00 & 0.76 & 1.93 & 2.00 & 1.48 & 1.54 & 2.85 & 2.17 \\  & ECO (J) & LLMA & 0.06 & 0.08 & 0.08 & 0.06 & 0.11 & 0.17 & 0.06 & **0.05** & 0.12 & 0.14 \\  & ACC (T) & LLMA & 27.0 & 34.34 & 56.4 & 81.6 & 34.77 & 58.12 & 48.24 & 50.41 & 6.67 & 25.85 \\  & SHU (J) & LLMA & 76.6 & 93.3 & 93.2 & 95.3 & **99.8** & 88.6 & 79.5 & 84.2 & 74.0 & **92.32** \\  & ENT (J) & LLMA & 2.83 & 2.49 & 1.55 & 0.05 & 24.3 & 4.77 & 22.36 & 20.6 & 2.42 & 3.06 \\ \cline{2-11}  & ECE (J) & Opcanal & OL & 0.03 & 0.29 & **0.02** & 0.20 & **0.06** & 0.19 & 0.11 & 0.15 & 0.07 \\ \cline{2-11} test-duxieni-60x & AC (T) & Opcanal & GG & 76.5 & 76.5 & 92.1 & **95.0** & 67.23 & **93.75** & 70.73 & 23.95 & **71.27** \\  & SHU (J) & OPAN & 90.5 & 97.8 & 99.1 & 99.8 & 96.2 & 98.2 & 92.4 & 97.4 & 98.8 & 97.9 \\  & ENT (J) & Opcanal & I.27 & 0.79 & 3.06 & **0.02** & 1.38 & 0.44 & 0.71 & 0.64 & 2.31 & 0.81 \\ \cline{2-11}  & REG (J) & Opcanal & O & 0.05 & **0.03** & 0.18 & 0.03 & 0.18 & 0.16 & 0.17 & 0.13 & 0.13 & **0.05** \\ gpt-1.5-turbe & ACC (T) & Opcanal & **84.0** & 82.40 & 82.40 & 82.73 & 97.25 & 86.5 & 61.51 & **71.23** & 55.21 & 62.91 \\  & SNI (J) & Opcanal & Q & 94.0 & 93.09 & 99.09 & 98.00 & 97.70 & 99.00 & 95.3 & **97.6** & 96.00 & 95.40 \\  & ENT (J) & Opcanal & o & 0.57 & **0.49** & 0.59 & **0.08** & 1.15 & 0.35 & 0.50 & **0.36** & **1.65** & 2.43 \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of Expected Calibration Error (ECE (J) ) , Accuracy (ACC (†) ) , Cosine Similarity (SIM (†) ) and Answer Entropy (ENT (�) ) across datasets. The darker blue shade highlights better performing prompting technique.

\begin{table}
\begin{tabular}{r|l|c c} \hline Model Type & LLMA models & OpenAI models & Both \\ \hline Fixed Effect & PAL : 0.010 & PAL : 0.103 & PAL : 0.103 & PAL : 0.067 \\ (ECE vs Promping Style) & 0.961 & 0.000 & 0.002 \\ \hline \end{tabular}
\end{table}
Table 3: Statistical analysis using mixed linear models, keeping ECE vs Prompting Style as a fixed effect and accuracy as a random effect.

 

leads to lower uncertainty in the probability distribution of the answer space and, hence, lower entropy. From Table 2, we thus can see that PaL helps produce similar generations that converge to the same answer space, which is also consistently correct. Hence, it achieves better performance and provides more confidence in its predictions.

For LLaMA models, we don't see this trend of PaL having higher generation similarity and lower answer entropy for all datasets. However, for almost all settings for LLaMA models and OpenAI models, the prompting strategy that produces more similar generations and lower answer entropy is also more calibrated.

To summarize, it is evident that lower generation diversity and lower answer entropy are correlated with higher calibration. (_-RQ3_)

**Better calibration observed for PaL when inducing similarity in generations for LLaMA models** We observe that for OpenAI models, PaL is not only more accurate but also more calibrated than CoT. Consequently, we explore whether the reduction in generation diversity, achievable through lower temperatures, can contribute to improved calibration for LLaMA models.

We perform a parameter sweep across temperature values between 0.1 and 0.7 with a step size of 0.2. We show the variation of accuracy, calibration, generation similarity, and answer entropy for two datasets in Figure 5. The plots for the remaining datasets are available in Appendix B, Figure 6. We can see that we obtain better calibration for both the LLaMA models in both PaL and CoT for temperatures below 1.0. From Tables 4 and 5, we note that in the majority of runs with T < 1.0, PaL is better calibrated than CoT. Considering accuracy and calibration, optimal performance is achieved at different temperatures for each dataset. For most T values, the similarity scores are higher while corresponding answer entropy values are lower for PaL compared to CoT. This mirrors the pattern

Figure 4: Typical output structure with PaL

Figure 3: Reliability Plots for various structured reasoning tasks for the model gpt-3.5–turbo. The x-axis represents confidence, and the y-axis represents accuracy.

 

observed for OpenAI models. For LLaMA-2 13b, PAL displays better calibration than CoT at lower temperatures. However, the optimal temperature for obtaining the best performance for calibration and accuracy is still T=1.0.

[MISSING_PAGE_POST]

* [noitemsep]
* [noitem]
* [noitemsep]
* [noitem]
* [noitemsep]
* [noitem]
* [noitem]
 



### Utilizing Language for Code Generation

The exploration of using natural language for code generation has taken diverse approaches in research. Initial efforts involved rule-based, predictive and deep-learning variations (Gulwani and Marron, 2014; Woods, 1973; Zelle and Mooney, 1996; Lin et al., 2017; Rabinovich et al., 2017). However, performance enhancements were observed using pre-trained models trained on code-based datasets (Chen et al., 2021; Nijkamp et al., 2022; Gao et al., 2022; Li et al., 2023). Employing pre-trained language models (LMs) for code generation as a way to solve tasks that require step-by-step structuring and various forms of reasoning has proven to be particularly effective (Ni et al., 2023a; Gao et al., 2022; Ni et al., 2023b). Intermediate execution results from code have been used for training (Chen et al., 2018) and inference (Wang et al., 2018). Majority-based voting on the results of code executions (which is the self-consistency-based methodology we employ) has also been shown to be an effective technique for selecting the right candidate (Li et al., 2022b; Cobbe et al., 2021; Shi et al., 2022).

## 6 Conclusion

In this study, we explore the impact of two distinct prompting styles, namely PaL and CoT, on the calibration of OpenAI models and LLaMA models. Our investigation spans 5 reasoning datasets, employing self-consistency as the methodology for eliciting calibration. We analyze four different metrics - calibration (ECE) , accuracy (ACC) , average similarity in generations (SIM) , and answer entropy (ENT) . Our findings are as follows:

* _RQ 1: Does one prompting style result in significantly better calibration than the other?_ Empirical results show that PaL generally has higher calibration and accuracy for 82.5% of the cases across OpenAI and LLaMA models for a varied range of temperatures.
* _RQ 2: Are the observed calibration trends different across OpenAI models and LLaMA models?_ We observed that OpenAI models are in general better calibrated for the reasoning tasks with up to 19% improvement in ECE.
* _RQ 3: Does the consistency of LLM generations affect performance?_ PaL prompting shows a general trend of having greater similarity in the generation over CoT, which we hypothesize could be due to the inherent structure present in the code. We see that greater generation similarity is accompanied by lower answer entropy and lower ECE.

We hope that this study will catalyze additional research aimed at holistically evaluating and gaining deeper insights into the role of prompts in various tasks and domains.

## 7 Limitations

Access to OpenAI models is only available through an API which limits the ability to exactly control the hyperparameters influencing the genera

Figure 5: Trends seen in temperature scaling for the model LLaMA2-7OB. Across datasets, the accuracy and calibration improve upon lowering the temperature to a certain extent. This is in line with having lower generation similarity and lower answer entropy. The optimal temperatures seen are 0.5 and 0.7 across datasets. For other datasets, refer Appendix, Figure 6.



tions. Moreover, OpenAI models are not transparent, which limits the ability to study these models. Because of this lack of transparency it is also hard to draw conclusive insights about any comparisons between OpenAI models and LLAMA models In our study, we report the results from a single run but due to combination of utilizing temperature value of 1.0 and hardware induced stochasticity, it is possible to get varying results for a given model.

## References

* Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Chen et al. [2022] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. _ArXiv_, abs/2211.12588.
* Chen et al. [2018] Xinyun Chen, Chang Liu, and Dawn Xiaodong Song. 2018. Execution-guided neural program synthesis. In _International Conference on Learning Representations_.
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _ArXiv_, abs/2110.14168.
* Desai and Durrett [2020] Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. _arXiv preprint arXiv:2003.07892_.
* Gao et al. [2022] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamic Callan, and Graham Neubig. 2022. Pal: Program-aided language models. _ArXiv_, abs/2211.10435.
* Gulwani and Marron [2014] Sumit Gulwani and Mark Marron. 2014. Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation. In _Proceedings of the 2014 ACM SIGMOD international conference on Management of data_, pages 803-814.
* Guo et al. [2017a] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017a. On calibration of modern neural networks. In _International conference on machine learning_, pages 1321-1330. PMLR.
* Guo et al. [2017b]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017b. On calibration of modern neural networks. In _International Conference on Machine Learning_.
* Holtzman et al. [2020] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
* Jagannatha and Yu [2020] Abhyuday Jagannatha and Hong Yu. 2020. Calibrating structured output predictors for natural language processing. In _Proceedings of the conference. Association for Computational Linguistics. Meeting_, volume 2020, page 2078. NIH Public Access.
* Jiang et al. [2020] Zhengbao Jiang, J. Araki, Haibo Ding, and Graham Neubig. 2020. How can we know when language models know? on the calibration of language models for question answering. _Transactions of the Association for Computational Linguistics_, 9:962-977.
* Jiang et al. [2021] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. _Transactions of the Association for Computational Linguistics_, 9:962-977.
* Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. _arXiv preprint arXiv:2207.05221_.
* Kamath et al. [2020]Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective question answering under domain shift. _arXiv preprint arXiv:2006.09462_.
* Kong et al. [2020] Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020. Calibrated language model fine-tuning for in-and out-of-distribution data. _arXiv preprint arXiv:2010.11506_.
* Kwon et al. [2023] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. _Proceedings of the 29th Symposium on Operating Systems Principles_.
* Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marcarone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! _arXiv preprint arXiv:2305.06161_.
* Li et al. [2022a] Yujia Li, David Choi, Junyoung Chung, Mate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022a. Competition-level code generation with qlapnode. _Science_, 378(6624):1092-1097.
* Li et al. [2020] Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Remi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, 


* [631] Xi Victoria Lin, Chenglong Wang, Deric Pang, Kevin Vu, and Michael D Ernst. 2017. Program synthesis from natural language using recurrent neural networks. _University of Washington Department of Computer Science and Engineering, Seattle, WA, USA, Tech. Rep. UW-CSE-17-03-01_.
* [632] Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. _arXiv preprint arXiv:2301.13379_.
* [633] Aman Maaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. _ArXiv_, abs/2210.07128.
* [634] Robert A McLean, William L Sanders, and Walter W Stroup. 1991. A unified approach to mixed linear models. _The American Statistician_, 45(1):54-64.
* [635] Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023a. Lever: Learning to verify language-to-code generation with execution. In _International Conference on Machine Learning_, pages 26106-26128. PMLR.
* [636] Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq R. Joty, Yingbo Zhou, Dragomir R. Radev, and Arman Cohan. 2023b. L2ceval: Evaluating language-to-code generation capabilities of large language models. _ArXiv_, abs/2309.17446.
* [637] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Liftu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis.
* [638] OpenAI. 2023. Opensai documentation. [https://platform.openai.com/docs/model-index-for-researchers](https://platform.openai.com/docs/model-index-for-researchers).
* [639] John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. _Advances in large margin classifiers_, 10(3):61-74.
* [640] Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. _arXiv preprint arXiv:1704.07535_.
* [641] Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural language to code translation with execution. _ArXiv_, abs/2204.11454.
* [642] Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastien Gehrmann, Yi Tay, Hyung Won Chung, Kaakansha Chowdhery, Quoc V Le, Ed H Chi, Dennis Zhou, et al. 2022a. Challenging big-bench tasks and whether chain-of-thought can solve them. _arXiv preprint arXiv:2210.09261_.
* [643] Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aghasksha Chowdhery, Quoc V. Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei. 2022b. Challenging big-bench tasks and whether chain-of-thought can solve them. _ArXiv_, abs/2210.09261.
* [644] Catherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. _arXiv preprint arXiv:2305.14975_.
* [645] Hugo Tourvon, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahari, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajijwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* [646] Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polozov, and Rishabh Singh. 2018. Robust text-to-sql generation with execution-guided decoding. _arXiv Computation and Language_.
* [647]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _ArXiv_, abs/2203.11171.
* [648]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. _ArXiv_, abs/2201.11903.
* [649]William A Woods. 1973. Progress in natural language understanding: an application to lunar geology. In _Proceedings of the June 4-8, 1973, national computer conference and exposition_, pages 441-450.
* [650] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023a. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in lms. _ArXiv_, abs/2306.13063.
* [651]Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023b. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. _arXiv_, _arXiv:2306.13063_.
* [652]Shunya Yao, Dian Yu, Jeffrey Zhao, Irzhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narayanhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. _ArXiv_, abs/2305.10601.

 

* [743] John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In _Proceedings of the national conference on artificial intelligence_, pages 1050-1055.
* [747] Li Zhang, Liam Dugan, Hai Xu, and Chris Callison-Burch. 2023a. Exploring the curious case of code prompts. _ArXiv_, abs/2304.13250.
* [750] Li Zhang, Hai Xu, Yue Yang, Shuyan Zhou, Weiqiu You, Manni Arora, and Chris Callison-Burch. 2023b. Causal reasoning of entities and events in procedural texts. In _Findings_.
* [751] Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In _International Conference on Machine Learning_, pages 12697-12706. PMLR.
* [752] Denny Zhou, Nathanaael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. _ArXiv_, abs/2205.10625.

 

Prompts
* [766] The following sections display one example of the few-shot prompts used for each dataset across prompting styles.
* [768]
* [769] A.1.1 GSM8K/GSM8K-Hard

[MISSING_PAGE_POST]

 



### CoT Prompts
* [79] A.2.1 GSM8K/GSM8K-Hard

Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today?

A: There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is 6.
* [80]

#### a.2.2 Object Counting

Q: I have a chair, two potatoes, a cauliflower, a lettuce head, two tables, a cabbage, two onions, and three fridges. How many vegetables do I have?

A: Chair, tables and fridges are not vegetables, so we are not counting them. Two potatoes, caulflow filter, lettuce head, cabbage and two onions are vegetables, so they will be counted The total number of vegetables are 2 + 1 + 1 + 1 + 2 The answer is: 7
* [82]

#### a.2.3 Date Understanding

Q: 2015 is coming in 36 hours. What is the date one week from today in MM/DD/YYYY?

A: If 2015 is coming in 36 hours, then it is coming in 2 days. 2 days before 01/01/2015 is 12/30/2014, so today is 12/30/2014. So one week from today will be 01/05/2015. So the answer is 01/05/2015.

#### a.2.4 Repeat Copy

Q: Repeat the word duck four times, but halfway through also say quack

A: On repeating the word duck four times we get: duck duck duck duck Halfway through if we say quack, we have to say quack in between the 2nd word and the 3rd word The answer is: duck duck duck duck duck



## Appendix B Temperature Scaling Experiments - Line Plots

Figure 6: Trends seen in temperature scaling for the datasets - _GSM8K-Hard_, _Date-Understanding_ and _Repeat-Copy_

Results of temperature scaling for LLaMA2-13B
* [790]

\begin{table}
\begin{tabular}{c c|c c|c c|c c|c c|c c} \hline Temp & \multicolumn{4}{c}{GSM8K} & Object-Counting & \multicolumn{4}{c}{Repeat-Copy} & \multicolumn{4}{c}{Date-Understanding} & GSM8K Hard \\ \hline  & & CoT & PaL & CoT & PaL & CoT & PaL & CoT & PaL & CoT & PaL \\ \hline
0.7 & ECE & 0.052 & 0.046 & 0.12 & 0.108 & 0.175 & 0.181 & 0.108 & 0.107 & 0.125 & 0.087 \\  & ACC & 36.16 & 39.27 & 61.60 & 80.80 & 34.37 & 59.37 & 50.60 & 56.63 & 10.91 & 30.32 \\  & SIM & 85.40 & 97.48 & 98.65 & 99.33 & 95.07 & 94.02 & 86.81 & 96.03 & 83.48 & 97.37 \\  & ENT & 2.405 & 2.222 & 0.970 & 0.303 & 2.048 & 1.949 & 1.505 & 1.330 & 2.892 & 2.332 \\ \hline
0.5 & EEC & 0.099 & 0.093 & 0.171 & 0.135 & 0.131 & 0.106 & 0.134 & 0.183 & 0.177 & 0.1381 \\  & ACC & 36.08 & 37.07 & 60.80 & 82.00 & 34.37 & 59.30 & 53.92 & 55.00 & 10.80 & 29.87 \\  & SIM & 88.70 & 98.10 & 99.21 & 99.83 & 97.12 & 97.74 & 89.94 & 97.63 & 87.37 & 98.03 \\  & ENT & 2.144 & 1.976 & 0.722 & 0.158 & 1.455 & 1.311 & 1.141 & 0.942 & 2.694 & 2.074 \\ \hline
0.3 & ECE & 0.160 & 0.108 & 0.231 & 0.154 & 0.200 & 0.206 & 0.219 & 0.304 & 0.246 & 0.231 \\  & ACC & 33.81 & 30.72 & 62.40 & 81.20 & 37.50 & 65.63 & 55.28 & 50.40 & 10.16 & 27.97 \\  & SIM & 91.51 & 98.54 & 99.56 & 99.93 & 97.62 & 96.81 & 93.02 & 98.41 & 90.48 & 98.44 \\  & ENT & 1.826 & 1.618 & 0.475 & 0.080 & 0.967 & 1.19 & 79.04 & 63.78 & 2.389 & 1.716 \\ \hline
0.1 & ECE & 0.372 & 0.334 & 0.311 & 0.174 & 0.4969 & 0.1812 & 0.341 & 0.423 & 0.372 & 0.334 \\  & ACC & 30.25 & 32.14 & 62.80 & 81.20 & 37.50 & 46.80 & 54.47 & 49.32 & 30.25 & 32.14 \\  & SIM & 95.12 & 99.19 & 99.80 & 99.98 & 97.24 & 98.68 & 97.00 & 99.32 & 95.12 & 99.19 \\  & ENT & 1.145 & 0.84 & 0.204 & 0.014 & 0.286 & 0.283 & 0.373 & 0.261 & 1.1458 & 0.840 \\ \hline \end{tabular}
\end{table}
Table 5: Results of temperature scaling for LLaMA2-13B. The darker blue shade highlights better performing prompting technique.

 



## Appendix D Reliability Plots for LLaMA2-70B

Figure 16: Reliability plots for all the datasets using CoT and PAL prompting for the model LLaMA2-70B



## Appendix E Reliability Plots for gpt-3.5-turbo

Figure 8: Reliability plots for all the datasets using CoT and PaL prompting for the model gpt-3.5-turbo