Soft Self-Consistency Improves Language Model Agents (2024)

Abstract: Generations from large language models (LLMs) can be improved by sampling and scoring multiple solutions to select a final answer. Current “sample and select” methods such as self-consistency (SC; Wang et al., 2023) rely on majority voting to score answers. However, when tasks have many distinct and valid answers, selection by voting requires a large number of samples. This makes SC prohibitively expensive for interactive tasks that involve generating multiple actions (answers) sequentially. After establishing that majority voting fails to provide consistent gains on such tasks, we demonstrate how to increase success rates by softening the scoring criterion. We introduce Soft Self-Consistency (SOFT-SC), which replaces SC’s discontinuous scoring with a continuous score computed from model likelihoods, allowing for selection even when actions are sparsely distributed. SOFT-SC improves both performance and efficiency on long-horizon interactive tasks, requiring half as many samples as SC for comparable or better performance. For a fixed number of samples, SOFT-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping (WebShop), and a 4.7% increase for an interactive household game (ALFWorld). Finally, we show that SOFT-SC can be applied to both open-source and black-box models.1

heading: 1 Introduction
text: The performance of large language models (LLMs) can be greatly improved by generating multiple samples and scoring their answers before making a final selection. One popular and effective “sample and select” approach is Self-Consistency (SC; Wang et al., 2023), which leverages chain-ofthought prompting (Wei et al., 2022) to generate
*Equal Contribution 1Our code is publicly available at: https://github.com/
HanNight/soft_self_consistency.
multiple solutions for each input query and then determines the final answer via a majority vote. While SC has demonstrated consistent benefits on question-answering datasets, we find it provides minimal gains in several interactive settings where LLMs act as agents to generate a sequence of actions. SC’s selection mechanism relies on exact match in order to tally votes, i.e., it scores answers based on their frequency. However, in interactive domains, multiple distinct and valid answers – in this case, actions – can be generated at each step. This diminishes the effectiveness of SC over actions because the likelihood of generating identical actions decreases as the number of plausible options grows. For instance, a model tasked with predicting bash commands based on user queries has a very large action space (all bash commands) and could generate semantically equivalent commands that differ in their surface form (e.g., ls -ltr vs ls -trl).2 Therefore, deriving a signal from voting in LLM-agent domains would require sampling a large number of actions at each step throughout a lengthy trajectory, reducing efficiency and making SC prohibitively expensive (cf. Fig. 1).
We hypothesize that relaxing the strict scoring criterion from votes tallied by exact match to a continuous score will address the shortcomings of SC in two ways: (i) improving task performance in sparse action spaces; and (ii) increasing sample efficiency, i.e., higher success rates with fewer samples. We propose Soft Self-Consistency (SOFT-SC), a continuous relaxation of exact-match sample and select methods. Unlike match-based voting, SOFTSC handles cases without a unique majority answer. Crucially, for a white-box model, SOFT-SC incurs no additional cost and requires no external tests or metrics, as the probabilities used are already produced. Finally, we show that SOFT-SC can be used
2For Bash program prediction with five samples, SC fails to produce a single majority action 86% of the time.
287
to rescore black-box models’ outputs and can be integrated into an efficient variant of SC.
We test SOFT-SC on three diverse interactive domains: Bash (Yang et al., 2023), WebShop (Yao et al., 2022), and ALFWorld (Shridhar et al., 2021). Summary of Key Findings: 1. We demonstrate that SOFT-SC outperforms SC
with the same number of samples, e.g., by up to 6.6% on WebShop using CodeLlama-34B. 2. SOFT-SC exhibits better sample efficiency i.e., produces better performance than SC with fewer samples (cf. Fig. 1). 3. SOFT-SC scales better with model size than SC, increasing performance by 8.8% on Bash as model size increases from 7B to 70B, as opposed to only 5.8% improvement by SC. 4. SOFT-SC can be combined with smaller LMs to score generations from black-box models. We observe that SOFT-SC outperforms SC on closed-source models such as GPT-4 (OpenAI, 2023) by up to 4% on WebShop.

heading: 2 Methodology

heading: 2.1 Soft Self-Consistency (SOFT-SC)
text: Following Wang et al. (2023), for a given input x containing the task description, we generate k solutions using temperature-based sampling (Ackley et al., 1985; Ficler and Goldberg, 2017). To perform selection, we score the action yi resulting from each solution using the aggregated probability of the action’s tokens. For an action y composed of tokens y1, . . . , yn, we define score(y) = f ( {PLM(yi|y<i,x) ∀i ∈ [1, n]} ) where f ∈ {min,mean, product}. We choose the aggregation method based on dev set performance. We use mean probability for Bash and ALFWorld and min probability for Webshop. We then choose an action ŷ with the highest score, i.e., ŷ = argmaxkj=1 score(yj). Further details and
results for f options are provided in Appendix A.6.

heading: 2.2 Adaptive Soft Self-Consistency
text: To improve efficiency, Aggarwal et al. (2023) introduce adaptive-consistency, which reduces the number of samples (k) by approximating the final vote tally per example via sampling discrete vote distributions from a prior and stopping when the samples converge. Instead of sampling from discrete distributions, we choose k by aggregating likelihood scores until a score threshold τ is reached. Following Stengel-Eskin and Van Durme (2023b), we use the minimum probability for comparing with the threshold. We sample one action at a time, stopping when ∑k j=1 min |yj | i=1PLM(yi|y<i,x) ≥ τ , where τ is chosen on the dev set (cf. Appendix A.8).

heading: 2.3 Datasets
text: We test on three representative English LLM agent datasets; further details can be found in Appendices A.3 to A.5.
Bash. We use Yang et al. (2023)’s bash data, which consists of 200 user queries or instructions that can be completed via bash actions. We split these into 50 dev and 150 test. The agent’s performance is measured via success rate. Bash represents a domain with a large action space, as the space of possible bash commands is very large, and many of the queries involve stringing multiple functionalities together into a complex command.
WebShop. WebShop (Yao et al., 2022) is a simulated online shopping website environment. Success is measured both by a score ∈ [0, 1] reflecting how well the purchased product matches the user’s criteria; the success rate is the rate of perfect scores. Following Zhou et al. (2023), we report performance on a subset of 50 user queries. WebShop also has a large action space, as there are 1.18 million real-world products to select from.
ALFWorld. ALFWorld (Shridhar et al., 2021) is a text-game adaption (Côté et al., 2019) of the embodied ALFRED benchmark (Shridhar et al., 2020) in which an agent performs household chores (e.g., cleaning a mug) via a series of low-level actions. We evaluate on 134 unseen tasks and report the overall success rate. ALFWorld requires agents to generate long action sequences, involving thousands of valid actions at each step for some tasks.
Metrics. All these interactive tasks provide a goal and associated environments to execute the LLM-generated actions to accomplish said goal. After executing each action, the environment returns the observation and reward. The observation is a natural language description of the state of the system after executing the action, and the reward indicates if the goal was successfully achieved. The reward can be used to obtain a success rate, the percentage of examples with the maximum reward possible. Further details on the rewards for each domain can be found in Appendices A.3 to A.5.

heading: 2.4 Baselines
text: We compare SOFT-SC against the following:
Greedy Decoding. We sample a single solution with greedy decoding on all datasets; all prompts are given in Appendix C. This is equivalent to both SC and SOFT-SC when k = 1, as no selection is needed for a single sample.
Self-Consistency (SC). We use self-consistency as described by Wang et al. (2023), with majority voting as the selection criterion. We tally votes towards each response using exact match.
Adaptive-Consistency (AC). As described in Sec. 2.2, Aggarwal et al. (2023) introduce an adaptive version of SC that improves efficiency by adap-
tively reducing the number of samples. We implement their Beta estimator for all of our settings. Further details can be found in Appendix A.8.

heading: 3 Results and Discussion
text: Unless mentioned otherwise, we report average performance on 3 random seeds for each test set.
For the same number of samples k, SOFT-SC outperforms SC. In Table 1, we compare SOFTSC against the baselines on all datasets using CodeLlama-34B on the test sets. While both SC and SOFT-SC boost performance over the greedy decoding baseline, we find SOFT-SC results in a larger margin of improvement, 8.6% on WebShop (SC only yields 2%). For the same number of samples (k=10), SOFT-SC outperforms SC by 1.3%, 6.6%, and 4.7% (success rate) on Bash, WebShop, and ALFWorld respectively. Comparing the adaptive version of SOFT-SC with Aggarwal et al. (2023), our likelihood-based scores not only improve efficiency by generally using fewer samples, but also outperforms AC, e.g., by 4% on WebShop and 3.1% on ALFWorld.
SOFT-SC exhibits better scaling with k. In Table 1, even with k=5, SOFT-SC can outperform SC with k=10, e.g., with 2.2% improvement on ALFWorld. In Fig. 1, we compare this trend across more values of k, showing the scaling of SOFTSC and SC with an increasing k. We observe that SC provides minimal gains even as k increases, e.g., on Bash increasing k from 5 to 20 only yields 1% point improvement in success rate. On the other hand, SOFT-SC consistently improves success rates with ∼3% points improvement as k goes from 5 to 20. While SC does improve the success rate of Mistral-7B on ALFWorld with increasing k, SOFT-SC yields greater performance gains us-
ing fewer samples, e.g., SOFT-SC with k = 5 is comparable to SC with k=10.
SOFT-SC effectively scales with model size. As we scale up the size of the LM, we find that SOFTSC continues to provide improvements over SC. Fig. 2 shows the scaling trends for CodeLlama models ranging from 7B to 70B parameters on Bash and WebShop with a fixed k=10. For each LM, SOFTSC always outperforms SC. Furthermore, SOFTSC often allows smaller LMs to outperform larger members of the same model class, e.g., CodeLlama13B with SOFT-SC outperforms CodeLlama-34B with SC. This points to additional efficiency gains from SOFT-SC, as it can allow smaller models to replace larger ones.
SOFT-SC improves black-box models more than SC. SOFT-SC requires access to token probabilities to score actions. However, the most performant LLMs are typically black-box models, often with limited or no access to logits (OpenAI, 2023; Pichai, 2023; Anthropic, 2023). In Fig. 3, we study whether (smaller) open-source LMs can be used to score generations from GPT-3.5 and GPT-4. Here, we observe that SOFT-SC offers improvements over SC for a given black-box model, e.g., 4% for GPT-4 on WebShop and 1.8% on Bash when SOFT-SC uses the same number of generations from the black-box models as SC. Furthermore, even though Soft-SC requires 2 model calls (one to the black-box model and one to a smaller open-source model), SOFT-SC with k = 5 (total 10 calls) outperforms SC with k = 15 (total 15 calls to the black-box LLM), which shows that our method is significantly more efficient and effective since it can achieve better performance with fewer calls. Note that half of the calls for SOFT-SC are to a 7B model, likely making them much less expensive than calls to the black-box model. Calibration is not required for strong SOFT-SC performance. Given that SOFT-SC selects op-
tions using scores based on token probabilities, we investigate whether a model has to be wellcalibrated for SOFT-SC to work. We compute the correlations between two standard calibration metrics – ECE (Naeini et al., 2015) and AUROC – and absolute SOFT-SC performance for CodeLlama34B across seeds and values of k on WebShop and Bash test sets. The full plot is shown in Appendix B. We find a moderate negative correlation with AUROC (r=−0.55) on Bash and no significant correlation on WebShop); there is no significant correlation for ECE. In other words, having a well-calibrated model is not a prerequisite for SOFT-SC. This may be because calibration metrics do not measure ranking performance, which is central to our approach.
Logit-based score outperforms verbalized confidence score. Recent work has explored prompting language models to express uncertainty or confidence score in human language (Lin et al., 2022; Tian et al., 2023; Xiong et al., 2024). We study whether verbalized confidence scores can be used for selection instead of logit-based scores. We follow Lin et al. (2022) in prompting models to generate verbalized scores, which we then use for selection. As shown in Table 2, verbalized scores perform poorly when used in place of logit-based scores on Bash: Soft-SC with logits outperforms the verbalized method by 2.4% with k = 10.

heading: 4 Related Work
text: Sample and Select Methods for LLMs. Ensembling via voting over or aggregating outputs (Breiman, 1996; Freund and Schapire, 1997) can
improve a classifier’s performance. Wang et al. (2023) apply this paradigm to improve LLMs on reasoning tasks, introducing self-consistency (SC). We find that the majority voting used in SC is not suited for LLM-agent domains because the LLM’s generations may not exactly match when the action space is large. Chen et al. (2023b) generalize SC by prompting the LLM to determine consistency. However, LLMs still struggle to determine consistency in interactive domains where the task is partially observable (Ruan et al., 2023). In contrast to SOFT-SC, past work examining re-ranking strategies in code generation (Chen et al., 2022; Li and Xie, 2024) or reasoning (Golovneva et al., 2023; Prasad et al., 2023b) rely on external test cases or model-based metrics to score responses.
LLM-Agents. LLMs have proven to be effective agents across a diverse array of multi-step tasks, e.g., mathematical reasoning (Wei et al., 2022), tool-usage (Schick et al., 2023; Qin et al., 2023), robotic navigation (Ahn et al., 2022; Singh et al., 2023), and code-generation (Yang et al., 2023). Standard LLM-agent solutions employ chain of thought prompting (Wei et al., 2022) interleaved with permissible actions within an environment (Yao et al., 2023b). Several follow-up works improve upon this pipeline by building feedback over multiple trials (Shinn et al., 2023), decomposing tasks (Prasad et al., 2023a), or searching over trajectories (Yao et al., 2023a). SOFT-SC is complementary to these approaches, which can be seen as improvements to CoT for a single generation. Note that our work focuses on a single LLM agent (Andreas, 2022) interacting with an external environment to accomplish tasks; this single agent is compatible with other lines of work on discussion among multiple LLM agents (Du et al., 2023; Chen et al., 2023a).

heading: 5 Conclusion
text: After establishing the shortcomings of standard voting-based SC in interactive tasks, we introduced SOFT-SC, which relaxes the exact-match scoring function used by SC to a continuous score. On three commonly used interactive benchmarks, we showed that SOFT-SC results in improved performance and increased efficiency. We also show that SOFT-SC is compatible with both white-box and black-box models and that it can be integrated into a more efficient adaptive variant of self-consistency. Finally, we find that a well-calibrated model is not
required for SOFT-SC to work well, and that logits outperform verbalized confidence scores.

heading: 6 Limitations and Broader Impacts
text: Limitations. In Sec. 1, we pointed out that excessive diversity can lead to failures for SC, as no majority will emerge. However, both SC and SOFT-SC rely on some amount of output diversity: if the model generates k identical samples, then the output will be no better than generating one. One major motivation for SOFT-SC is efficiency; SOFT-SC substantially improves performance and is able to do so with fewer samples than SC, but it still requires multiple samples from an LLM. Thus, like all sample and select methods, SOFT-SC has a greater cost than greedy decoding. In Sec. 3, we demonstrate that SOFT-SC can be used to rerank outputs from other models that do not consistently provide logits. While SOFTSC shows major improvements in reranking the outputs of black-box models, it could be applied directly without a smaller scoring model if the generation model’s underlying logits (which exist by design) were made accessible to users.
Broader Impacts. Large language models have the potential for negative applications and malicious use (Weidinger et al., 2021; Bommasani et al., 2021). Our work improves LLM performance, meaning it could also be negatively applied. As our work is applied to LLMs operating as agents, it shares the inherent risk of all LLM agent work, namely that the LLM agent could potentially make mistakes and that its actions could lead to negative outcomes for the user. Overall, we believe this risk is mitigated by our use of simulated benchmarks (i.e., no agent we evaluate or develop can affect the world) and by the fact that our work improves agent accuracy, making adverse outcomes less likely.

heading: Acknowledgements
text: We thank Justin Chen and Swarnadeep Saha for their valuable help and feedback on the paper. This work was supported by NSF-AI Engage Institute DRL-2112635, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031, and the Accelerate Foundation Models Research program. The views contained in this article are those of the authors and not of the funding agencies.

heading: A Method and Dataset Details

heading: A.1 Hyperparameters
text: We select the threshold τ on the dev set for both Adaptive-Consistency baseline and Adaptive SOFTSC. For Adaptive-Consistency baseline, we set the threshold τ of 0.8, 0.85, and 0.8 for Bash, WebShop, and ALFWorld respectively. For Adaptive SOFT-SC, we set the threshold τ to 0.95, 3.0, and 3.5 for Bash, WebShop, and ALFWorld respectively. Because Adaptive SOFT-SC accumulates minimum probabilities over k samples for comparing with the threshold, the threshold may be ≥ 1.
For greedy decoding, we use a temperature of 0.7 for all datasets. In case of sampling k > 1 outputs from the model, we set the temperature of opensource models to 0.7 for Bash, 0.9 for WebShop, and 0.9 for ALFWorld, with top-p value of 0.9 and top-k value of 40, and with max_tokens set to 100. For obtaining generations from the OpenAI API, we use a temperature of 0.7 for Bash, 0.9 for WebShop and ALFWorld and top-p value of 1 for all datasets.

heading: A.2 Model Checkpoints and Licenses
text: Webshop, Bash, and ALFWorld all have MIT licenses. CodeLlama is released under a custom permissive license available here: https://github.com/facebookresearch/ llama/blob/main/LICENSE. Mistral uses an Apache License 2.0. For CodeLlama, we used the CodeLlama-*b-Instruct checkpoints. For Mistral, we used the Mistral-7B-Instruct-v0.2
checkpoint. All open-source models were accessed via Huggingface Transformers (Wolf et al., 2019). For OpenAI models, we used the gpt-3.5-turbo-0613 and gpt-4 checkpoints. All models were run for inference only with int-8 quantization on Nvidia 40GB A100 GPUs. We will release our code under an MIT license.

heading: A.3 Bash
text: Yang et al. (2023) propose an interactive benchmark for evaluating LMs on a bash coding task, created by bootstrapping queries from NLP2Bash benchmark (Lin et al., 2018). The dataset has 200 user queries or instructions that can be completed via bash actions, which we split into 50 dev and 150 test. After each action is executed, the agent observes the corresponding output from the file system. The agent’s performance is measured via success rate, which is determined by a reward function based on modifications to the file system with respect to a gold command as well the latest execution output – a success means the reward is 1.0. For example, given a query "find files in the /workspace directory and sub-directories, that changed within last hour", the agent generates a corresponding command find /workspace -cmin -60.
Setup. We focus on the single-turn setting instead of the multi-turn setting because we find the observation (i.e., the execution output of the action) from the Bash environment and the oracle reward rarely helps the agent generate correct commands. In our preliminary experiments, we observed that generating multiple commands using temperature-based sampling under the single-turn setting resulted in a success rate comparable to or even better than the multi-turn setting. Furthermore, in real-world scenarios, it is impossible to obtain oracle rewards to determine whether the generated commands are correct. Therefore, we prompt the LLM with a simple description of the task setting to sample k commands that would address the query. The final command selected by different methods is executed in the InterCode Bash environment and the response is scored to get the success rate.
Metric. After submitting the generated action, the environment returns a reward r ∈ [0, 1]. The reward function takes into account the differences in the file system resulting from executing the predicted command and the file system resulting from executing the gold command, as well as the latest
execution output. The Success Rate (SR) metric is defined as the proportion of tasks where r = 1.

heading: A.4 WebShop
text: WebShop (Yao et al., 2022) is a simulated online shopping website environment with 1.18 million real-world products. The underlying task requires an agent to navigate a simulation of a shopping website via a series of commands and buy a suitable product as per the user’s instruction (e.g., 3oz bottle of natural citrus deodorant for sensitive skin under $30). At the end of the trajectory, the environment returns a numeric score ∈ [0, 1] reflecting the degree to which the bought product matches the input criteria. Performance is measured based on the score as well as the success rate (i.e., a perfect score of 1). WebShop also has a large action space, as there are millions of products to select from. We use 30 user queries not in the test set to finalize our prompts and thresholds used for adaptive consistency as well as adaptive SOFT-SC.
Setup. Following Prasad et al. (2023a), we factorize the underlying agent into two modules: (i) selecting a suitable product, and (ii) buying a selected product. This simulates a “cart” functionality in online shopping. Given a user query, the agent first employs the search functionality and picks a few relevant products from the search page. It then explores the corresponding product page, matches its features, and determines if it can be added to the cart. We prompt the LLM to generate k such trajectories, potentially adding up to k products to the cart. In the end, we select a product by majority vote over product IDs and use a separate prompt to get the agent to buy the product while selecting relevant product options such as color, size, etc. The corresponding prompts are shown in Appendix C.
Note that due to the discrete and discontinuous nature of exact match (Schaeffer et al., 2023), SC can only perform selection over products. Given a description, SC navigates through the environment and selects multiple product pages, indexed by their IDs; these IDs can be aggregated via voting. However, within each product page, there are numerous follow-up options that must be selected, and which cannot be voted on as their selection happens across multi-step trajectories. Once a majority product is selected, SC uses a greedy action trajectory based on ReAct (Yao et al., 2023b) to specify the options for a selected product; this often results in suboptimal products being bought, as SC
often picks the default option. In contrast, the scoring criterion in SOFT-SC allows us to score and select from trajectories to first select products as well as to specify their options and buy them, generating and scoring k trajectories overall. Thus, SOFT-SC accounts for diversity in each stage and yields higher performance. For example, for the user query “natural looking long clip in extensions under $40” SC tallies votes for products IDs the cart after the product selecting stage: [B09QQLDJ93, B093BKWHFK, B09QQLDJ93], picking the B09QQLDJ93 as it forms a majority. It then uses a greedy ReAct trajectory to select the final options (e.g., the color) and to buy the item. SOFT-SC, on the other hand, can differentiate between action trajectories sampled for buying the same product ID, allowing it to distinguish between a final selection that has the default color “pink” and the correct product that uses the color “brown” – resulting in different scores from the environment.
Metric. When the LLM agent generates a buy action at the end of the trajectory, the environment returns a reward r ∈ [0, 1] reflecting the degree to which the bought product matches the input criteria. The Success Rate metric is defined as the portion of tasks where r = 1. The Score metric is defined as (100× avg. reward), which captures the average reward obtained across different task trajectories.

heading: A.5 ALFWorld
text: ALFWorld (Shridhar et al., 2021) is a text-game adaption (Côté et al., 2019) of the embodied ALFRED benchmark (Shridhar et al., 2020). The underlying task requires the agent to perform basic household chores such as finding a mug, cleaning it, and putting it on a countertop via a series of low-level actions (e.g., “go to sink”). After each action, the environment provides textual feedback (e.g., the contents of the cabinet after it is opened). We evaluate on 134 unseen tasks spanning 6 task types and report the overall success rate. In Fig. 1, due to computational requirements of using a larger number of samples, we report performance on a subset of the test split consisting of a total of 30 tasks, picking 5 from each task type. For the dev set, we use a disjoint set of 12 tasks from the ‘valid seen’ split of ALFWorld. This is only used to select the scoring criteria, e.g., mean, min, or product, and the thresholds for the adaptive variants.
Setup. Unlike WebShop, tasks in ALFWorld cannot be decomposed uniformly such that each sub-
task is handled by an independent agent without significant planning and communication overhead (Prasad et al., 2023a). For instance, the subtasks involved in “putting a clean mug on a countertop” vary considerably from the sub-tasks involved in “examining a spray-bottle under a desklamp”. Therefore, in ALFWorld, at each step, we sample k actions, and for SC perform majority voting over these k actions. Note that both SOFT-SC and SC only score actions, not thoughts or comments generated by the agent to aid in problem-solving. We continue sampling responses until a valid action is reached, skipping “thought” actions (i.e., generations starting with “Think:”) as well as comments. We only allow the selection of actions, ignoring the reasoning generated before the action. Note that both SC and SOFT-SC are more computationally demanding in the case of ALFWorld, since we perform selection over actions at each step, as compared to WebShop, where selection is performed once at the end of the selection phase over products. Following Yao et al. (2023b), the prompt to the LLM includes one in-context trajectory corresponding to a query from the same task type as the test instance.
Metric. After each action generated by the LLM agent, the environment provides textual feedback (e.g., the contents of the cabinet after it is opened). The feedback “You won!” in addition to reward r = 1 indicates that the agent has completed the task successfully. The Success Rate metric is the percentage of tasks where the agent succeeds.

heading: A.6 Aggregation Methods
text: For a given input x containing the task description and a corresponding sampled action y composed of tokens y1, · · · , yn, we can compute score(y) using the following probability aggregation methods: • Mean: score(y) = 1n n∑
i=1 PLM(yi|y<i,x)
• Min: score(y) = min 1≤i≤n PLM(yi|y<i,x) • Length-Normalized Product: score(y) = exp ( 1 n ∑n i=1 logPLM(yi|y<i,x) ) .
For Bash and ALFWorld, we perform scoring and selection at the action level, where the mean probability serves as an effective measure of the overall confidence in an action being the correct response to a given query. WebShop involves trajectory-level evaluations, where the correctness of a sequence of actions (a trajectory) towards accomplishing a task is assessed. In the case of WebShop, the trajectory

heading: Method Bash WebShop ALFWorld
text: represents a sequence of actions to select a suitable product based on the user query by navigating through a series of webpages; this sequential nature makes min better-suited. We also demonstrate experimental results on dev set for all aggregation methods to validate our explanation in Table 3.

heading: A.7 Baselines
text: Greedy Decoding. We sample trajectories with greedy decoding on all datasets; prompts are given in Appendix C. For WebShop and ALFWorld, we follow a ReAct prompt format (Yao et al., 2023b) while for Bash we follow the standard format provided by Yang et al. (2023). This is equivalent to both SC or SOFT-SC when k = 1 (since with a single sample, there is no selection needed, making the selection strategy irrelevant).
Self-Consistency (SC). We use self-consistency as described by Wang et al. (2023), with majority voting as the selection criterion. We tally multiple votes towards a response only if the model generates the exact response multiple times.

heading: A.8 Adaptive SOFT-SC
text: To improve sample efficiency, Aggarwal et al. (2023) introduce adaptive-consistency (AC), which reduces the number of samples (k) needed for selection by approximating the final vote tally through sampling. Specifically, AC adds generations one at a time (i.e., it increments k starting from 1) and terminates when a stopping criterion is satisfied or the number of generations has reached the maximum allowed. The stopping criterion is based on samples from a discrete distribution over vote distributions, parameterized by the current vote counts; these samples represent likely future vote distributions given the current trends. If the samples have converged, then further generations are unnecessary. For example, if 5/10 samples have been generated and 4 are identical, then the probability that the next 5 will change the majority vote is vanishingly small, meaning that generating further solutions is
wasteful. On the other hand, if there is no clear majority winner after 5 samples, further solutions would be needed.
We can apply a similar methodology to SOFTSC. However, instead of estimating k by sampling from a discrete vote distribution, we estimate the stopping criterion for sampling by aggregating likelihood scores until a sufficient score threshold τ is reached. While we use average probability across tokens for selection, we find that this score is poorly calibrated. Following Stengel-Eskin and Van Durme (2023a), who found minimum token probabilities to be better calibrated, we use the minimum probability for comparing with the threshold. Therefore, we sample actions one-at-a-time and stop when the number of samples k is such that∑k
j=1 min |yj | i=1Pθ(yi|y<i,x) ≥ τ . The threshold τ
is a domain-specific hyperparameter that we select based on a dev set (discussed in Appendix A.1). Specifically, we set the threshold τ to 0.95, 3.0, and 3.5 for Bash, WebShop, and ALFWorld respectively. Note that in this case, the threshold can be > 1 as it represents a threshold on cumulative confidence values, rather a threshold on true probability distribution. This differs from adaptive-consistency, for which the threshold is over a normalized probability, i.e., it must be less than ≤ 1.

heading: B Calibration
text: Following past work (Kuhn et al., 2023; StengelEskin and Van Durme, 2023a), we use Expected Calibration Error (ECE) and Area Under the Receiver Operator Characteristic curve (AUROC) to check the calibration of scores used in SOFT-SC:
Expected Calibration Error (ECE) (Naeini et al., 2015) is used to quantify how well a model is calibrated. It computes the difference between the accuracy and confidence of the model, where accuracy is averaged across examples falling into
confidence bins. A well-calibrated model will have a low ECE, as it will have a smaller difference between the predicted rate of success (the average confidence) and the actual rate of success (the average accuracy) of a given set of predictions. While ECE is a standard metric, it suffers from sensitivity to the number of confidence bins used (Ding et al., 2020). To mitigate this, we use Stengel-Eskin and Van Durme (2023a)’s implementation of Ding et al. (2020)’s adaptive binning approach, which dynamically adjusts bin sizes to reduce bias in the confidence estimate.
Area Under the Receiver Operator Characteristic curve (AUROC) assesses the ability of the estimated confidence to distinguish correct and incorrect samples. AUROC measures the area under the curve formed by comparing the true positive rate to the false positive rate. If a model is wellcalibrated, then there is some threshold for which we can separate predictions into correct predictions (above the threshold) and incorrect ones (below the threshold). In general, as we adjust the threshold there will be a tradeoff between true positives and false positives (e.g., a low threshold will result in a large number of false positives, while a high threshold will reduce the number of true positives). A higher AUROC score is better, with a perfect classifier achieving an AUROC of 1 while a random estimator would score 0.5.
Figure 4 illustrates Pearson correlations between two standard calibration metrics – ECE and AUROC – with SOFT-SC performance. For Bash, we find no significant correlation with ECE and a moderate negative correlation with AUROC. For Webshop, neither metric is significantly correlated. Therefore, we conclude that a well-calibrated model is not a prerequisite for SOFT-SC. This may be because calibration metrics do not measure ranking performance, which is central to our approach.

heading: C Prompts
text: We provide the prompts along with in-context examples supplied to the LLM for sampling trajectories for Bash and WebShop in Fig. 5, Fig. 6, and Fig. 7. As mentioned in Appendix A.5, for ALFWorld, we use the prompts and in-context examples provided in Yao et al. (2023b).

heading: Future Work
text: The stopping criterion is based on samples from a discrete distribution over vote distributions, parameterized by the current vote counts; these samples represent likely future vote distributions given the current trends. If the samples have converged, then further generations are unnecessary. For example, if 5/10 samples have been generated and 4 are identical, then the probability that the next 5 will change the majority vote is vanishingly small, meaning that generating further solutions is
wasteful. On the other hand, if there is no clear majority winner after 5 samples, further solutions would be needed. We can apply a similar methodology to SOFTSC. However, instead of estimating k by sampling from a discrete vote distribution, we estimate the stopping criterion for sampling by aggregating likelihood scores until a sufficient score threshold τ is reached. While we use average probability across tokens for selection, we find that this score is poorly calibrated. Following Stengel-Eskin and Van Durme (2023a), who found minimum token probabilities to be better calibrated, we use the minimum probability for comparing with the threshold. Therefore, we sample actions one-at-a-time and stop when the number of samples k is such that∑k
j=1 min |yj | i=1Pθ(yi|y<i,x) ≥ τ. The threshold τ
is a domain-specific hyperparameter that we select based on a dev set (discussed in Appendix A. 1). Specifically, we set the threshold τ to 0. 95, 3. 0, and 3. 5 for Bash, WebShop, and ALFWorld respectively. Note that in this case, the threshold can be > 1 as it represents a threshold on cumulative confidence values, rather a threshold on true probability distribution. This differs from adaptive-consistency, for which the threshold is over a normalized probability, i. e. , it must be less than ≤ 1