Venue: NAACL 2025

# Waste Not, Want Not\({}^{*}\);

Recycled Gumbel Noise Improves Consistency in Natural Language Generation

Damien de Mijolla, Hannan Saddiq, Kim Moore

Faculty Science Ltd

damien.de-mijolla@faculty.ai

###### Abstract

Consistency in the output of language models is critical for their reliability and practical utility. Due to their training objective, language models learn to model the full space of possible continuations, leading to outputs that can vary significantly in style and content, even for similar or repeated inputs. To address this, we propose a novel decoding algorithm that enhances response consistency across different prompts with no degradation in response quality. By incorporating a latent variable into the next-token sampling process based on the Gumbel reparametrisation trick, our method outperforms standard sampling by up to 10% across semantic and stylistic consistency benchmarks. Additionally, our approach integrates seamlessly with existing sampling methods with negligible computational overhead, providing a practical solution for improving the reliability of language model outputs.

+
Footnote †: _(Volume 1: Long Papers)_, pages 5662–5686

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

In recent years, state-of-the-art language models (LMs) have demonstrated remarkable performance across a wide range of benchmarks, often rivaling human capabilities in tasks such as translation, summarization, and question-answering Brown et al. (2020); Dubey et al. (2024). However, these advancements have not always translated into practical usefulness for real-world applications, where reliability and consistency are crucial Kaddour et al. (2023).

One of the primary challenges is the inconsistency of these models' responses, which can vary significantly in style, factual accuracy, and tone Bommasani et al. (2021). This inconsistency, a byproduct of the probabilistic nature of language model training, can lead to a range of issues, including reduced trust in outputs, exposure to more diverse failure modes, and less reliable behaviour Ye et al. (2023).

Although traditional methods (e.g the use of random seeds) can be applied to introduce determinism in natural language generation, ensuring identical responses for identical inputs, they do not help ensure similar responses when inputs are similar. In practice, due to the richness of language, input queries can often be rewordded in many ways while retaining their meaning. To achieve greater consistency, it is desirable for the model to generate similar responses across all these variations Ribeiro et al. (2018).

In this paper, we investigate whether next-token sampling procedures can be modified to enhance consistency across different prompts. Our main contributions include:

1. We propose a simple, computationally inexpensive sampling procedure that (i) can be applied to any model, (ii) does not require any additional training, and (iii) has negligible impact on inference costs. We also ensure that the probability of any individual response is unchanged and so does not compromise response quality.
2. We also leverage an auxiliary approach to further improve consistency between model responses using distributional ensembling, which can be applied in conjunction with our aforementioned sampling procedure.
3. We investigate the performance of our approach against standard sampling across a number of benchmarks covering semantic and stylistic similarity, across a number of different models.

In particular, we highlight that our combined sampler outperforms standard sampling across all benchmark suites and models tested, by up to 10% in some cases.



Related works

Decoding approachesLanguage model decoding strategies can be broadly classified into two categories: optimization and sampling-based approaches Ji et al. (2024). Optimization-based approaches, such as greedy decoding and beam search Lowerre (1976); Jurafsky and Martin (2009), frame text generation as an optimization problem, searching for sequences that maximize a specific metric such as probability, whereas sampling-based approach incorporate stochasticity into the next-token selection process. Optimization-based approaches are typically perceived as yielding less engaging but more accurate responses and so are often favoured for closed-ended tasks expecting a fixed answer Holtzman et al. (2020). However, recent work has put into question the greater accuracy of their responses Renze and Guven (2024).

In contrast, sampling-based approaches are usually preferred for open-ended tasks, as they typically yield more engaging answers Basu et al. (2021); Ji et al. (2024). Our proposed method falls within this category. Many existing methods in the literature, such as nucleus sampling and mirostat Holtzman et al. (2020); Basu et al. (2021); Fan et al. (2018), aim to improve text generation quality by directly modifying the probability distribution from which tokens are sampled. We consider these methods, which directly alter the next-token distribution, as complementary to our approach, which maintains the next-token distribution and instead modifies the joint distribution over responses.

Our approach is methodologically most closely related to methods Vilnis et al. (2023); Kool et al. (2019) which also adjust the joint distribution of sampled responses. However, while these methods aim to maximize response diversity--an advantage when ensembling multiple responses as done in self-consistency voting Wang et al. (2023)--our approach is distinct in its focus on minimizing response diversity to achieve more consistent outputs.

Self-ConsistencyLanguage models lack robustness to prompt variations Huang et al. (2024); Elazar et al. (2021) and give contradictory responses in such cases, motivating the need for enhanced self-consistency. Self-consistency in language models has been studied from many different angles, but usually with a focus on factual rather than stylistic consistency. Prior work has proposed a number of fine-tuning approaches for increasing self-consistency, including fine-tuning approaches for increasing the ability of language models to respond consistently to paraphrases of questions Elazar et al. (2021); Yan et al. (2024), and approaches for correcting model contradictions using a factor graph over beliefs Mitchell et al. (2022).

Our approach is methodologically orthogonal to previous approaches for enhancing self-consistency. Previous work has relied on fine-tuning which not only is more cumbersome to implement but also modifies the raw next-token probabilities, potentially affecting responses in unforeseen ways or contributing to catastrophic forgetting.

Since our approach only modifies the joint distribution over responses without modifying the next-token probability distribution, it does not suffer from the same issues, and comes with principled guarantees around maintaining the model's original response style and quality. Additionally, it enhances _all_ aspects of self-consistency, not just factual consistency of responses.

## 3 Problem statement

Let \(X\) be a language model prompt composed of a sequence of tokens drawn from a vocabulary of size \(N_{v}\), and let \(\pi_{\theta}\) be a language model trained on the task of next-token-prediction. For the remainder of the paper we denote a forward pass through the language model by \(h_{t}=\pi_{\theta}(X,Y_{1:t-1})\) where \(h_{t}\in\Delta^{N_{v}-1}\) represents a probability distribution over the token vocabulary, with \(\Delta^{N_{v}-1}\) denoting the \((N_{v}-1)\)-dimensional probability simplex. The sequence \(Y_{1:T}\) represents the full response obtained by auto-regressively applying the language model with the next token at each step sampled from the categorical distribution parameterized by the model, \(Y_{t}\sim\text{Cat}(h_{t})\). In what follows, we use a subscript to represent position in a sequence, and a superscript to represent the token index. So for example, \(h_{t}^{i}\) represents the probability of sampling token \(i\) at position \(t\).

Suppose that \(U\) is a different prompt that is semantically similar to \(X\) for which we generate a response \(V=V_{1:M}\). Motivated by the inconsistency of LM responses, our goal is to modify the LM sampling procedure in a way that increases the similarity between responses \(Y\) and 

\(V\) according to some yet-to-be-specified notion of similarity. Furthermore, we focus on sampling approaches that modify the joint probability of responses \(p(Y,V)\) without affecting the marginal probability of individual responses, \(p(Y)\) and \(p(V)\), to guarantee that quality of the original responses is maintained.

## 4 Approach

Our proposed sampling approach, motivated in Figure 1, modifies the joint probability distribution over responses by introducing a latent variable \(g\) to the sampling process. Conditioning the generation of distinct responses on a common realisation of this latent variable introduces a statistical dependency between them. Generating responses with greater similarity can then be straightforwardly done by conditioning the generation of all responses on a common realisation of the latent variable, that is to say to sample \(Y\sim p(Y|X,g)\) and \(V\sim p(V|U,g)\).

To ensure the efficacy of the approach, we design the latent variable in such a way that conditioning responses on a common value of the latent variable makes responses as similar as possible. To ensure the preservation of the probability distribution parameterized by the language model, we sample the latent variable from a probability distribution \(g\sim p(g)\) such that marginalising over the latent variable recovers the original distribution over responses, \(\mathbb{E}_{g}[p(Y|g)]=p(Y)\).

To construct a latent variable with the above properties, we employ the reparametrization trick for categorical distributions. Introduced for normal distributions in Kingma and Welling (2014) and extended to categorical distributions in Maddison et al. (2014, 2017); Jang et al. (2017), the reparametrization trick is a procedure that refactors the sampling from a distribution into a deterministic function of the parameters and a draw from some independent noise with a fixed distribution. For a categorical distribution with parameters \(p^{1},...,p^{N_{v}}\), this can be cast as first drawing random noise \(g=(g^{1},...,g^{N_{v}})\) where each \(g^{i}\sim G(0,1)\) is independently drawn from the Gumbel distribution Gumbel (1954) and selecting a category \(k\) according to \(k=\arg\max_{i}(\log p^{i}+g^{i})\).

**Theorem 4.1**.: _Suppose we have two different categorical distributions parametrized by \(p^{1},...,p^{N_{v}}\) and \(q^{1},...,q^{N_{v}}\). Define a joint distribution over pairs of categories \((Y,V)\) by defining_

\[Y =\operatorname*{arg\,max}_{i}(\log p^{i}+g^{i}),\] \[V =\operatorname*{arg\,max}_{i}(\log q^{i}+g^{i}),\]

Figure 1: Motivating toy example highlighting the aim of our approach. Even when language models yield similar probability distributions over responses (top), responses sampled independently (bottom left) can be inconsistent or contradictory due to the inherent stochasticity of sampling. By generating responses in a correlated manner (bottom right) it is possible to alleviate inconsistencies across responses while still respecting the marginal probabilities of each response. In this paper we propose, Gumbel Consistent Sampling, an approach for increasing response consistency through drawing correlated responses, by conditioning all responses on a shared latent variable, that is robust to differences between probability distributions over responses.

 

_where \(g^{1},...,g^{N_{v}}\sim G(0,1)\) are independent. We have that_

\[P(Y=k,V=k)=\frac{p^{k}q^{k}}{p^{k}q^{k}+\sum_{i\neq k}\max\{p^{i}q^{k},q^{i}p^{k }\}}.\]

Theorem 4.1 (proved in Appendix A) shows that interpreting the Gumbel noise as a latent variable and conditioning sampling events on the same realisation of this latent variable increases the probability of selecting the same category with both distributions compared to sampling from each categorical distribution independently, with identical sampling outcomes in the limit where \(p\) and \(q\) become identical.

Since generating a response using a LM consists of successive draws from categorical distributions, the above idea can be applied to language modelling in order to increase the token overlap across distinct responses. Indeed, we can generate ahead of time a sequence of independent Gumbel latent vectors, \(g_{1:t}\), one for each position in the sequence up to the maximum sequence length, and sample each token using the Gumbel latent vector assigned to that position in the sequence when generating a response. That is to say, drawing \(Y_{t}\sim p(Y_{t}|h_{y,t},g_{t})\) and \(V_{t}\sim p(V_{t}|h_{v,t},g_{t})\), where here we denote by \(h_{y,t}\) and \(h_{v,t}\), the next-token probabilities obtained by running the language model on the context-up-to-now (i.e. \(h_{y,t}=\pi_{\theta}(X,Y_{1:t-1})\), \(h_{v,t}=\pi_{\theta}(U,V_{1:t-1})\) ). We refer to the above approach as **Gumbel Consistency Sampling, GCS**.

This sequential Gumbel sampling approach increases similarity of responses by increasing the rate at which identical tokens are generated at fixed positions in the sequence \(p(Y_{i}=k,V_{i}=k)\) but has the limitation of not increasing the co-occurrence across sequence positions \(p(Y_{j}=k,V_{i}=k)\). We expect that two similar responses are likely to contain some of the same tokens, but likely in different positions, so it would be advantageous for our final sampling approach to reflect this.

Introducing such an inter-position correlation in sampling outcomes across sequences is made challenging by the requirement of conditional independence between sampling steps. Indeed, to respect the LM's probability distribution, it is necessary for sequential sampling steps to be independent of each other, i.e. for \(p(Y_{t+1}|X,Y_{1:t})=p(Y_{t+1}|h_{t+1})=\text{Cat}(Y_{t+1};h_{t+1})\) which prevents the direct reuse of Gumbel samples across sequence positions.

**Theorem 4.2**.: _Consider a sequence of tokens \(Y_{1:T}\) generated auto-regressively according to the following update rule, where \(k:=\arg\max_{j}(g_{t}^{j}+\log h_{t}^{j})\), \(Q(\cdot)\) is the quantile function for the \(G(0,1)\) distribution and \(\pi_{\theta}(\cdot)\) is a language model:_

\[g_{1} \sim G(0,1)\] \[h_{t+1} =\pi_{\theta}(X,Y_{1:t})\] \[g_{t+1}^{k}\mid g_{t},h_{t} \sim G(0,1)\] \[g_{t+1}^{i}\mid g_{t},h_{t} =Q\left(\tfrac{Q^{-1}(g_{t}^{j})}{Q^{-1}(g_{t}^{k}+\log h_{t}^{k }-\log h_{t}^{j})}\right),\] _for \(i\neq k\)_

\[Y_{t+1}=\operatorname*{arg\,max}_{j}\left(\log h_{t+1}^{j}+g_{t+1}^{j}\right)\]

_With this update procedure, the probability distribution over a given token conditioned on preceding tokens is_

\[p(Y_{t+1}\mid X,Y_{1:t})=\text{Cat}(Y_{t+1};h_{t+1})\]

In Theorem 4.2 we introduce a procedure for recycling a Gumbel vector after applying the Gumbel reparameterization trick. We prove in Appendix B.1 that this procedure is functionally equivalent to independently sampling each token from the true model probability distribution. This means that repeated application of the Gumbel reparametrization trick with this recycling procedure yields sequences that are indistinguishable from those obtained by independent sampling at each step from the model's categorical distribution.

This property enables the generation of highly correlated responses while preserving adherence to the model's probability distribution. By sampling a single Gumbel vector and reusing it across all generated responses, each response remains faithful to the model's predicted probabilities while also being inherently correlated due to the shared Gumbel noise. Moreover, because the Gumbel noise remains highly similar before and after recycling, responses exhibit strong inter-position correlations across different sequence positions.

We explicitly present the overall procedure in Algorithm 1 as well as an illustrative Python implementation in Appendix I. We refer to this generation approach as **Gumbel Consistency Sampling with Recycling, (GCSwR)**. At each sequence position, the algorithm resamples a new 

Gumbel noise value for the position corresponding to the chosen token while recycling a rescaled version of the existing Gumbel values for all other positions. To prevent divergences among responses due to resampling, we precompute and store Gumbel noise resamplings for each token in the vocabulary. This allows the same noise values to be reused across different responses. In the algorithm, this process is managed using the counter variable \(c\).

The standard procedure for autoregressive token sampling, which is equivalent to independent sampling of a new Gumbel latent vector for every sequence position and every sequence, acts as a baseline for subsequent experiments, and is denoted as **Independent Sampling, (IS)**.

## 5 Ensembling semantically similar responses

A complementary approach to enhance consistency between responses given semantically similar prompts is to reduce the impact of semantically irrelevant prompt attributes on the next-token probability distributions, which can be achieved by increasing the similarity between the sampling distributions.

In our experiments, we explore sampling tokens from an ensembled probability distribution over semantically equivalent prompts as a means of minimising impact of semantically irrelevant prompt variations on responses. Specifically, we generate semantically equivalent variations of the user prompt by asking a separate LM (gpt-4o mini) to rephrase the prompt. We then run the target LM separately on all of the prompts, producing a set \(\{P_{i}\}\) of next-token probability distributions. We then sample from an ensembled distribution, ensembled using the following formula:

\[Q^{j}=\frac{1}{Z}\prod_{i=1}^{n}(P_{i}^{j})^{\frac{1}{n}} \tag{1}\]

where Z is the normalisation constant that ensures \(Q\) defines a valid probability distribution function:

\[Z=\sum_{j}\prod_{i=1}^{n}(P_{i}^{j})^{\frac{1}{n}}\]

This formula corresponds to selecting the categorical distribution that minimizes the average forward-KL divergence over all next-token probability distributions (see Appendix D). We found that direct averaging (which can equivalently be shown to minimize the reverse-KL distribution) tended to generate worse-quality responses due to at times sampling tokens that were only high-probability for a subset of question recordings.

Note that, contrary to our proposed Gumbel sampling approach, ensembling comes at a cost of additional inference-time compute and also modifies the language model probability distributions. We highlight that ensembling can be applied in conjunction with any of the three samplers discussed in section 4, and we investigate the performance of each sampler with and without ensembling in our experiments.

## 6 Experiments

In our experiments, we empirically demonstrate the utility and limitations of GCS and GCSwR. We begin by quantifying the utility of the procedure for enhancing semantic similarity of responses, and highlight a number of stylistic dimensions of text along which Gumbel sampling improves consistency. Details for reproducing experiments are shown in Appendix E.

### Semantic similarity

We start by quantifying the improvement in the semantic similarity between responses for semantically equivalent queries by using our Gumbel sampling variants (GCS and GCSwR). To measure semantic similarity, we use ES\({}_{\text{mistral.7b}}\), a specialised 

state-of-the-art model trained specifically on the task of semantic similarity (Wang et al., 2024).

We create semantically equivalent pairs of questions for evaluation by randomly sampling 300 questions from the Alpaca dataset (Taori et al., 2023) -- a popular human-preference dataset - and rephrasing them using gpt-4o mini. We then generate responses to the original and rephrased version of each question using Meta-Llama-3-8B-Instruct, Meta-Llama-3-8B, Mistral-7B-v0.1, Llama-2-7b-chat-hf (AI@Meta, 2024; Touvron et al., 2023; Jiang et al., 2023). In all cases we sample from the raw unmodified next-token probabilities predicted by the language models (i.e. temperature of 1) and for Gumbel sampling, we resample the Gumbel latent vector for each pair of questions such that responses are correlated within but not between pairs.

The aggregated results, shown in Table 1, demonstrate that the most performant sampling scheme tested (GCSwR with ensembling) significantly increases response similarity to semantically equivalent questions across all models considered, by more than 10% when compared to the baseline in some cases. We note more pronounced enhancements from Gumbel sampling for unaligned models like Mistral and Llama3 Base, which we hypothesise is caused by their lower base semantic similarity compared to their instruction fine-tuned counterparts.

The above trends appear to be consistent across different choices of semantic similarity metric which we show in Appendix H, where we reproduce results using the Jaccard similarity, a simple token overlap metric, and using all-mpnet-base-v2, the semantic similarity model recommended by the popular sentencetransformer repository (Reimers and Gurevych, 2019). In both cases, we find relative performances between approaches to be consistent with those quoted in the main paper body.

### Semantic similarity as a function of temperature

Next, we investigate how the effectiveness of GCSwR varies with sampling temperature. We compare the semantic similarity metric on the Alpaca dataset as a function of temperature in Figure 2 with IS as a baseline, without using ensembling in both cases. GCSwR improves the semantic consistency of responses across all temperatures, except temperature 0, where the model probabilities with and without GWSwR become identical due to the fully deterministic nature of model outputs at this temperature1 Example responses for Llama3 models at temperature 0.8 can be found in Appendix G.

Footnote 1: We note that responses can still differ under greedy decoding if several tokens are tied for maximum probability. In experiments this occurred a non-negligible amount of times due to the limited numerical precision of bfloat16.

It is also interesting to note that although GCSwR improves self-consistency at all non-zero temperatures, the highest self-consistency achieved is with greedy decoding (i.e. temperature 0) which is where both approaches behave identically. However, we caution that this result does not imply that greedy decoding will always be preferable to higher-temperature Gumbel sampling. Using greedy decoding is widely considered to decrease the quality of responses across a number of important dimensions and so model providers typically use non-zero default temperatures (Basu et al., 2021; Ji et al., 2024; Zhang et al., 2021). Gumbel sampling offers a way of increasing the consistency of responses without the negative side-effects associated with excessively lowering the sampling temperature. We also note that using Gumbel sampling is much more effective at increasing self-consistency of responses than decreasing temperature, with temperatures needing to be roughly halved in order to match the benefits of using Gumbel consistency sampling.

### Stylistic similarity

In this section, we study Gumbel consistency sampling's ability to enhance stylistic consistency across several distinct stylistic dimensions, evaluating GCSwR without ensembling using Mistral-7B-v0.1 (Jiang et al., 2023).

We conduct our experiments on two datasets: Code-Alpaca and Aleatoric-List. The Code-Alpaca dataset (Chaudhary, 2023) consists of coding-related questions, from which we select a subset of 20 random questions that are agnostic to programming languages. For this dataset, we assess stylistic consistency based on several factors: whether the response contains a code snippet, whether the response starts directly with the code snippet or begins with freeform text, whether the code snippet includes comments, and the programming language used in the response (such as Python, JavaScript, or C++).

 

The second dataset, Aleatoric-List, is a synthetic dataset we created containing 20 questions that ask for five different items fitting a specific category. An illustrative example question is "Give me the names of five capital cities in Europe." For this dataset, we evaluate stylistic consistency based on whether the answer is terse, whether it contains bullet points, and whether these bullet points are numerical.

To evaluate stylistic consistency along each dimension, we begin by generating 100 Gumbel latent vectors. Then, for each Gumbel vector, we generate a response to all questions in the dataset which we classify along each of the stylistic dimensions through prompting gpt-4o mini (with prompts shown in Appendix F). For each factor, we then define the stylistic consistency as the probability that responses to two randomly selected questions share the same label, denoted as \(p_{repeat}\). We then compare this probability with the equivalent probability when the responses are generated with our independent sampling baseline (IS).

Let \(Z\) be a Bernoulli random variable that denotes whether a randomly sampled response is labelled with a given stylistic dimension, \(p(Z=1)=p\). For IS, \(p_{repeat}=p^{2}+(1-p)^{2}\). However, for GCS and GCSwR, \(p_{repeat}=\mathbb{E}_{g}[p_{g}^{2}+(1-p_{g})^{2}]\) where \(p_{g}\) denotes the probability of a randomly sampled response generated using Gumbel latent

\begin{table}
\begin{tabular}{l l c c} \hline \hline
**Model** & **Sampler** & **Without Ensembling** & **With Ensembling** \\ \hline  & IS & 86.34\(\pm\)0.07 & 87.56\(\pm\)0.29 \\ Llama2 Chat & GCS & 88.28\(\pm\)0.10 & 90.26\(\pm\)0.27 \\  & GCSwR & **88.61\(\pm\)0.15** & **90.38\(\pm\)0.25** \\ \hline  & IS & 72.00\(\pm\)0.27 & 72.34\(\pm\)0.93 \\ Mistral & GCS & 78.55\(\pm\)0.22 & 81.17\(\pm\)0.77 \\  & GCSwR & **80.94\(\pm\)1.05** & **82.74\(\pm\)0.81** \\ \hline  & IS & 85.61\(\pm\)0.18 & 86.90\(\pm\)0.16 \\ Llama3 Instruct & GCS & 86.81\(\pm\)0.46 & 89.01\(\pm\)0.35 \\  & GCSwR & **87.37\(\pm\)0.27** & **89.68\(\pm\)0.08** \\ \hline  & IS & 71.23\(\pm\)0.41 & 71.46\(\pm\)0.70 \\ Llama3 Base & GCS & 76.68\(\pm\)0.80 & 78.71\(\pm\)0.82 \\  & GCSwR & **80.10\(\pm\)0.80** & **82.04\(\pm\)0.81** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Average semantic similarity results by sampler type with and without our ensembling approach as measured by E5\({}_{\text{mistral-7b}}\). Scores shown as mean\(\pm\)std.err with std.err obtained from 3 independent runs. Bold indicates highest scores for each model in both ensembling categories.

\begin{table}
\begin{tabular}{c r r r} \hline \hline
**Dataset** & **Stylistic Dimension** & \multicolumn{2}{c}{**Sampler**} \\ \cline{3-4}  & & **IS** & **GCSwR** \\ \hline
**Code-Alpaca** & Is Python & 0.67 & **0.73** \\  & Is JavaScript & 0.78 & **0.84** \\  & Is C++ & 0.92 & **0.94** \\  & Contains Code Snippet & 0.71 & **0.81** \\  & Answers Directly & 0.50 & **0.73** \\  & Contains Comments & 0.71 & **0.80** \\ \hline
**Aleatoric-List** & Does Not Use Bullets & 0.75 & **0.82** \\  & Uses Numerical Bullets & 0.82 & **0.87** \\  & Terseness & 0.50 & **0.64** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of Gumbel consistency sampling with recycling (GCSwR) vs. independent sampling (IS) on Stylistic Consistency 

vector \(g\) taking value \(Z=1\). These expressions follow directly from the conditional independence of responses generated with a common initial Gumbel latent vector \(g\) and generated independently, and additionally from marginalisation over initial Gumbel latent vectors \(g\).

Although the estimator \(\hat{p}=\frac{1}{n}\sum_{i=1}^{n}Z_{i}\) is an unbiased estimator of \(p\), \(\hat{p}^{2}+(1-\hat{p})^{2}\) yields a biased estimator of \(p^{2}+(1-p)^{2}\). To correct for this bias, we use the following estimator \(\frac{N}{N-1}(\hat{p}^{2}+(1-\hat{p})^{2})-\frac{1}{N-1}\) which we show in Appendix C to be unbiased.

We show, in Table 2, the results of this experiment, using Mistral-7B to generate responses. Across all stylistic dimensions considered, using GCSwR increases the frequency with which generated responses follow a common style. For many factors, the increase is significant (>10%), showing that Gumbel consistency sampling can have an appreciable impact on style consistency.

## 7 Conclusion

We have introduced Gumbel consistency sampling, a straightforward and computationally inexpensive sampling approach for increasing consistency amongst model responses. The method requires no additional fine-tuning, additional language model calls or apriori knowledge of what prompts will be used, and guarantees responses indistinguishable to those obtained using standard sampling at the level of individual responses. The approach enhances consistency by sampling responses in a correlated manner through the introduction of a latent variable, in a way that increases the token overlap across responses. In our experiments, we find that this approach is not only able to enhance semantic similarity between responses but also stylistic similarity. These results showcase how Gumbel consistency sampling offers a principled quick and easy way of enhancing language model consistency.

Future work could extend the Gumbel consistency sampling to imposing local rather than global correlation to responses. Currently, all responses are globally coupled due to dependence on the same global latent variable, which makes localised adjustments to model behaviour impossible. However, the framework could easily enable for latent variables to be varied locally depending on question specifics, which would enable finer-grain control of model behaviour and could increase the overall response diversity. Another,

Figure 2: Mean semantic consistency between responses to paraphrased questions as a function of temperature, comparing independent sampling (IS) against Gumbel Consistency Sampling with Recycling (GCSwR).

 

promising direction for extending the work could be to treat the Gumbel noise as a learnable task-specific parameter. Such an approach may be especially useful for building stronger model safeguards while preserving general utility.

## 8 Limitations

Increasing the consistency amongst a set of responses necessarily decreases their diversity making our proposed sampling approach unsuitable for use cases requiring high response diversity. In particular, using the proposed sampling approach, leads to fully deterministic sampling where responses will always be identical for identical input prompts.

More generally, use of the sampling approach is likely to lead to responses favoring specific topics and figures of speech over others. This arises due to the specific Gumbel noise value utilised during text generation encoding relative preferences between tokens and is not inherently a weakness of the approach. Indeed, analogously every members of the human population also exhibit their own individual preferences and mannerisms.

Finally, it is important to emphasize that while Gumbel consistency sampling enhances consistency amongst responses it does not guarantee it. Responses, generated using the approach, may still lack self-consistency making the approach on its own inadequate for use cases requiring perfect consistency.

## References

* Al@Meta (2024) Al@Meta. 2024. Llama 3 model card.
* Basu et al. (2021) Sourya Basu, Govardana Sachithanandam Ramachandran, Nitish Shirish Keskar, and Lav R. Varshney. 2021. Mirostat: a neural text decoding algorithm that directly controls perplexity. In _International Conference on Learning Representations_.
* Bommasani et al. (2020) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosseltur, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumboupva, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goed, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahl Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Persehe Khanil, O. Khattab, Pang Wei Koh, Mark S. Krasnajv Krishna, Rohith Kutditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Milroschan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasvaram Srinivasan, Alex Tamkin, Rohan Taorj, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiauxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. _ArXiv_.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sasstry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alice Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_, NIPS '20, Red Hook, NY, USA. Curran Associates Inc.
* Chaudhary (2023) Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [https://github.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca).
* Volume 1_, NIPS'12, page 1223-1231, Red Hook, NY, USA. Curran Associates Inc.
* Dubey et al. (2017) Abhimanyu Dubey, Abhinav Jauhr, Abhinav Pandey, Abhishek Kadain, Ahmad Al-Dahle, Aiesha Letman, Akhi Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aoob Yong, Archi Mitra, Archie Stavankumar, Artem Korenev, Arthur Hinsyark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptise Roziere, Bethaby Brion, 

Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Choe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corine Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allconsius, Daniel Song, Danielle Pintz, Danny Livshits, David Estijobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Neil, Gregoire Mialon, Guangdong Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Laurens Geftert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnston, Joshue Saxe, Ju-Qing Jia, Kaylan Vasudden Alwala, KUpasani, Kate Plhawik, Keqian Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaaan, Lubo Malo, Lukas Blecher, Lukas Landazat, Luke de Oliveira, Madeline C. Munzi, Mahes Babu Pasqupleit, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, May Pavlova, Melissa Hall Melanic Rambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladir S. Chatterji, Olivier Duchenne, Onur Celebi, Patrick Alransky, Pengchuan Zhang, Pengwei Li, Petrar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubbal, Praveen Krishnan, Punit Singh Korura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Poliforo, Roshan Sumbaly, Ross Taylor, Run Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sabhana Chennalosappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosalie, Shunn Yandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gurrangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasho, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihalyo, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gongquet, Virginie Do, Vish Vogei, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenjun Fu, Whitney Meers, Xavier Martinet, Xiaoodong Wang, Xiaoqing Ellen Lan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yunning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakiops, Aadiitya K. Singh, Aaron Grattafiori, Abia Gangidi, Jain, Adam Kelsey, Adam Shajfeld, Adi Gangidi, Adolfo Victoria, Ahiva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandran, Annie Franco, Apariaria Sarf, Arkabandhu Chowdhury, Ashby Gabriel, Ashwin Bharame, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben Leonhardt, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changshan Wang, Changxyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tidal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang-Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testugging, Delia David, Devi Parikh, Didena Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presni, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm'an, Frank J. Kanayer, Frank Seide, Gabriela Medina Florez, Gabrilella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan, Hamid Shojanzzeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufunor, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Jupiter Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carrill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U Kamflou, Karan Saxena, Karthika Prasad, Karlixy Khandelwal, Katayoun Zandt, Kathystoisch, Kaushik Veerraghabavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kwslah Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madan Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennite, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kenneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanet, Mohammad Ratsegari, Munishi Bansal, Nandhnii Santhanam, Natascha Parks, Natasha White, Navyata Baswa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chermoguz, Olivia Hart, Omkar 

Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bondrager, Pierre Roux, Piot Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghodham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadaru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramasawamy, Shaum Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Statterfield, Sudarshan Govindaprasad, Sumit Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tezook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montaze, Vijai Mohan, Vinay Satish Kumar, Vishal Mangles, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _ArXiv_, abs/2407.21783.
* Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schutze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. _Transactions of the Association for Computational Linguistics_, 9:1012-1031.
* Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
* Gumbel (1954) Emil Julius Gumbel. 1954. _Statistical theory of extreme values and some practical applications: a series of lectures_. 33. US Govt. Print. Office.
* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Huang et al. (2020) Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furoug Huang, Hao Liu, Heng Ji, Hongyi Wang, Huaxh Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jinghong Wang, Joaquin Vanschoren, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quaquan Gu, Ran Xu, Rev Ying, Shuxiwang Ji, Supnan Jana, Tianlong Cheng, Tianming Liu, Tianyi Zhou, William Yang Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. 2024. Trustlm: Trustworthiness in large language models. In _Forty-first International Conference on Machine Learning_.
* Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Ji et al. (2024) Haozhe Ji, Pei Ke, Hongning Wang, and Minlie Huang. 2024. Language model decoding as direct metrics optimization. In _The Twelfth International Conference on Learning Representations_.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulinier, Lelo Renard Lavaud, Mariene Amexuchaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b. _CoRR_, abs/2310.06825.
* Jurafsky and Martin (2009) Dan Jurafsky and James H. Martin. 2009. _Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition_. Pearson Prentice Hall, Upper Saddle River, NJ.
* Kaddour et al. (2023) Jean Kaddour, J. Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models. _ArXiv_, abs/2307.10169.
* Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
* Kool et al. (2019) Wouter Kool, Herke van Hoof, and Max Welling. 2019. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 3499-3508. PMLR 

B. T. Lowerre. 1976. The Happy speech recognition system. Ph.D. thesis, Carnegie Mellon University, Pennsylvania.
* Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Maddison et al. (2014) Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014. A\({}^{*}\) Sampling. In _Advances in Neural Information Processing Systems 27_.
* Mitchell et al. (2022) Eric Mitchell, Joseph Noh, Siyan Li, Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. 2022. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1754-1768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bertnetworks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Renze and Guven (2024) Matthew Renze and Erhan Guven. 2024. The effect of sampling temperature on problem solving in large language models. _CoRR_, abs/2402.05201.
* Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In _Annual Meeting of the Association for Computational Linguistics_.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford aplaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca).
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Becher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibub, Jude Fernandes, Jeremy Fu, Wenynju Fu, Brian Fuller, Cynthia Gao, Vedauji Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inn, Marcin Kardas, Viktor Kerkez,adian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Korua, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yunjing Mao, Xavier Martinet, Todor Milaylov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashii Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Pavxin Xu, Zhengxu Yan, Iliyam Zarov, Yuchen Zhang, Angela Fan, Melanie Lambadour, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciailom. 2023. Libara 2: Open foundation and fine-tuned chat models. _ArXiv_, abs/2307.09288.
* Vilnis et al. (2023) Luke Vilnis, Yury Emelyanskiy, Patrick Murray, Alexandre Passos, and Sumit Sanghahi. 2023. Arithmetic sampling: parallel diverse decoding for large language models. In _Proceedings of the 40th International Conference on Machine Learning_, ICML'23. JMLR.org.
* Wang et al. (2024) Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11897-11916, Bangkok, Thailand. Association for Computational Linguistics.
* Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Akakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations_.
* Yan et al. (2024) Tianyi Yan, Fei Wang, James Y. Huang, Wencxun Zhou, Fan Yin, Aram Galstyan, Wenpeng Yin, and Muhao Chen. 2024. Contrastive instruction tuning. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 10288-10302, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Ye et al. (2023) Wentao Ye, Mingfeng Ou, Tianyi Li, Yipeng Chen, Xuetao Ma, Yifan Yanggong, Sai Wu, Jie Fu, Gang Chen, Haobo Wang, and Junbo Zhao. 2023. Assessing hidden risks of llms: An empirical study on robustness, consistency, and credibility. _CoRR_, abs/2305.10235.
* Zhang et al. (2021) Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021. Trading off diversity and quality in natural language generation. In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_, pages 25-33, Online. Association for Computational Linguistics.

 



## Appendix A Proof for Theorem 4.1

**Theorem**.: _Suppose we have two different categorical distributions parametrised by \(p^{1},...,p^{N_{v}}\) and \(q^{1},...,q^{N_{v}}\). Define a joint distribution over pairs of categories \((Y,V)\) by defining_

\[Y=\operatorname*{arg\,max}_{i}(\log p^{i}+g^{i}),\;V=\operatorname*{arg\,max}_{ i}(\log q^{i}+g^{i}), \tag{2}\]

_where \(g^{1},...,g^{N_{v}}\sim G(0,1)\) are independent. We have that_

\[P(Y=k,V=k)=\frac{p^{k}q^{k}}{p^{k}q^{k}+\sum_{i\neq k}\max\{p^{i}q^{k},q^{i}p^{ k}\}}.\]

Proof.: If

\[k=\operatorname*{arg\,max}_{i}\{\log p^{i}+g^{i}\}=\operatorname*{arg\,max}_{ i}\{\log q^{i}+g^{i}\},\]

then we must have that for all \(i\neq k\),

\[g^{i}<\log p^{k}+g^{k}-\log p^{i}\text{ and }g^{i}<\log q^{k}+g^{k}-\log q^{i},\]

i.e.,

Denoting by \(F\) the CDF of the Gumbel distribution, we can write

\[P(Y=k,V=k|g^{k})=\prod_{i\neq k}F(g^{k}+\min\{\log p^{k}/p^{i},\log q^{k}/q^{i}\}).\]

Denoting the PDF of the Gumbel distribution by \(f\) and marginalising we deduce that

\[P(Y=k,V=k)=\int_{-\infty}^{\infty}\prod_{i\neq k}F(g+\min\{\log p^{k}/p^{i}, \log q^{k}/q^{i}\})f(g)dg.\]

Expanding, we can write this as

\[P(Y=k,V=k) =\int\prod_{i\neq k}\exp\{-\exp\{-g-\min\{\log p^{k}/p^{i},\log q^{ k}/q^{i}\}\}\}\exp\{-g-\exp(-g)\}dg\] \[=\int\exp\{-g-\exp(-g)-\sum_{i\neq k}\exp\{-g-\min\{\log p^{k}/p^ {i},\log q^{k}/q^{i}\}\}\}dg\] \[=\int\exp\{-g-\exp(-g)(1+\sum_{i\neq k}\exp\{-\min\{\log p^{k}/p^ {i},\log q^{k}/q^{i}\}\})\}dg.\]

Recall that

\[\frac{d}{dx}e^{ex}=e^{x}e^{ex}=e^{x+e^{x}},\]

and so for any \(A\) independent of \(x\) we have that

\[\frac{d}{dx}e^{-Ac^{-z}}=Ae^{-x}e^{-Ac^{-x}}=Ae^{-x-Ae^{-z}}.\]

Therefore, we may deduce that

\[P(Y=k,V=k) =\left[\frac{1}{1+\sum_{i\neq k}\exp\{-\min\{\log p^{k}/p^{i},\log q ^{k}/q^{i}\}\}}e^{-Ac^{-g}}\right]_{-\infty}^{\infty}\]

Since

\[\exp\{-\min\{\log p^{k}/p^{i},\log q^{k}/q^{i}\}\} =\max\{\exp\{-\log p^{k}/p^{i}\},\exp\{-\log q^{k}/q^{i}\}\}\] \[=\max\{p^{i}/p^{k},q^{i}/q^{k}\},\]

we deduce

\[P(Y=k,V=k)=\frac{1}{1+\sum_{i\neq k}\max\{p^{i}/p^{k},q^{i}/q^{k}\}}.\]

as claimed. 

## Appendix B Proofs relating to Gumbel recycling procedure

Note that in the following proofs, we denote for notational simplicity that for a random vector \(x\), where each element of \(x\) is independently sampled according to a Gumbel distribution, \(x^{k}\sim G(0,1)\), \(p(x)=\prod_{k}G(x^{k};0,1)=G(x;0,1)\).

### Proof for Theorem 4.2

**Theorem**.: _Consider a sequence of tokens \(Y_{1:T}\) generated auto-regressively according to the following update rule, where \(k:=\operatorname*{arg\,max}_{j}(g_{t}^{j}+\log h_{t}^{j})\), \(Q(\cdot)\) is the quantile function for the \(G(0,1)\) distribution and \(\pi_{\theta}(\cdot)\) is a language model:_

\[g_{1} \sim G(0,1)\] \[h_{t+1} =\pi_{\theta}(X,Y_{1:t})\] \[g_{t+1}^{k}\mid g_{t},h_{t} \sim G(0,1)\] \[g_{t+1}^{i}\mid g_{t},h_{t} =Q\left(\frac{Q^{-1}(g_{t}^{i})}{Q^{-1}(g_{t}^{k}+\log h_{t}^{k}- \log h_{t}^{i})}\right),\quad\text{for $i\neq k$}\] \[Y_{t+1} =\operatorname*{arg\,max}_{j}\left(\log h_{t+1}^{j}+g_{t+1}^{j}\right)\]

_With this update procedure, the probability distribution over a given token conditioned on preceding tokens is_

\[p(Y_{t+1}\mid X,Y_{1:t})=\textit{Cat}(Y_{t+1};h_{t+1})\]

Proof.: We proceed through proof by induction. We make two assumptions that following expressions hold for \(t\), then prove that the expressions hold for \(t+1\) under those assumptions (and that they hold for the base case). The assumptions are that:

**Assumption 1**.: \[p(Y_{t}\mid X,Y_{1:t-1})=Cat(Y_{t};h_{t+1})\]

**Assumption 2**.: \[p(g_{t}\mid X,Y_{1:t-1})=G(g_{t};0,1)\]

N.B that in the base cases (\(p(Y_{1}\mid X)\) and \(p(g_{1}\mid X)\)), the expressions are trivially valid by the Gumbel reparameterization trick and by construction of \(g_{1}\) respectively.

Now, let's prove that the expressions hold for \(t+1\). We will first prove the following: 

by first considering the probability \(p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t})\):

\[p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t}) =\int p(g_{t+1}^{i}<a,g_{t}~{}|~{}X,Y_{1:t})\,dg_{t}\] \[=\int p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t},g_{t})p(g_{t}~{}|~{}X,Y_{1: t-1},Y_{t})\,dg_{t}\] \[=\int p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t},g_{t})\frac{p(Y_{t}~{}|~{}X, Y_{1:t-1},g_{t})p(g_{t}~{}|~{}X,Y_{1:t-1})}{p(Y_{t}~{}|~{}X,Y_{1:t-1})}\,dg_{t}\] \[=\frac{1}{p(Y_{t}~{}|~{}X,Y_{1:t-1})}\int p(g_{t+1}^{i}<a~{}|~{}h_ {t},Y_{t},g_{t})p(Y_{t}~{}|~{}h_{t},g_{t})G(g_{t};0,1)\,dg_{t}\]

Firstly, consider the case where \(Y_{t}=i\). In this case, we know that \(g_{t+1}^{i}\) is newly sampled from \(G(0,1)\). Therefore, using the Gumbel reparameterization trick for the last step, we have that:

\[p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t}) =\frac{1}{h_{t}^{i}}\int p(g_{t+1}^{i}<a~{}|~{}h_{t},Y_{t},g_{t})p (Y_{t}~{}|~{}h_{t},g_{t})G(g_{t};0,1)\,dg_{t}\] \[=\frac{1}{h_{t}^{i}}\int Q^{-1}(a)p(Y_{t}~{}|~{}h_{t},g_{t})G(g_{ t};0,1)\,dg_{t}\] \[=\frac{h_{t}^{i}}{h_{t}^{i}}Q^{-1}(a)=Q^{-1}(a)\]

Turning our attention to the case where \(Y_{t}=j\neq i\)

\[p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t}) =\frac{1}{p(Y_{t}=j~{}|~{}X,Y_{1:t-1})}\int p(g_{t+1}^{i}<a~{}|~{} h_{t},Y_{t}=j,g_{t})p(Y_{t}=j~{}|~{}h_{t},g_{t})G(g_{t};0,1)\,dg_{t}\] \[=\frac{1}{h_{t}^{i}}\int p(g_{t+1}^{i}<a~{}|~{}h_{t},Y_{t}=j,g_{t })p(Y_{t}=j~{}|~{}h_{t},g_{t})G(g_{t};0,1)\,dg_{t}\]

We simplify notation by denoting the following events:

\[E^{\prime} =\left\{Q\left(\frac{Q^{-1}(g_{t}^{i})}{Q^{-1}(g_{t}^{j}+\log h_{ t}^{j}-\log h_{t}^{i})}\right)<a\right\}\] \[E_{p} =\left\{g_{t}^{p}+\log h_{t}^{p}<g_{t}^{j}+\log h_{t}^{j}\right\}\]

Now, we can rewrite the following probabilities using these definitions:

\[p(g_{t+1}^{i}<a~{}|~{}h_{t},Y_{t}=j,g_{t}) =\mathbf{1}_{E^{\prime}(g_{t})}\] \[p(Y_{t}=j~{}|~{}h_{t},g_{t}) =\left(\prod_{p\not=j}\mathbf{1}_{E_{p}(g_{t})}\right)\] \[p(g_{t+1}^{i}<a~{}|~{}X,Y_{1:t}) =\frac{1}{h_{t}^{j}}\int\mathbf{1}_{E^{\prime}(g_{t})}\left(\prod _{p\not=j}\mathbf{1}_{E_{p}(g_{t})}\right)G(g_{t};0,1)\,dg_{t}\]

Since \(Q^{-1}(x)\) is a monotonic function, \(E_{i}\) is equivalently defined as:

\[E_{i}=\left\{Q^{-1}\left(g_{t}^{i}\right)<Q^{-1}\left(g_{t}^{j}+\log h_{t}^{j}- \log h_{t}^{i}\right)\right\}\]

Additionally, \(E^{\prime}\) can be rewritten as 

Since \(Q^{-1}(a)\in[0,1]\), the occurrence of \(E^{\prime}\) is a sufficient condition for the occurrence of \(E_{i}\). Therefore, we can simplify the integral to:

\[p(g_{t+1}^{i}<a\mid X,Y_{1:t})=\frac{1}{h_{t}^{j}}\int\mathbf{1}_{ E^{\prime}(g_{t})}\left(\prod_{p\neq i,j}\mathbf{1}_{E_{p}(g_{t})}\right)G( g_{t};0,1)\,dg_{t}\]

The CDF of the Gumbel distribution can be written \(Q^{-1}(x)=e^{-e^{-x}}\), so \(Q^{-1}(x+c)=\big{(}Q^{-1}(x)\big{)}^{e^{-c}}\). With this fact and application of the monotonic transformation \(Q(\cdot)\), we can rewrite the events :

\[E^{\prime} =\left\{Q^{-1}\left(g_{t}^{i}\right)Q^{-1}\left(g_{t}^{j}\right) ^{\frac{h_{t}^{j}}{h_{t}^{j}}}<Q^{-1}(a)\right\}\] \[E_{p} =\left\{Q^{-1}\left(g_{t}^{p}\right)<Q^{-1}\left(g_{t}^{j}\right) ^{\frac{h_{t}^{p}}{h_{t}^{j}}}\right\}\]

We now use the fact that \(Q^{-1}(g_{t}^{i})\coloneqq U_{t}^{i}\sim\mathcal{U}[0,1]\,\,\,\forall i\) to rewrite the events like so:

\[E^{\prime} =\left\{U_{t}^{i}\left(U_{t}^{j}\right)^{-\frac{h_{t}^{j}}{h_{t}^ {j}}}<Q^{-1}(a)\right\}\] \[E_{p} =\left\{U_{t}^{p}<\left(U_{t}^{j}\right)^{\frac{h_{t}^{p}}{h_{t}^ {j}}}\right\}\]

In conjunction with lemma B.1, this gives us the desired cumulative density function:

\[p(g_{t+1}^{i}<a\mid X,Y_{1:t})=\frac{1}{h_{t}^{j}}(h_{t}^{j})Q^{-1}(a)=Q^{-1}(a)\]

Since the cumulative density function in both cases (\(Y_{t}=i\) and \(Y_{t}\neq i\)) is \(Q^{-1}(a)\), we have that, under our initial assumptions, \(p(g_{t+1}\mid X,Y_{1:t})=G(g_{t+1};0,1)\).

Finally, we then introduce and marginalise over the Gumbel noise vector at the previous timestep for the distribution over \(Y_{t+1}\), where the final step follows from the Gumbel reparameterization trick:

\[p(Y_{t+1}\mid X,Y_{1:t}) =\int p(Y_{t+1},g_{t+1}\mid X,Y_{1:t})\,dg_{t+1}\] \[=\int p(Y_{t+1}\mid X,Y_{1:t},g_{t+1})p(g_{t+1}\mid X,Y_{1:t})\, dg_{t+1}\] \[=\int p(Y_{t+1}\mid h_{t+1},g_{t+1})G(g_{t+1};0,1)\,dg_{t+1}\] \[=\text{Cat}(Y_{t+1};h_{t+1})\]

Therefore, since the expressions are valid for the base case of \(t=1\), and we have shown them to be valid for \(t+1\) if assumptions 1 and 2 hold, they must be true for all \(t\), by induction. 

### Statement and Proof of lemma b.1

**Lemma B.1**.: \(X\)_, \(Y\) and \(Z_{1:N}\) are random variables each independently drawn from \(\mathcal{U}[0,1]\). A, \(B\), \(C_{1:N}\) and \(D\) are positive constants between 0 and 1, and \(A+B+\sum_{n}C_{n}=1\). Defining the events \(E^{*}=\left\{XY^{-\frac{A}{B}}<D\right\}\) and \(E_{n}=\left\{Z_{n}<Y^{\frac{Cn}{B}}\right\}\), the probability of the intersection of events is given by:_

\[P\left(E^{*}\cap\bigcap_{n=1}^{N}E_{n}\right)=BD\] 

Proof.: We can write down the following probabilities that are conditional on \(Y\):

\[P(E^{*}|Y)= P\left(X\leq DY^{\frac{A}{B}}\right)=DY^{\frac{A}{B}}\] \[P(E_{n}|Y)= P\left(Z_{n}\leq Y^{\frac{C_{n}}{B}}\right)=Y^{\frac{C_{n}}{B}}\]

Therefore, the probability of the complement is given by integrating the product of these quantities over \(p(y)\):

\[P\left(E^{*}\cap\bigcap_{n=1}^{N}E_{n}\right) =\int_{0}^{1}P(E^{*}|Y)\prod_{n=1}^{N}P(E_{n}|Y)dY\] \[=\int_{0}^{1}\left(DY^{\frac{A}{B}}\right)\prod_{n=1}^{N}\left(Y ^{\frac{C_{n}}{B}}\right)dY\] \[=\int_{0}^{1}\left(DY^{\frac{A+\sum_{n}C_{n}}{B}}\right)dY\] \[=D\frac{1}{\frac{A+\sum_{n}C_{n}}{B}+1}\] \[=D\frac{1}{\left(\frac{A+\sum_{n}C_{n}+B}{B}\right)}=BD\]

## Appendix C Proof of unbiased estimator for \(p_{repeat}\)

**Claim**.: _Let \(p\) denote the probability of some Bernoulli event. an unbiased estimator of \(p\) given by a finite set \(N\) of samples \(Z_{1:N}\) from the distribution is given by:_

\[\hat{p}=\frac{1}{N}\sum_{i=1}^{N}Z_{i}\]

_An unbiased estimator of \(p_{repeat}=p^{2}+(1-p)^{2}\) is:_

\[\frac{N}{N-1}(\hat{p}^{2}+(1-\hat{p})^{2})-\frac{1}{N-1}\]

Proof.: Calculate the expectation of \(\hat{p}^{2}\):

\[\mathbb{E}(\hat{p}^{2})=E\left(\left(\frac{1}{N}\sum_{i=1}^{N}Z_{i}\right)^{2}\right)\]

Expand the square inside the expectation:

\[\mathbb{E}(\hat{p}^{2})=\frac{1}{N^{2}}\,\mathbb{E}\left(\sum_{i=1}^{N}Z_{i}^ {2}+\sum_{i\neq j}Z_{i}Z_{j}\right)\]

Since \(Z_{i}^{2}=Z_{i}\), and by linearity of expectation:

\[\mathbb{E}(\hat{p}^{2})=\frac{1}{N^{2}}\left(Np+N(N-1)p^{2}\right)\]

Simplify the expression: 

Using this result, we have the following:

\[{\mathbb{E}}(\hat{p}^{2}+(1-\hat{p})^{2}) =E\left(2\hat{p}^{2}-2\hat{p}+1\right)\] \[=2E(\hat{p}^{2})-2E(\hat{p})+1\] \[=2\left(\frac{1}{N}p+\frac{N-1}{N}p^{2}\right)-2p+1\] \[=\frac{1}{N}\left((N-1)(2p^{2}-2p+1)+1\right)\] \[=\frac{N-1}{N}\left((2p^{2}-2p+1)+\frac{1}{N-1}\right)\] \[=\frac{N-1}{N}p_{repeat}+\frac{1}{N}\]

Therefore, we can debias the naive estimator using the following expression:

\[\frac{N}{N-1}(\hat{p}^{2}+(1-\hat{p})^{2})-\frac{1}{N-1}\]

## Appendix D Justification for ensembling procedure

**Theorem**.: _Suppose we have a set of categorical distributions \(\{P_{i}\}_{i=1}^{n}\), define \(Q^{*}\) as the distribution minimizing the average forward Kullback-Leibler divergence to each \(\{P_{i}\}_{i=1}^{n}\):_

\[Q^{*}=\arg\min_{Q}\frac{1}{n}\sum_{i=1}^{n}D_{KL}(Q\|P_{i}) \tag{3}\]

_then \(Q^{*}(x)\) can be expressed as_

\[Q^{*}(x)=\frac{1}{Z}\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}} \tag{4}\]

_where Z is the normalisation constant to ensure \(Q^{*}\) defines a valid probability distribution function_

\[Z=\sum_{x}\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}}\]

Proof.: Expanding the KL divergence

\[\frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})=\frac{1}{n}\sum_{ i=1}^{n}\sum_{x}Q(x)\log\frac{Q(x)}{P_{i}(x)}\]

Changing the order of sums, this can be re-expressed as

\[\frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})=\frac{1}{n}\sum _{x}Q(x)\log\frac{Q(x)^{n}}{\prod_{i=1}^{n}P_{i}(x)}=\sum_{x}Q(x)\log\frac{Q( x)}{\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}}}\]

Introducing the normalisation constant Z

\[\frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})=\sum_{x}Q(x)\log \frac{\frac{1}{Z}Q(x)}{\frac{1}{Z}\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}}}= \frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})\] 

separating the Z in the numerator

\[\frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})=\sum_{x}Q(x)\log\frac{Q(x)}{ \frac{1}{Z}\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}}}+\log\frac{1}{Z}\]

thus

\[\frac{1}{n}\sum_{i=1}^{n}D_{\text{KL}}(Q\|P_{i})=D_{\text{KL}}(Q\|\frac{1}{Z} \prod_{i=1}^{n}P_{i}^{\frac{1}{n}})+\log\frac{1}{Z}\]

this will be minimised when the right-hand side KL is equal to zero which occurs at \(Q^{*}(x)=\frac{1}{Z}\prod_{i=1}^{n}P_{i}(x)^{\frac{1}{n}}\) 

## Appendix E Experimental details

For all experiments, answer generation is done using language models quantised to bfloat16 Dean et al. (2012). Chat and instruction-tuned models are prompted using default templates whereas base models (mistral, llama3-base) are prompted with a template consisting of a single in-context example to help steer away from off-topic answers. The addition of this in-context example was found to not materially impact the efficacity of Gumbel sampling but impact the quality of responses.

To avoid excessive experiment run-times, we restrict generated responses to a maximum length after which we interrupt text generation. This limit was set to 50 new tokens for semantic similarity experiments and to 200 new tokens for stylistic similarity experiments. We ran small-scale experiments with larger maximum response length and did not find material evidence of experimental findings being impacted by this response truncation.

When measuring semantic similarity between responses we measure consistency between responses rather than response-question pairs with any follow-up questions or answers hallucinated by the language model removed programmatically.

## Appendix F Stylistic consistency prompts

### Aleatoric-list

Listing 1: Zero-shot classification prompt for whether model-generated response contains bulletpoints. Placeholders for question-specific content are shown in red.

```
Doesthefollowingresponseseparatiitemsintheanswerusingbulletpoints(*/-),letters(a/b/c...),numerics(1,2,3)orifitemsarennotseparatedrespondwith'doesn'tseparate'? Response:''(response)'''Tomakeyouranswereasytoextractrespondwithonlyoneofthefollowingoptions'usesbullets'/'usesletters'/'usessnumerics'/'doesn'tseparate'
```

Listing 2: Zero-shot classification prompt for whether model-generated response is terse. Placeholders for question-specific content are shown in red.

```
Istheresponsetreseroot?Response:''(response)''.Tommakesyoursanwereeasytoextractedrespondwithonlyoneofthefollowingoptions'terse'/'notterse'.
```

Listing 3: Zero-shot classification prompt for determining programming language of model-generated response. Placeholders for question-specific content are shown in red.

``` Whatistherogramminglanguageusedintheprovidedresponse.IfnoprogramminglanguageissuedreturnNone Response:''(response)''.Yourresponseshouldonlycontainthemaswerandnothingelse.

 

Listing 4: Zero-shot classification prompt for determining if model-generated response contains comments. Placeholders for question-specific content are shown in red.

```
Doesthecodesnippetinthefollowingresponsecontaincomments. Response::'{response}''. Answereitherwithyes/no.Yourresponseshouldonlycontainthemaswerandnothingelse.
```

Listing 5: Zero-shot classification prompt for determining if model-generated response contains code snippet. Placeholders for question-specific content are shown in red.

```
Doesthefollowingresponsecontainacodesnippet. Response:'''{response}''Answereitherwithyes/no.Yourresponseshouldonlycontainthemaswerandnothingelse.
```

Listing 6: Zero-shot classification prompt for determining if model-generated response contains code snippet. Placeholders for question-specific content are shown in red.

## Appendix G Example responses and semantic similarity

### Llama3-base-7B without gumbel consistency sampling at T=0.8

\begin{table}
\begin{tabular}{p{199.2pt} p{199.2pt} p{199.2pt} r} \hline \hline
**Question** & **Response 1** & **Response 2** & **Similarity (\%)** \\ \hline Break down the application of word embeddings in Natural Language Processing / Illuminate the use of word embedings in Natural Language Processing & Natural Language ProcessingNLP is a branch of Artificial Intelligence (AI) which deals with the interaction between computers and human languages. Word2vec is the most popular class of word embeddin... & It is used to represent words and their context with numerical values. & 68.0 \\ Expand on the function of a computer motherboard. / Illustrate the function of a computer motherboard. & The motherboard is the framework of the computer. It houses the CPU, RAM, and other components. & The computer’s motherboard serves a number of functions. It directs information from the CPU to the memory, makes sure the proper voltage levels are reached, and directs the data flow in and out of th... & 78.0 \\ Decode the abbreviation DNA. & DNA is an abbreviation for deoxyribonucleic acid. & DNA is also known as Deoxyribonucleic Acid. It is found in every single cell in the body and each cell contains the complete set of genetic information. It is the genetic blueprint that determines the... & 78.0 \\  / Share the meaning behind DNA. & 



### Llama3-base-7B with gumbel consistency sampling (recycling) at T=0.8

### Llama3-instruct-7B without gumbel consistency sampling at T=0.8

### Llama3-instruct-7B without gumbel consistency sampling at T=0.8

\begin{table}
\begin{tabular}{p{142.3pt} p{56.9pt} p{142.3pt} p{142.3pt}} \hline \hline
**Question** & **Response 1** & **Response 2** & **Similarity (\%)** \\ \hline Break down the application of word embeddings in Natural Language Processing / Illumina the use of word embeddings in Natural Language Processing & Embedding is the encoding of words in the context with a real-valued vector of a certain dimension. The low dimension helps to reduce the computational complexity of the models. Embedding can be learn... & Embedding is the encoder of words in the context of a sentence with a real-valued vector. The use of word embeddings helps to solve the problem of ‘Word sense disambiguation’. & 85.0 \\ \hline Expound on the function of a computer motherboard. & The motherboard of a computer is responsible for providing a workspace where all the hardware components can be connected. The motherboard also controls the componentation between the various component.. & The motherboard of a computer is responsible for providing communication between all the various components. The motherboard also controls basic functions such as the timing of the hardware, start-up ... & 90.0 \\ \hline Decode the abbreviation DNA. / Share the meaning behind DNA. & DNA stands for deoxyribonucleic acid & DNA means Deoxyribonucleotide acid & 90.0 \\ \hline \end{tabular}
\end{table}
Table 4: Examples of semantic similarity between responses for various questions

\begin{table}
\begin{tabular}{p{142.3pt} p{142.3pt} p{142.3pt} p{142.3pt}} \hline \hline
* Deoxy N
- Nucleica A
- Acid. In other words, DNA is Deoxyribose N... & What an fascinating topic! DNA (Deoxyribonucleic acid) is a molecule that contains the genetic instructions used in the development and function of all living organisms. It’s often r... & 78.0 \\ \hline \end{tabular}
\end{table}
Table 5: Examples of semantic similarity between responses for various questions




### Llama3-instruct-7B with gumbel consistency sampling (recycling) at T=0.8

## Appendix H Evaluation of semantic similarity using different metric choices

### Jaccard similarity

In Table 7, we reproduce mean semantic similarity results quoted in Section 6.1 using Jaccard similarity where we measure Jaccard similarity on the set of tokens produced by each model's own associated tokenizer.

### Sentencebert

In Table 8, we reproduce mean semantic similarity results quoted in Section 6.1 using the all-mpnet-base-v2 model recommended by the popular popular sentencetransformer repository [Reimers and Gurevych, 2019].

\begin{table}
\begin{tabular}{p{54.1pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline
**Question** & **Response 1** & **Response 2** & **Similarity (\%)** \\ \hline Break down the application of word embeddings in Natural Language Processing / Humanate the use of word endodings in Natural Language Processing & Embedding is the encoding of words in the context with a real-valued vector of a certain dimension. The low dimension helps to reduce the computational complexity of the models. Embedding can be learn... & Embedding is the encoding of words in the context of a sentence with a real-valued vector. The use of word embeddings helps to solve the problem of ‘Word sense disambiguation’. & 85.0 \\ \hline Expound on the function of a computer motherboard. / Illustrate the function of a computer motherboard. & The motherboard of a computer is responsible for providing a workspace where all the hardware components can be connected. The motherboard also controls the communication between the various component... & The motherboard of a computer is responsible for providing communication between all the various components. The motherboard also controls basic functions such as the timing of the hardware, start-up ... & 90.0 \\ \hline Decode the abbreviation DNA. / Share the meaning behind DNA. & DNA stands for deoxyribonucleic acid & DNA means Deoxyribonucleic acid & 98.0 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Examples of semantic similarity between responses for various questions

\begin{table}
\begin{tabular}{l l c r} \hline \hline
**Model** & **Sampler** & **Without Ensembling** & **With Ensembling** \\ \hline  & IS & 0.351\(\pm\)0.002 & 0.385\(\pm\)0.011 \\ Llama2 Chat & GCS & 0.420\(\pm\)0.006 & 0.495\(\pm\)0.005 \\  & GCSwR & **0.444\(\pm\)0.005** & **0.521\(\pm\)0.007** \\ \hline  & IS & 0.115\(\pm\)0.004 & 0.119\(\pm\)0.005 \\ Mistral & GCS & 0.266\(\pm\)0.010 & 0.340\(\pm\)0.020 \\  & GCSwR & **0.335\(\pm\)0.013** & **0.393\(\pm\)0.008** \\ \hline  & IS & 0.320\(\pm\)0.011 & 0.346\(\pm\)0.000 \\ Llama3 Instruct & GCS & 0.365\(\pm\)0.017 & 0.442\(\pm\)0.004 \\  & GCSwR & **0.410\(\pm\)0.003** & **0.479\(\pm\)0.008** \\ \hline  & IS & 0.086\(\pm\)0.001 & 0.096\(\pm\)0.009 \\ Llama3 Base & GCS & 0.224\(\pm\)0.015 & 0.281\(\pm\)0.009 \\  & GCSwR & **0.314\(\pm\)0.012** & **0.371\(\pm\)0.010** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Model results by sampler type, for the Jaccard similarity, with and without our ensembling approach. Scores shown as mean\(\pm\)std.err with std.err obtained from 3 independent runs. Bold indicates highest scores for each model in both ensembling categories.





## Appendix I Python implementation

We provide below a self-contained reference Python implementation of our GCSwR algorithm.

``` fromtypingimportOptional,Union importnumpyasnp imports numpy.typingasmpt importtorch defuniform_to_gumble_fn(z,mu=0,beta=1): samples = mu - beta* torch.log(-torch.log(z)) returnsamples def gumbel_to_uniform_fn(z,mu=0,beta=1): uniform_samples = torch.exp(-torch.exp(-(z - mu)/beta)) returnuniform_samples classGumbelSampler: def__init__( self, rng_seed:int, memory_size:int=100, recycle_strategy:Union[str,int]="always", ) ->None: self.rng_seed=rng_seed self.is_initialised=False self.memory_size=memory_size self.recycle_strategy=recycle_strategy defsample(self,logprobs:torch.Tensor)->int: gumbel_noise=self.getcurrent_gumbel(num_cats=len(logprobs)) sampled_idx=torch.argmax(gumbel_noise.squeeze()+logprobs.squeeze()).item  () self.recycle_gumbel(gumbel_noise,logprobs,sampled_idx) returnsampled_idx defeq_current_gumbel(self,num_cats:int=None)->torch.Tensor:""Getthecurrentgumbenoisecvectorandinitializegumbelnoiseifithasnotyetbeeninitialized."" ifnofself.is_initialised:

self.rng =np.random.default_rng(self.rng_seed) self.gumbel_mem =self.make_gumbel_noise( num_cats=num_cats,num_samples=self.memory_size,rng=self.rng  ) self.mem_loc =torch.zeros(num_cats,dtype=torch.int64) self.is_initialised =True gumbel_noise = torch.gather(self.gumbel_mem,1,self.mem_loc[:,None]) returngumbel_noise defset_current_gumbel(self,gumbel_noise:torch.Tensor) -> None: self.gumbel_mem.scatter_{1,self.mem_loc[:,None],gumbel_noise[:,None]}

defrecycle_gumbel( self, gumbel_noise:torch.Tensor, logprobs: torch.Tensor, sampled_idx:int, ) -> None: ifself.recycle_strategy=="never": self.mem_loc +=1 else: uniform_noise =gumbel_to_uniform_fn(gumbel_noise) scaler = gumbel_to_uniform_fn( logprobs[sampled_idx] + gumbel_noise[sampled_idx] -logprobs ) updated_gumbel =uniform_to_gumble_fn(uniform_noise.squeeze()/scaler) self.set_current_gumbel(updated_gumbel) self.mem_loc[sampled_idx] +=1 @staticmethod deff make_gumbel_noise( num_cats:int, num_samples:int, ring: Optional[np.random.generator.Generator]=None, ) -> np.array: ifrng: returntorch.Tensor(rng.gumbel(0,1,size=(num_cats,num_samples))) else: returntorch.Tensor(np.random.gumbel(0,1,size=(num_cats,num_samples)) ) defreset(self): self.is_initialised = False

defsample_n_new( n:init, N_vocab:int, rng_seed:Optional[int] = 0, lopprobs:Optional[npt.NDArray]=None, ) -> list[in]: iflogprobs isNone: norm_probs = np.random.dirichlet(np.ones(N_vocab)) logprobs = np.log(norm_probs) sampler = GumbelSampler(rng_seed=rng_seed) sequence = [] for__inrange(n): sequence.append(sampler.sample(torch.Tensor(logprobs))) returnsequence

if___name__=="main__': n =2 N_vocab = 2 num_samples = 10000 logprobs = np.log(np.array([0.4,0.6])) results = [] forsed imrange(num_samples): results.append( 

sample_n_new( n, N_vocab, seed, logprobs, ) print(f*IS:Probability_of_sampling_l_in_both_positions:{np.exp(logprobs)[1]"} print( f^GCSwR:Empirical_probability_of_sampling_l_in_positions_l_&2:{np.array( results).mean(axis=0)}-  )