Title: Reranking Laws for Language Generation: A Communication-Theoretic Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: language generation

reranking

communication theory

reliability
Abstract: To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.

Reranking Laws for Language Generation: A Communication-Theoretic Perspective

Abstract

To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free ( i.e. , yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.

1 Introduction

Large language models (LLMs) have shown remarkable performance across many tasks in natural language processing, computer vision, and speech recognition. Despite their capabilities, instances of hallucinations and other critical errors occasionally arise, casting doubt on the reliability of their predictions, without clear indication of when and how badly they might fail (Ji et al., 2023; Guerreiro et al., 2023). This is particularly concerning as these models are increasingly used in high-stakes applications such as those within the medical or legal domains (Hung et al., 2023) or as agents that can perform multiple tasks, including generating and executing code (Wang et al., 2024).

The most common mitigation strategy is to 'steer' the LLM with the aid of a reward model or directly from human preferences, either at training time (Stiennon et al., 2020; Yuan et al., 2024; Rafailov et al., 2024) or during decoding (Liu et al., 2024; Huang et al., 2024). A simple and effective decoding-time strategy is first to generate multiple hypotheses and then use a reranker to select the most appropriate one. Several generation techniques used with modern LLMs, including voting procedures (Borgeaud and Emerson, 2020; Wang et al., 2023; Liévin et al., 2024; Shi et al., 2022), minimum Bayes risk decoders (Eikema and Aziz, 2020; Freitag et al., 2022), quality-aware decoders (Fernandes et al., 2022), or other types of hypothesis ensembling/reranking techniques (Farinhas et al., 2023; Ni et al., 2023; Bertsch et al., 2023; Li et al., 2024), embody this idea. An essential aspect of these procedures is that they all add redundancy as an intermediate step (by generating multiple hypotheses) to increase the chances of returning an acceptable answer as the final output.

The idea of adding redundancy to decrease the error rate in noisy channels is a cornerstone of communication theory , more specifically in forward error correction methods. In its simplest

Generator

Sender

Reranker

Receiver

form-repetition codes-a message block is sent multiple times, and the decoder uses some form of majority voting to recover the original message with high probability (MacKay, 2002; Cover and Thomas, 2006). The same idea underlies more sophisticated error-correcting codes (Hamming, 1950; Reed and Solomon, 1960; Gallager, 1962; Berrou et al., 1993).

In this paper, we draw a parallel between these two worlds by regarding generator-reranker LLMs as communication systems (§2 and Fig. 1, left). We conceptualize the LLM generator G as a sender transmitting N message descriptions in parallel through noisy channels, leading to N potentially corrupted hypotheses. Then, the receiver, which corresponds to the reranker R , decodes the message by ranking the potentially corrupted descriptions and selecting the one found to be most reliable. The goal is for the combined ( G,R ) system to have lower error rate than G alone, and for the error rate to decay quickly with N . Our main contributions are as follows:

We show that when the channel distributions are independent, this simple protocol is asymptotically error-free ( i.e. , it generates an acceptable answer almost surely when N →∞ ), even in scenarios where the reranker is imperfect, e.g. , governed by a Mallows or a Zipf-Mandelbrot model. In the former case, the error probability decays exponentially fast (§3).

We show that the protocol is still asymptotically error-free if we assume that the channel distributions are statistically dependent. When they are coupled by a Beta prior, we show that the error probability decays as a power law when the reranker is perfect (§4).

We use our framework to obtain 'reranking laws', which we validate empirically on text-to-code generation with DeepSeek-Coder 7B (§5.1), on machine translation of medical data with TowerInstruct 13B (§5.2), and on mathematical and commonsense reasoning benchmarks (App. B.3).

Notation. We denote [ N ] := { 1 , ..., N } and we use the shorthand notation X 1: N := ( X 1 , ..., X N ) . We use capital letters ( X,Y,... ) for random variables and represent probability distributions by P ( X ) , P ( Y ) , etc. We denote expectations of functions f under P ( X ) by E X [ f ( X )] .

2 A Communication-Theoretic Perspective of Generator-Reranker Systems

The focus of our paper is on generator-reranker systems : a generator G (such as an LLM) is prompted with a query q ( e.g. , a question to be answered, a source text to be translated, or a textual prompt for code). As a response to this query, G generates N candidate answers y 1 , ..., y N (called hypotheses ). We are agnostic about the internals of G and the way the hypotheses are generated: they could come from the same system through sampling or beam search, or they could come from an ensemble of different systems. These hypotheses are then processed by a reranker R , which ranks them and returns as the final output the one which is found to be the best answer. We are also agnostic about how R is built-it could be an external system or it could be part of (or share parameters with) the generator. Commonly used rerankers are quality estimators (Fernandes et al., 2022), energy-based models (Bhattacharyya et al., 2021), reward models (Li et al., 2022), and minimum Bayes risk decoders (Kumar and Byrne, 2002; Eikema and Aziz, 2020; Freitag et al., 2022; Shi et al., 2022).

Regardless of specific design decisions, the goal of the generator-reranking system ( G,R ) is to leverage the reranker R to produce better answers (according to some quality metric) than the ones which would be obtained through G alone ( e.g. , a single sample). In this paper, we show that the propensity for this combined system to generate unacceptable outputs, such as those containing critical errors or hallucinations, decays quickly enough with N under mild assumptions on G and R .

We draw an analogy with communication theory as follows. Let Σ be an underlying alphabet and Σ ∗ := ⋃ ∞ i =0 Σ i its Kleene closure, i.e. , the set of strings from Σ . Given the query q , we denote by X ( q ) ⊆ Σ ∗ the set of acceptable answers . 1 We assume the communication system depicted in Fig. 1 (left), a form of multiple description source coding (Ozarow, 1980; Gamal and Cover, 1982; Laneman et al., 2005). In this framework, the sender transmits N acceptable answers (called descriptions ) x 1 , ..., x N ∈ X ( q ) N in parallel through noisy channels. These descriptions are corrupted according to a distribution P ( y 1 , ..., y N | x 1 , ..., x N ) , so that some hypotheses y i may become unacceptable ( y i ∈ Σ ∗ \ X ( q ) ). This 'channel noise' is a way to conceptualize the imperfections of the generator G . On the receiver side, a decoder processes the (potentially) corrupted descriptions and estimates ˆ x = g ( y 1 , ..., y N ) using some decoding function g . The overarching goal is to achieve a low error probability P err ( N ; q ) := P ( ˆ X / ∈ X ( q ) | q ) for any query q . By bounding the maximal probability of error (over all queries), the average error probability is automatically bounded (Cover and Thomas, 2006, §8). In this paper, we focus on rerankers as the decoding functions, where g ( y 1 , ..., y N ) returns the top ranked answer, i.e. , g ( y 1 , ..., y N ) = y i for some i ∈ [ N ] .

We formalize this construction by considering different models for G and R in the following sections, studying the conditions under which the resulting protocol is asymptotically error-free :

Definition 1. A protocol is asymptotically error-free if, for any query q , the probability of the decoder outputting an unacceptable answer approaches zero as N tends to infinity, i.e.,

For simplicity, we assume that X 1 , ..., X N are conditionally independent given the query q , i.e. , that P ( x 1 , ..., x N | q ) = ∏ N i =1 P ( x i | q ) . 2 We also assume that Y 1: N are independent from q given X 1: N such that q → X 1: N → Y 1: N forms a Markov chain. Taken together, these two assumptions mean that P ( x 1: N , y 1: N | q ) = P ( x 1: N | q ) P ( y 1: N | x 1: N ) = ( ∏ N i =1 P ( x i | q ) ) P ( y 1: N | x 1: N ) .

3 Generator-Reranker Systems with Independent Hypotheses

We first consider the case where the corrupted descriptions Y 1: N are conditionally independent and identically distributed (i.i.d.) given X 1: N and where Y i depends only on X i , that is, P ( y 1: N | x 1: N ) = ∏ N i =1 P ( y i | x i ) . Conceptually, this is the scenario where the parallel channels do not interfere, and it corresponds to the graphical model shown in Fig. 1 (right) without the part in red. While this case may not be very realistic in practice-for example, when the hypotheses produced by the generator are all sampled from the same model-it makes the analysis simpler. We will show later in §4 how the analysis can be extended when this assumption does not hold, reusing the results from this section.

In the sequel, given a query q , we let ϵ denote the probability of a hypothesis being unacceptable, ϵ := P ( Y i / ∈ X ( q ) | X i = x i , q ) = P ( Y i / ∈ X ( q ) | X i = x i ) .

3.1 Perfect and random rerankers

We start by assuming that R is a perfect reranker , which implies that it produces an acceptable output when presented with a set of N hypotheses if and only if at least one of them is acceptable. In

this case, the error probability becomes

Thus, P err ( N ; q ) goes to zero exponentially fast with N for any ϵ ∈ [0 , 1) , indicating that when the hypotheses are independent and the reranker is perfect, the protocol is error-free.

On the other end of the spectrum, if the reranker is random -i.e. , if it selects one of the N hypotheses uniformly at random, we obtain

that is, we obtain the same error probability as the generator alone, as expected.

3.2 Imperfect reranker: Mallows model

We consider now more realistic rerankers. A statistical ranking model widely used in machine learning applications is the Mallows model (Klementiev et al., 2008, 2009; Chierichetti et al., 2018; Tang, 2019). Let Π denote the set of permutations over N elements, and let d : Π × Π → R + be a distance function between permutations. In this paper, we use the Kendall-tau distance d ( π, π ′ ) , which returns the number of adjacent transpositions needed to turn π into π ′ . Given a location parameter π 0 ∈ Π and a scale parameter λ ∈ R + , the probability of a ranking π according to the Mallows model is P ( π ; π 0 , λ ) = exp( -λd ( π, π 0 )) /Z ( λ ) , where Z ( λ ) is the partition function.

In our setting, we assume that π 0 is the ground truth (oracle) ranking 3 of the hypotheses y 1 , ..., y N and π is the ranking obtained by the reranker model, so that P ( π ; π 0 , λ ) expresses how imperfect the reranker might be. Note that the family of Mallows models include both perfect and random rerankers as limit cases, respectively as λ → + ∞ and as λ = 0 . 4

Let η j denote the marginal probability that the reranker places at the top the j th highest ranked hypothesis according to the oracle, i.e. , η j = P ( π 0 ( π -1 (1)) = j ) . When K out of the N hypotheses are unacceptable, the reranker will pick an unacceptable hypothesis with probability ∑ N j = N -K +1 η j . Combining this with the fact that the probability of G generating K unacceptable hypotheses is a binomial distribution, the error probability becomes

Note that (4) holds for any reranker with top-1 (marginal) probability mass function η = [ η 1 , ..., η N ] , not only Mallows models. Naively determining η would require marginalizing P ( π ; π 0 , λ ) by summing over all permutations π satisfying π 0 ( π -1 (1)) = j , which is intractable due to the factorial number of terms involved. Fortunately, tractable combinatorial expressions exist for Mallows models (Fligner and Verducci, 1986; Lebanon and Mao, 2008): the partition function has the compact expression Z ( λ ) = ∏ N j =1 (1 -e -λj ) / (1 -e -λ ) , and we have (Lebanon and Mao, 2008, Prop. 5):

0

50

100

N

120

90

60

30

0

Log failure rate (wrt baseline)

0

50

100

N

4

3

2

1

0

e

=1.00

e

=0.75

e

=0.50

e

=0.25

e

=0.00

=1.0

=0.5

=0.25

Plugging (5) into (4), invoking the binomial theorem, and simplifying, we obtain

Notably, when λ → + ∞ (perfect reranking), the failure probability becomes ϵ N , as expected (see (2)), demonstrating the model's ability to interpolate between scenarios of random reranking ( λ = 0 ) with a failure probability of ϵ (see (3)), and optimal reranking ( λ → + ∞ ) with a failure probability of ϵ N . A plot is shown in Fig. 2 (left), for several values of e -λ ∈ [0 , 1] .

Our next result, proved in App. A.1, shows that, even with an imperfect reranker, an asymptotically error-free protocol is possible:

Proposition 1. When R is a Mallows reranker, for any λ > 0 , the protocol is asymptotically error-free and the error probability decays exponentially fast, P err ( N ; q ) = O (( e -λ (1 -ϵ ) + ϵ ) N ) .

This result shows that P err ( N ; q ) converges Q-linearly to zero with rate of convergence e -λ (1 -ϵ ) + ϵ > ϵ . Therefore, Mallows rerankers behave asymptotically as a perfect reranker but where the generator has an increased error probability.

Given this result, one might wonder whether any reranker 'slightly better than random' suffices to obtain an asymptotically error-free protocol. This it not the case, as the next counter-example shows.

Example 1. Assume a reranker with probability mass function η j ∝ ( N -j +1) . The resulting protocol is not asymptotically error-free; we have P err ( N ; q ) = O ( ϵ 2 ) . Therefore, the error is reduced from O ( ϵ ) to O ( ϵ 2 ) but it is not eliminated. More generally, if η j ∝ ( N -j +1) r for a fixed positive integer r , we have P err ( N ; q ) = O ( ϵ r +1 ) . See App. A.2 for a proof and plots.

Next, we present a class of rerankers weaker than Mallows which still lead to error-free protocols.

3.3 Imperfect reranker: Zipf-Mandelbrot model

For Mallows models (using the Kendall-tau distance), the marginal probabilities (5) can be written as η = softmax( -λ z ) , where z = [0 , 1 , ..., N -1] ⊤ . We now consider transformations that yield distributions with heavier tails, which we will see later in §5 to be a better empirical fit in several applications. A known extension to softmax is the γ -entmax (Peters et al., 2019), 5 a family of transformations parametrized by γ ≥ 0 ,

which recovers softmax as a limit case when γ → 1 . In (7), τ is a constant which ensures that γ -entmax( z ) is normalized. When γ > 1 , γ -entmax can return sparse distributions (Blondel et al., 2020). Conversely, when γ < 1 , γ -entmax leads to heavy-tailed distributions (see App. A.3).

Let us now consider η = γ -entmax( -λ z ) , where z = [0 , 1 , ..., N -1] ⊤ , instead of (5). Letting p := 1 / (1 -γ ) , b = λ/p , and a = p + τ λ -1 (where a is seen here as a normalizing constant that replaces τ ), and assuming a > -1 and γ < 1 , we can write the γ -entmax model as η j = b -p ( a + j ) -p . Note that γ < 1 is equivalent to p > 1 . This is called a Zipf-Mandelbrot model (Zipf, 1932; Mandelbrot, 1965). This model generalizes the famous Zipf's law, which applies empirically to many practical contexts, such as the frequency table of words in a corpus of natural language (Powers, 1998). The constant a is determined to satisfy ∑ N j =1 ( a + j ) -p = b p . When N →∞ , the left hand side becomes the Hurwitz zeta function (Hurwitz, 1882), which equals the Riemann's zeta when a = 0 ,

The following result, proved in App. A.4, shows that Zipf-Mandelbrot rerankers (which are weaker than Mallows rerankers and become the latter when γ → 1 ) still ensure error-free protocols. The proof makes use of the integral representation of the Hurwitz zeta function (8) and of the dominated convergence theorem, reusing the result for Mallows models in Proposition 1.

Proposition 2. When R is a Zipf-Mandelbrot reranker, for any λ > 0 and γ < 1 , the protocol is asymptotically error-free.

Fig. 2 (right) shows how this model differs from the one presented in §3.2. Since the reranker is weaker, the error curves bend causing the error decrease to be slower, but still convergent to zero.

4 Generator-Reranker Systems with Dependent Hypotheses

We assume now a more realistic scenario where the independence assumption of §3 might not hold. For example, ( X 1 , Y 1 ) , ..., ( X N , Y N ) might be only exchangeable -this is the case, for example, when the hypotheses are generated from G by sampling from a given model, conditioned on the query. In communication theory parlance, this assumes the presence of channel 'interference' that introduces dependencies between the errors at the various channels, although permuting the messages at each channel does not change the joint distribution. By de Finetti's theorem (Diaconis and Freedman, 1980), exchangeability implies that there is some mixture variable h ∈ H such that P ( x 1: N , y 1: N ) = ∫ H d P ( h ) ∏ N i =1 P ( x i | h ) P ( y i | x i , h ) .

0

50

100

N

3

2

1

0

Log failure rate (wrt baseline)

e

=1.00

e

=0.75

e

=0.50

e

=0.25

e

=0.00

=1.0

=0.5

=0.25

̸

We assume further that h = ( q, τ ) can be decoupled into the query variable q , which conditions x , and a random variable τ , which conditions y , such that P ( x i | h ) := P ( x i | q ) and P ( y i | x i , h ) := P ( y i | x i , τ ) . This corresponds to the graphical model in Fig. 1 (right), including the part in red. We let τ be a continuous random variable in [0 , 1] such that E [ τ ] = ϵ = P ( Y i = X ( q ) | X i ) . A convenient choice is a Beta distribution with parameters α and β , p ( τ ; α, β ) := Γ( α + β ) Γ( α )Γ( β ) τ α -1 (1 -τ ) β -1 .

Perfect reranker and Beta coupling. If R is a perfect reranker, the error probability is

When τ ∼ Beta ( τ ; α, β ) , the N th -raw moment (9) has a closed form, leading to P err ( N ; q ) = ∏ N i =1 α + i -1 α + β + i -1 . The next result, proved in App. A.5 using Gautschi's inequality (Gautschi, 1959) and the Stirling's formula, shows that we still obtain an error-free protocol, albeit the error decays slower than in the independent case-no longer exponentially but rather following a power law.

Proposition 3. When τ ∼ Beta ( τ ; α, β ) and with a perfect reranker, the protocol is error-free and the error probability decays as a power law, P err ( N ; q ) = O ( N -β ) . Furthermore, for β < 1 , we have P err ( N ; q ) ∈ ( Γ( α + β ) Γ( α ) ( α + β + N ) -β , Γ( α + β ) Γ( α ) ( α + β + N -1) -β ) .

Imperfect reranker. When τ ∼ Beta ( τ ; α, β ) , the probability of exactly K out of N messages being corrupted is (due to the conjugacy between the Beta prior and the binomial distribution) ( N K ) ∫ 1 0 dτ p ( τ ; α, β ) τ K (1 -τ ) N -K = ( N K ) ∏ K i =1 ( α + i -1) ∏ N -K i =1 ( β + i -1) ∏ N i =1 ( α + β + i -1) . Therefore, using the reranker marginals η as in (4), we get

which leads to the plot in Fig. 3 for Mallows and Zipf-Mandelbrot models. 6

The next result, proved in App. A.6, shows that the dependencies considered in this subsection do not compromise the error-free protocol when it exists for any density p ( τ ) which is finite in (0 , 1) (not necessarily a Beta distribution). The proof invokes the dominated convergence theorem to enable commuting the limit with the integral sign.

Proposition 4. Let G τ be a generator producing independent hypotheses (§3) where each hypothesis is acceptable with probability 1 -τ . Let the reranker R be such that ( G τ , R ) has error probability P indep err ( N ; q, τ ) → 0 for every τ ∈ (0 , 1) (i.e., it is asymptotically error-free). Assume that the function τ ↦→ P indep err ( N ; q, τ ) is measurable for every N ∈ N . Then, when R is used with a generator G which produces exchangeable hypotheses with arbitrary distribution p ( τ ) , finite in (0 , 1) , the system ( G,R ) is still asymptotically error-free.

This result has important implications: it tells us that, to design error-free protocols, it is sufficient to verify if they are error-free in the simpler case where hypotheses are independent.

5 Experiments

In this section, we demonstrate the validity of our reranking laws on two different tasks: 7 text-to-code generation (§5.1) and machine translation of medical data (§5.2). Following existing literature on scaling laws for language modeling, we fit all curves on the development set using least squares (Ghorbani et al., 2022, App. E) and plot them on the unseen test set. 8 In all cases, we consider the generalized model presented in §4 with parameters α , β , and a Zipf-Mandelbrot reranker with parameters γ , and e -λ , which becomes a Mallows reranker when γ → 1 . This is done in two steps: first, we fit α and β using the data for the perfect reranker ( e -λ = 0 ). Then, we fit γ and e -λ using the already estimated α and β and the data for the imperfect reranker. Our code is available at https://github.com/deep-spin/reranking-laws.

5.1 Code generation

We use a sanitized version of the MBPP dataset (Austin et al., 2021; Liu et al., 2023), a widely used benchmark for evaluating code LLMs, which includes 400 programming problems in Python. For each problem, the dataset includes ground-truth programs and three test cases with input and ground-truth output. We split the dataset in two equally sized parts to get development and test splits.

Wegenerate 200 hypotheses with DeepSeek-Coder 7B (Guo et al., 2024) using a sampling temperature of 1 (see App. B.1 for the prompt template). As in previous work, for simplicity, we use only one test case for each problem (Shi et al., 2022), and select one candidate by taking a majority vote over the

0

50

100

150

200

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

majority voting

oracle

0

50

100

150

200

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

majority voting

oracle

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

execution results, dismissing hypotheses that fail to execute on the test case (Wang et al., 2023). A hypothesis is considered unacceptable if the result of at least one test case (out of three) is different from the ground truth.

Fig. 4 (top) shows the log failure rate on the dev and test sets (left and right, respectively) as a function of N . Even though the oracle fit is not perfect, we get α = . 1 , β = . 309 , γ = . 001 , and e -λ = . 003 for the imperfect reranker with majority voting, which fits the data well, as shown by the red curve.

5.2 Machine translation

We use the TICO-19 dataset (Anastasopoulos et al., 2020), which includes 3071 English sentences in the medical domain ( i.e. , COVID-19 related content) translated into 38 languages. We use the official splits, which contain 971 examples for development and 2100 for testing, focusing on translating from English (EN) to Portuguese (PT), Spanish (ES), and Russian (RU).

For each source sentence, we sample 50 translation hypotheses with a temperature of 1 from TowerInstruct 13B (Alves et al., 2024) using the prompt template in App. B.2. 9 Folowing Farinhas et al. (2023), we consider two reranking strategies: selecting the best candidate with MBRdecoding using COMET-22 as the utility metric (Eikema and Aziz, 2020; Rei et al., 2022a) and reranking based on quality estimation using the reference-free CometKiwi (Fernandes et al., 2022; Rei et al., 2022b). Since we cannot afford to collect human evaluation scores for each sampled hypothesis, we consider a translation to have a critical mistake ( i.e. , to be unacceptable) if its COMET-22 score is below 0 . 85 , and an oracle (perfect) reranker that picks the translation with the highest COMET-22 score.

We follow the described procedure using the data from all language pairs together. Fig. 4 (bottom) shows the log failure rate on the dev and test sets as a function of N . We get α = 0 . 1 and β = 0 . 46 . Additionally, we have γ = 0 . 182 and e -λ = 0 . 001 for MBR decoding and γ = 0 . 001 and e -λ = 0 . 005 for QE reranking. See App. B.2 for additional plots showing these curves when the data

from each language pair is used to fit a separate model. Again, we see a reasonable fit, especially for the imperfect rerankers, with MBR decoding leading to lower failure rates than reranking with QE.

6 Discussion and Related Work

We believe the communication-theoretic perspective introduced in this paper might inspire the design of new protocols for increasing the quality and safety of LLMs. The generator-reranker system studied in this paper bears resemblance with repetition codes, a very naive (and inefficient) class of errorcorrecting codes. Can more powerful designs (Hamming, 1950; Reed and Solomon, 1960; Gallager, 1962; Berrou et al., 1993) inspire more efficient protocols? In machine translation, other forms of adding redundancy, such as lattice generation (Singhal et al., 2023) and hypothesis recombination (Vernikos and Popescu-Belis, 2024), suggest that more efficient designs are indeed possible.

Recent work also suggests that LLM-based evaluators could be used as highly effective rerankers in specific tasks (Kim et al., 2024). While LLMs are not yet ready to fully replace human evaluators across diverse NLP tasks (Bavaresco et al., 2024), in some cases, they can even provide fine-grained assessments in addition to single scores (Kocmi and Federmann, 2023; Fernandes et al., 2023a).

Another class of communication systems allow for feedback , e.g. , in 'automatic repeat request' protocols (Lin et al., 1984), where the receiver has a backchannel to request the sender to retransmit missing bits of information. This framework can be useful to analyze LLM protocols where the generator generates a varying number of hypotheses interactively, relying on feedback from another module, such as a reward model or a confidence estimator, as in Quach et al. (2023). Communication with feedback was also used recently by Jung et al. (2024) for summarization when the generator error probability ϵ is large-our mild conditions for asymptotically error-free protocols (Propositions 1-4) suggest that 'bootstrapping' a correct answer is possible even in scenarios where G is very weak. Additionally, recent work has shown that LLMs may struggle with planning or self-verification, advocating instead for tighter integration between LLMs and external model-based verifiers (Kambhampati et al., 2024). This supports our view that using external feedback models can improve LLMs by enabling interactive, error-correcting communication.

We provide reranking laws , which allow us to predict how many hypotheses are necessary to achieve a desired error probability. This links to a rich body of literature aiming to predict the performance of deep learning models in terms of fundamental parameters, such as the model size or the amount of compute and data used to train them (Hestness et al., 2017, 2019). These so called 'neural scaling laws' have been studied in the context of language modeling (Kaplan et al., 2020; Hoffmann et al., 2022) and machine translation (Ghorbani et al., 2022; Fernandes et al., 2023b), where we observe a power-law scaling for the performance as a function of each fundamental parameter. Our paper complements this line of work by considering the decoding dimension for generator-reranker systems.

The analysis and theoretical results of this paper focus on binary acceptable/unacceptable decisions; however it is possible to extend our framework to consider also continuous quality metrics (such as COMET scores for translation (Rei et al., 2020)) by replacing the notion of 'asymptotically error-free' protocol (Definition 1) by a more general concept associated to a quality target. A possible path is to posit a probability density for the continuous quality metric (instead of a Bernoulli error probability) for each hypothesis coming from the generator, such as a Gaussian or uniform distribution with some input-dependent parameters. For a perfect reranker and independent hypotheses, the resulting output after reranking would be distributed according to the corresponding extreme value distribution (this models the distribution of the highest evaluation metric score among the N hypotheses). Extreme value distributions are an important subject of study in order statistics (David and Nagaraja, 2004) and their densities have closed form expressions in some restricted cases: for example, the Gaussian assumption above yields a Gumbel distribution, and a uniform assumption yields a Beta distribution. The asymptotic case ( N →∞ ) corresponds to one of Gumbel, Fréchet or Weibull families (this is a consequence of the Fisher-Tippett-Gnedenko theorem (David and Nagaraja, 2004)). From the extreme value distribution, we can obtain the expected evaluation metric score or the probability of a quality score being below an acceptable threshold. However, the generalization to imperfect rerankers (such as the Mallows or Zipf-Mandelbrot rerankers described in §3.2 and 3.3) seems harder than in the binary case and requires further investigation.

7 Conclusions

Wepresented a communication-theoretic perspective of generator-reranker LLMs, where the generator G is conceptualized as a sender transmitting N descriptions in parallel through noisy channels, and the reranker R decodes the message by selecting the most appropriate description. Under mild conditions, the combined system ( G,R ) yields an acceptable answer almost surely when N →∞ . Experiments on text-to-code generation and machine translation with LLMs validate our framework.

8 Limitations and Broader Impacts

We regard our paper as a first step connecting communication theory and LLMs, as discussed in §6. However, it should be noted that our work has several limitations. First, the guarantees of error-free protocol in Propositions 1-4 are only asymptotic, and in certain cases a large N may be necessary to achieve a large enough error decrease. We provide convergence rates only for Mallows rerankers (with independent hypotheses and also in the dependent case, when combined with a Beta prior). Second, there is no simple recipe to determine if the Mallows and Zipf-Mandelbrot reranker models are a good empirical fit to concrete rerankers. The same applies to the prior distribution p ( τ ) which makes hypotheses dependent. Third, while our experiments in §5 suggest a reasonable fit in two tasks (code generation and machine translation), the fit is not perfect. A challenge is that, for large N , errors are rare events, and therefore prone to statistical inaccuracies (this is visible in the 'steps' observed in the code generation plots). Finally, although our framework focuses on binary acceptable/unacceptable decisions, it can be extended to continuous evaluation metrics, but this would require modifications to some concepts ( e.g. , the notion of asymptotically error-free protocols). Despite these limitations, the binary case remains highly relevant in practice-for example, in code generation, where the output either executes correctly or it does not. We expect future work to overcome some of these limitations.

In considering the broader impact of our work, it is crucial to acknowledge its early stage and predominantly theoretical nature, which lends the discussion a speculative quality. We believe that our research can significantly enhance the reliability of LLMs by facilitating the identification of potential system failures, holding promise in fields such as natural language processing and computer vision, where robustness and error prediction are paramount. While not directly addressing environmental concerns shared across different LLMs (Strubell et al., 2019), our work could indirectly contribute to energy efficiency efforts by quantifying the efficiency of reranking methods, potentially reducing computational requirements while maintaining requisite quality thresholds during inference.

Acknowledgments

We would like to thank Ben Peters, Duarte Alves, Marcos Treviso, Mário Figueiredo, Sweta Agrawal, and the SARDINE lab team for helpful discussions. This work was supported by EU's Horizon Europe Research and Innovation Actions (UTTER, contract 101070631), by the project DECOLLAGE (ERC-2022-CoG 101088763), by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI), and by Fundação para a Ciência e Tecnologia through contract UIDB/50008/2020.

References

Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. Let's sample step by step: Adaptive-consistency for efficient reasoning and coding with LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12375-12396, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.761. URL https: //aclanthology.org/2023.emnlp-main.761 .

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. Tower: An open multilingual large language model for translation-related tasks, 2024.

Antonios Anastasopoulos, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Franscisco Guzmán, Junjie Hu, Macduff Hughes, Philipp Koehn, Rosie Lazar, Will Lewis, Graham Neubig, Mengmeng Niu, Alp Öktem, Eric Paquin, Grace Tang, and Sylwia Tur. TICO-19: the translation initiative for COvid-19. In Karin Verspoor, Kevin Bretonnel Cohen, Michael Conway, Berry de Bruijn, Mark Dredze, Rada Mihalcea, and Byron Wallace, editors, Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020 , Online, December 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.nlpcovid19-2. 5. URL https://aclanthology.org/2020.nlpcovid19-2.5 .

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.

Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks, 2024. URL https: //arxiv.org/abs/2406.18403 .

Claude Berrou, Alain Glavieux, and Punya Thitimajshima. Near shannon limit error-correcting coding and decoding: Turbo-codes. 1. In Proceedings of ICC'93-IEEE International Conference on Communications , volume 2, pages 1064-1070. IEEE, 1993.

Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew Gormley. It's MBR all the way down: Modern generation techniques through the lens of minimum Bayes risk. In Yanai Elazar, Allyson Ettinger, Nora Kassner, Sebastian Ruder, and Noah A. Smith, editors, Proceedings of the Big Picture Workshop , pages 108-122, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.bigpicture-1.9. URL https://aclanthology.org/2023. bigpicture-1.9 .

Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. Energy-based reranking: Improving neural machine translation using energy-based models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 4528-4537, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.349. URL https://aclanthology.org/2021.acl-long.349 .

Mathieu Blondel, AndrÃ© F.T. Martins, and Vlad Niculae. Learning with fenchel-young losses. Journal of Machine Learning Research , 21(35):1-69, 2020. URL http://jmlr.org/papers/ v21/19-021.html .

Sebastian Borgeaud and Guy Emerson. Leveraging sentence similarity in natural language generation: Improving beam search using range voting. In Alexandra Birch, Andrew Finch, Hiroaki Hayashi, Kenneth Heafield, Marcin Junczys-Dowmunt, Ioannis Konstas, Xian Li, Graham Neubig, and Yusuke Oda, editors, Proceedings of the Fourth Workshop on Neural Generation and Translation , pages 97-109, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.ngt-1.11. URL https://aclanthology.org/2020.ngt-1.11 .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri,

Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.

Flavio Chierichetti, Anirban Dasgupta, Shahrzad Haddadan, Ravi Kumar, and Silvio Lattanzi. Mallows models for top-k lists. Advances in Neural Information Processing Systems , 31, 2018.

Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. Adaptively sparse transformers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2174-2184, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1223. URL https://aclanthology.org/D19-1223 .

Gonçalo Correia, Vlad Niculae, Wilker Aziz, and André Martins. Efficient marginalization of discrete and structured latent variables via sparsity. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 11789-11802. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper/2020/file/887caadc3642e304ede659b734f79b00-Paper.pdf .

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing) . Wiley-Interscience, USA, 2006. ISBN 0471241954.

H.A. David and H.N. Nagaraja. Order Statistics . Wiley Series in Probability and Statistics. Wiley, 2004. ISBN 9780471654018. URL https://books.google.pt/books?id=bdhzFXg6xFkC .

Persi Diaconis and David Freedman. Finite exchangeable sequences. The Annals of Probability , pages 745-764, 1980.

Bryan Eikema and Wilker Aziz. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics , pages 4506-4520, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/ v1/2020.coling-main.398. URL https://aclanthology.org/2020.coling-main.398 .

António Farinhas, José de Souza, and Andre Martins. An empirical study of translation hypothesis ensembling with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 11956-11970, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v1/2023.emnlp-main.733. URL https://aclanthology.org/2023.emnlp-main.733 .

Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1396-1412, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.100. URL https: //aclanthology.org/2022.naacl-main.100 .

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation , pages 1066-1083, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.100. URL https://aclanthology.org/2023.wmt-1.100 .

Patrick Fernandes, Behrooz Ghorbani, Xavier Garcia, Markus Freitag, and Orhan Firat. Scaling laws for multilingual neural machine translation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 10053-10071. PMLR, 23-29 Jul 2023b. URL https://proceedings.mlr. press/v202/fernandes23a.html .

A. Fligner and J. S. Verducci. Distance based ranking models. Journal of the Royal Statistical Society. Series B (Methodological) , 48(3):359-369, 1986. ISSN 00359246. URL http://www. jstor.org/stable/2345433 .

Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics , 10:811-825, 2022.

Robert Gallager. Low-density parity-check codes. IRE Transactions on information theory , 8(1): 21-28, 1962.

A.E. Gamal and T. Cover. Achievable rates for multiple descriptions. IEEE Transactions on Information Theory , 28(6):851-857, 1982. doi: 10.1109/TIT.1982.1056588.

Walter Gautschi. Some elementary inequalities relating to the gamma and incomplete gamma function. J. Math. Phys , 38(1):77-81, 1959.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics , 9:346-361, 04 2021. ISSN 2307-387X. doi: 10.1162/tacl_a_00370. URL https://doi.org/10.1162/tacl_a_00370 .

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id= hR_SMu8cxCV .

Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. Hallucinations in Large Multilingual Translation Models. Transactions of the Association for Computational Linguistics , 11:1500-1517, 12 2023. ISSN 2307-387X. doi: 10.1162/tacl_a_00615. URL https://doi.org/10.1162/tacl_a_00615 .

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming - the rise of code intelligence, 2024.

Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal , 29(2):147-160, 1950.

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.

Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: computational challenges in deep learning. In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming , PPoPP '19, page 1-14, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362252. doi: 10.1145/3293883.3295710. URL https: //doi.org/10.1145/3293883.3295710 .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=iBBcRUlOAPR .

James Y Huang, Sailik Sengupta, Daniele Bonadiman, Yi-an Lai, Arshit Gupta, Nikolaos Pappas, Saab Mansour, Katrin Kirchoff, and Dan Roth. Deal: Decoding-time alignment for large language models. arXiv preprint arXiv:2402.06147 , 2024.

Chia-Chien Hung, Wiem Ben Rim, Lindsay Frost, Lars Bruckner, and Carolin Lawrence. Walking a tightrope - evaluating large language models in high-risk domains. In Dieuwke Hupkes, Verna Dankers, Khuyagbaatar Batsuren, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell, and Elia Bruni, editors, Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP , pages 99-111, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.genbench-1.8. URL https://aclanthology.org/2023.genbench-1.8 .

Adolf Hurwitz. Einige eigenschaften der dirichlet'schen funktionen, die bei der bestimmung der klassenanzahlen binärer quadratischer formen auftreten. Zeitschrift fur Mathematik und Physik , 1882.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv. , 55(12), mar 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https: //doi.org/10.1145/3571730 .

Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation: from low-quality model to high-quality dataset & model for summarization and paraphrasing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2024.

Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B Murthy. Position: LLMs can't plan, but can help planning in LLM-modulo frameworks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 22895-22907. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr. press/v235/kambhampati24a.html .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing finegrained evaluation capability in language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=8euJaTveKw .

Alexandre Klementiev, Dan Roth, and Kevin Small. Unsupervised rank aggregation with distancebased models. In Proceedings of the 25th international conference on Machine learning , pages 472-479, 2008.

Alexandre Klementiev, Dan Roth, Kevin Small, and Ivan Titov. Unsupervised rank aggregation with domain-specific expertise. In Twenty-First International Joint Conference on Artificial Intelligence , 2009.

Tom Kocmi and Christian Federmann. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation , pages 768-775, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.wmt-1.64. URL https: //aclanthology.org/2023.wmt-1.64 .

Shankar Kumar and William Byrne. Minimum Bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002) , pages 140-147. Association for Computational Linguistics, July 2002. doi: 10.3115/1118693.1118712. URL https://aclanthology.org/W02-1019 .

J.N. Laneman, E. Martinian, G.W. Wornell, and J.G. Apostolopoulos. Source-channel diversity for parallel channels. IEEE Transactions on Information Theory , 51(10):3518-3539, 2005. doi: 10.1109/TIT.2005.855578.

Guy Lebanon and Yi Mao. Non-parametric modeling of partially ranked data. Journal of Machine Learning Research , 9(79):2401-2429, 2008. URL http://jmlr.org/papers/v9/lebanon08a. html .

Haau-Sing Li, Patrick Fernandes, Iryna Gurevych, and André F. T. Martins. Doce: Finding the sweet spot for execution-based code generation, 2024. URL https://arxiv.org/abs/2408.13745 .

Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Cheung, and Siva Reddy. Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022 , pages 926-937, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.75. URL https://aclanthology.org/2022.findings-acl.75 .

Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns , 5(3):100943, 2024. ISSN 2666-3899. doi: https://doi.org/10.1016/j.patter.2024.100943. URL https://www.sciencedirect.com/ science/article/pii/S2666389924000424 .

Shu Lin, Daniel J Costello, and Michael J Miller. Automatic-repeat-request error-control schemes. IEEE Communications magazine , 22(12):5-17, 1984.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and LINGMING ZHANG. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https: //openreview.net/forum?id=1qvx610Cu7 .

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models. arXiv preprint arXiv:2402.02992 , 2024.

David J. C. MacKay. Information Theory, Inference & Learning Algorithms . Cambridge University Press, USA, 2002. ISBN 0521642981.

Benoît Mandelbrot. Information theory and psycholinguistics. BB Wolman and E , 1965.

André Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. Sparse and continuous attention mechanisms. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 20989-21001. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/f0b76267fbe12b936bd65e203dc675c1-Paper.pdf .

André F. T. Martins, Marcos Treviso, António Farinhas, Pedro M. Q. Aguiar, Mário A. T. Figueiredo, Mathieu Blondel, and Vlad Niculae. Sparse continuous distributions and fenchel-young losses. Journal of Machine Learning Research , 23(257):1-74, 2022. URL http://jmlr.org/papers/ v23/21-0879.html .

Pedro Henrique Martins, Vlad Niculae, Zita Marinho, and André F. T. Martins. Sparse and structured visual attention. In 2021 IEEE International Conference on Image Processing (ICIP) , pages 379-383, 2021. doi: 10.1109/ICIP42928.2021.9506028.

Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-Tau Yih, Sida Wang, and Xi Victoria Lin. LEVER: Learning to verify language-to-code generation with execution. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 26106-26128. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/ni23b.html .

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.

Ozarow. On a source-coding problem with two channels and three receivers. The Bell System Technical Journal , 59(10):1909-1921, 1980. doi: 10.1002/j.1538-7305.1980.tb03344.x.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.168. URL https://aclanthology.org/2021.naacl-main.168 .

Ben Peters and André F. T. Martins. Smoothing and shrinking the sparse Seq2Seq search space. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy,

Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2642-2654, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.210. URL https://aclanthology.org/2021.naacl-main.210 .

Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequence models. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 1504-1519, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1146. URL https: //aclanthology.org/P19-1146 .

David MW Powers. Applications and explanations of zipf's law. In New methods in language processing and computational natural language learning , 1998.

Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations , 2023.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Irving S Reed and Gustave Solomon. Polynomial codes over certain finite fields. Journal of the society for industrial and applied mathematics , 8(2):300-304, 1960.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2685-2702, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.213. URL https://aclanthology.org/2020. emnlp-main.213 .

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. URL https://aclanthology.org/2022.wmt-1.52 .

Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. URL https://aclanthology.org/2022.wmt-1.60 .

Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3533-3546, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.231. URL https://aclanthology.org/2022.emnlp-main.231 .

Prasann Singhal, Jiacheng Xu, Xi Ye, and Greg Durrett. Eel: Efficiently encoding lattices for reranking. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 9299-9316, 2023.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David Traum, and Lluís Màrquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3645-3650,

Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1355. URL https://aclanthology.org/P19-1355 .

Wenpin Tang. Mallows ranking models: maximum likelihood estimate and regeneration. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 6125-6134. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/tang19a.html .

Giorgos Vernikos and Andrei Popescu-Belis. Don't rank, combine! combining machine translation hypotheses using quality estimation. arXiv preprint arXiv:2401.06688 , 2024.

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents , 2024. URL https://openreview.net/forum?id=8oJyuXfrPv .

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=1PL1NIMMrw .

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems , 36, 2024.

George Kingsley Zipf. Selected studies of the principle of relative frequency in language . Harvard university press, 1932.

A Proofs and Visualizations

A.1 Proof of Proposition 1

Let λ ϵ := -log ( e -λ (1 -ϵ ) + ϵ ) and define F ( N ) = log P err ( N ; q ) = e -λϵN -e -λN 1 -e -λN . Observe that 0 < λ ϵ < λ for any λ > 0 and ϵ ∈ (0 , 1) . We extend the domain of F to the real numbers in [1 , + ∞ ) . We will prove that F ( N ) is decreasing and that lim N →∞ F ′ ( N ) = -λ ϵ . This shows that P err ( N ; q ) → 0 at asymptotic rate e -λ ϵ . We have

hence F ( N ) is decreasing. Since the second term tends to zero, the limit is given by the first term:

where we used the fact that e ( -λ + λ ϵ ) N → 0 and e ( -λ ϵ + λ ) N → + ∞ . This proves the desired claim, that is, the error probability decreases exponentially fast with rate e -λ ϵ . Note that, for a perfect reranker ( λ →∞ ), we get e -λ ϵ = ϵ and we recover the rate ϵ N seen in §3.1.

A.2 Proof of Example 1

Wefirst provide a proof for r = 1 . Wehave ∑ N j = N -K +1 η j = ∑ K j =1 η N -K + j = ∑ K j =1 j ∑ N j =1 j = K ( K +1) N ( N +1) . Plugging this into Eq. (4), we obtain

where B ( N,ϵ ) denotes the binomial distribution with parameters N and ϵ and we use the facts that E K ∼ B ( N,ϵ ) [ K ] = Nϵ and E K ∼ B ( N,ϵ ) [ K 2 ] = Nϵ (1 -ϵ ) + N 2 ϵ 2 . Therefore, we obtain lim N →∞ P err ( N ; q ) = ϵ 2 .

We now prove the general case r ≥ 1 . From Faulhaber's formula, we have ∑ K j =1 j r = 1 r +1 ∑ r j =0 ( r +1 j ) B j K r -j +1 , where B j = ∑ j ℓ =0 1 ℓ +1 ∑ ℓ m =0 ( ℓ m ) ( -1) m ( m + 1) j denotes the j th Bernoulli number. Therefore, we get

Plugging this into Eq. (4), we obtain

We now use the fact that the raw moments of the binomial distribution B ( N,ϵ ) are given by E K ∼ B ( N,ϵ ) [ K m ] = ∑ m ℓ =0 { m ℓ } N ℓ ϵ ℓ , where { m ℓ } := 1 ℓ ! ∑ ℓ i =0 ( -1) ℓ -i ( ℓ i ) i m are the Stirling numbers of the second kind, and N ℓ := N ! ( N -ℓ )! is the ℓ th falling power of N . Therefore, when N →∞ , (13) becomes

The plots in Fig. 5 show examples for several values of r .

0

100

200

N

0.4

0.5

0.6

0.7

0.8

Perr

0

100

200

N

0

100

200

N

A.3 Entmax

When γ > 1 , γ -entmax can return sparse distributions (Blondel et al., 2020). This case has been extensively studied as a way to, e.g. , filter large output spaces (Correia et al., 2020; Peters and Martins, 2021) or to produce more interpretable predictions (Correia et al., 2019; Martins et al., 2020, 2021, 2022). Conversely, when γ < 1 , γ -entmax leads to distributions with heavier tails, which is the case of our interest, as described in §3.3. See Fig. 6 for an illustration of γ -entmax for different values of γ in the two-dimensional case. For γ > 1 , all mappings saturate at z = ± 1 /γ -1 ; this does not happen for γ ≤ 1 .

A.4 Proof of Proposition 2

Note that we can write

and

2

0

2

z

0.0

0.5

1.0

=0.25

=0.5

=1.0 (softmax)

=1.5

=2.0

The error probability is

Since a is the normalizing constant such that 1 = ζ ( p, a + 1) = 1 b p Γ( p ) ∫ ∞ 0 dt t p -1 e ( a +1) t (1 -e -t ) (cf. Eq. (8)), we can interpret the expression above as the expectation of f N ( t ) := ((1 -ϵ ) e -t + ϵ ) N -e -tN under the probability distribution on (0 , ∞ ) with density π ( t ) := 1 b p Γ( p ) t p -1 e ( a +1) t (1 -e -t ) . Since f N ( t ) → 0 pointwise for t ∈ ]0 , ∞ [ and it is bounded in that interval, we can invoke the dominated convergence theorem to commute the limit and integral sign. We then have that P err ( N ; q ) → 0 .

A.5 Proof of Proposition 3

Let us consider first the case where β = 1 . Then,

Now consider the case where β > 1 . We have for each term in the product that α + i -1 α + β + i -1 < α + i -1 α + i , hence we must have P err ( N ; q ) < α α + N . Since the sequence is positive (since all terms are positive) and decreasing (since all terms are < 1 ), we must also have P err ( N ; q ) → 0 when β > 1 .

Finally, let us analyze the case where 0 < β < 1 . From (9), we have

We invoke Gautschi's inequality, which states that x 1 -s < Γ( x +1) Γ( x + s ) < ( x + 1) 1 -s for any x and s ∈ (0 , 1) . We set s := 1 -β and x := α + β + N -1 , from which we obtain the desired result.

To show that the error probability decays as a power law for any β > 0 , we use Stirling's formula, which states that

Therefore,

A.6 Proof of Proposition 4

Let P indep err ( N ; q, τ ) denote the error probability of the generator-reranker system when the hypotheses are independent and each hypothesis produced by G has error probability τ . The error probability of the generator-reranker system with exchangeable hypotheses is given by

Therefore, lim N →∞ P err ( N ; q ) = lim N →∞ ∫ 1 0 dτ p ( τ ) P indep err ( N ; q, τ ) . Since P indep err ( N ; q, τ ) ∈ [0 , 1] for any N ∈ N and τ ∈ [0 , 1] , we have that p ( τ ) P indep err ( N ; q, τ ) ∈ [0 , p ( τ )] , and therefore the integrand is bounded by p ( τ ) , which integrates to 1 . Therefore we can invoke the dominated convergence theorem and switch the limit and integral signs. Since by assumption lim N →∞ P indep err ( N ; q, τ ) = 0 pointwise for any τ ∈ (0 , 1) , we obtain lim N →∞ P err ( N ; q ) = ∫ 1 0 dτ p ( τ ) lim N →∞ P indep err ( N ; q, τ ) = 0 .

B Experimental Details

B.1 Text-to-code generation

Licenses. We use DeepSeek-Coder 7B (Guo et al., 2024), which is available under a permissive license that allows for both research and unrestricted commercial use. We report results on the MBPP dataset (Austin et al., 2021; Liu et al., 2023), released under an Apache license.

Prompt template. We generate hypotheses with DeepSeek-Coder 7B (Guo et al., 2024) using the following prompt template:

You are an AI programming assistant, utilizing the DeepSeek Coder model, developed by DeepSeek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer.

### Instruction:

Please complete the following Python function in a markdown style code block: ‵‵‵

python

[prompt]

‵‵‵

### Response:

‵‵‵ python

MBR-exec. We use MBR-exec, an approach proposed by Shi et al. (2022) that consists of (1) sampling programs from an LLM, (2) executing each program on one test case, and (3) selecting the example with the minimal execution result-based Bayes risk. We use a 0/1 matching loss between execution results, and the Bayes risk of a program is defined by the sum of the loss between itself and the other sampled programs (the ground-truth program output is not used). We break ties by selecting the program with the smallest sampling index, corresponding to a random selection. See Shi et al. (2022, Section 3) for more details.

B.2 Machine translation

Licenses. We use TowerInstruct 13B (Alves et al., 2024), which is released under a CC-BY-NC-4.0 license. We report results on the TICO-19 dataset (Anastasopoulos et al., 2020), publicly available through a Creative Commons CC0 license.

Prompt template. We generate hypotheses with TowerInstruct 13B (Alves et al., 2024) using the following prompt template:

Visualizations. In §5.2 we obtained a single reranking law for the all language pairs; we now fit different models for each language pair. Fig. 7 shows the log failure rate on the dev and test sets as a function of N for EN-PT, EN-ES, and EN-RU. While the fits on the dev set are good, there is some degradation on the test set, especially for EN-ES (oracle and MBR decoding), possibly due to a shift in the distribution of scores/errors. We leave the investigation of more robust techniques and how to adapt to these cases for future work.

B.3 Mathematical and commonsense reasoning

Our approach is fully general and can be useful in other domains other than code and language generation. In this subsection, we present additional experiments on mathematical and commonsense reasoning benchmarks, as prior work has shown that generating multiple hypotheses as an intermediate step is also advantageous in these scenarios (Wang et al., 2023).

We use samples generated by Aggarwal et al. (2023) with code-davinci-002, a GPT-3-based model with 175 billion parameters (Brown et al., 2020) which is part of the Codex series (Chen et al., 2021) (please refer to their Section 4 for more details; these samples were made publicly available by the authors at https://github.com/Pranjal2041/AdaptiveConsistency). We apply self-consistency over diverse reasoning paths (Wang et al., 2023), selecting the most frequent answer in the candidate set, and report results on the SV AMP (Patel et al., 2021) and StrategyQA (Geva et al., 2021) datasets. Following §5.1, we split the datasets in two equally sized parts to get development and test splits.

Similarly to Fig. 4, Fig. 8 shows the log failure rate on the dev and test sets (left and right, respectively) as a function of N , confirming that the same trends hold also for these two additional tasks.

B.4 Computing infrastructure

Our insfrastructure consists of 2 machines, each equipped with 8 NVIDIA RTX A6000 GPUs (46GB) and 12 Intel Xeon Gold 6348 CPUs (2.60GHz, 1TB RAM). The machines were used interchangeably, and all experiments were executed on a single GPU.

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

N

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

0

10

20

30

40

50

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

mbr

oracle

N

N

0

10

20

30

40

N

2.5

2.0

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

self-consistency

oracle

0

10

20

30

40

N

2.5

2.0

1.5

1.0

0.5

0.0

Log failure rate (wrt baseline)

self-consistency

oracle

0

16

32

48

64

N

6

5

4

3

2

1

0

Log failure rate (wrt baseline)

self-consistency

oracle

0

16

32

48

64

N

6

5

4

3

2

1

0

Log failure rate (wrt baseline)

self-consistency

oracle

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and the introduction state our assumptions and contributions. We summarize them in bullet points in §1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in §8.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state the full set of assumptions of our theoretical results (Propositions 1-4) and include complete proofs in App. A. We provide short proof sketches in the main paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the information needed to reproduce our results in §5 and App. B. Our code is available at https://github.com/deep-spin/reranking-laws.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly available data in our experiments (although we are not introducing new datasets). Our code is available at https://github.com/deep-spin/rerankinglaws, with a CC-BY 4.0 license.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the details in §5 and App. B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: While we do not report error bars, for machine translation, we consider multiple language pairs and report individual and combined results (see §5.2 and App. B.2).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the type of computing resources used in our experiments in App. B.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact of our work in §8.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the creators of the datasets/models used in our paper, including the name of the corresponding licenses in App. B.1 and App. B.2.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is available at https://github.com/deep-spin/reranking-laws, with a CC-BY 4.0 license.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a reranking principle for language generation from a communication-theoretic perspective.  The paper conceptualizes the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. A receiver is designed to decode the message by ranking the descriptions and selecting the one found to be most reliable. Experiments show the effectiveness of proposed method in text-to-code generation task and machine translation of medical data task.

STRENGTHS:
1 This paper proposes a reranking principle for language generation from a communication-theoretic perspective. The motivation is interesting and the theoretic analysis seems reasonable.

2 The paper is well-written and easily readable.

WEAKNESSES:
1 The related work analysis is not comprehensive. There are several ranking and reranking works in recommended systems, and none of them is mentioned or compared in this paper.

2 The experiments are not convincing enough. This paper only conducts two downstream experiments, i.e., Code generation and Machine translation. The results should be evaluated through more common and popular downstream tasks, such as QA (question choice) scenarios.

3 There is only 1 baseline for the text-to-code generation task and 2 baselines for the machine translation task. In particular, the one baseline for the text-to-code generation task is majority voting, which is not representative enough.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA number of recent works in language generation can be framed as proposing two step methods, with a method to generate proposal strings, and another to rank these strings before choosing the best one to be output (this includes, e.g., MBR decoding).\nThis paper analyses this practice with a communication-theoretic approach.\nThey first assume that, given a query $q$, the goal of decoding is to generate a string $y$ in a set $\\mathcal{X}(q)$.\nThey then assume the generator’s output $p(y_{1:N} \\mid q)$ can be decomposed as $p(x_{1:N} \\mid q)p(y_{1:N} \\mid x_{1:N})$, with $x_n \\in\\mathcal{X}(q)$ and $p(y_{1:N} \\mid x_{1:N})$ representing some kind of noise perturbation. \nThey then analyse the probability $p(rank(y_{1:N}) \\notin\\mathcal{X}(q)  \\mid q)$ under different assumptions, showing that for many this value goes to zero as $N$ goes to infinity.\nFinally, they run two experiments showing their theoretically derived predictions $p(rank(y_{1:N}) \\notin\\mathcal{X}(q)  \\mid q)$ seem to correlate with the empirical probability of decoding errors.\n\nSTRENGTHS:\nThe paper is well written and in general easy to follow (although I think section 4 could use a bit more hand-holding).\n\nThe paper provides an interesting theoretical analysis to a widely popular text generation framework.\n\nThe paper then investigates whether these theoretical insights are reflected empirically in real decoding settings.\n\nWEAKNESSES:\nIn general, I liked this paper. In my opinion however, its main weaknesses are:\n* limited empirical evaluation, with only two tasks, one generator model, and two reranking methods (besides an oracle ranker).\n* the evaluation also makes some (in my opinion) debatable claims. E.g., in line 236 the authors state “[...] the imperfect reranker with majority voting, which fits the data well, as shown by the red curve.”. However, analysing Fig 4 (top), I would argue that the solid lines do not capture the data behaviour that well. In fact, the model’s performance seems to be empirically close to convergence with N, but the solid lines go monotonically down. Maybe running this analysis for larger values of $N$ would show whether the data indeed fits the predictions (specially if the predicted power law would generalise to larger values of $N$ as fit in the current data, and without fitting it on the new results).\n\nQUESTIONS:\nIn the code generation experiments, the paper says “we use only one test case for each problem (Shi et al., 2022), and select one candidate by taking a majority vote over the execution results, dismissing hypotheses that fail to execute on the test case.” If a single test case is used, how is majority voting performed exactly? More details here could be helpful.\n\n\nAs a minor suggestion: I found the use of a “communication theoretical” framing here a bit distracting, and it seems to me it could be discarded with no significant change to the paper’s contributions. The authors, for instance, discuss error correcting codes early in the paper, but then they (admittedly) do not require generated strings to be error-corrected. (The selected string simply needs to be in an acceptable set $rank(y_{1:N}) \\notin\\mathcal{X}(q) $.) Besides, the generator and ranker are framed as a sender and a receiver—with a noisy channel in between them—but no message is actually decoded by the receiving ranker. Alternatively, highlighting the role that a communication-theoretical framing has in the paper (and why it is needed) could be useful.\n\nLIMITATIONS:\nThe authors properly discuss their analysis limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes to regard generator-reranker LLMs, i.e., LLMs generating multiple outputs and then reranking them, as communication systems. The idea is to consider the outputs noisy with the objective for the reranker to find the less noisy one.

STRENGTHS:
- the approach is very flexible. It doesn’t depend on a particular architecture and the outputs to rerank can be generated by multiple different models.
- a sound parallel is made with communication theory which helps to understand why this approach works.
- the approach is well formalized. I couldn’t find any error but be aware that I’m not very familiar with Zipf-Mandelbrot and Mallows model.
- Two scenarios are taken into account: with and without an independence assumption
- Experiments with machine translation are very relevant for generator-reranker LLMs

WEAKNESSES:
- Absence of analysis of the experiment results. This is a critical weakness of the paper. The experimental settings are described, and the results are given (plots), but without any comment. For the MT experiments, the authors wrote that they got some scores, and then we have the next section.

QUESTIONS:
Please comment on your results. What do you conclude from them? Why are they insightful (or not insightful)? How can they be used in future work? etc.
The paper is really good but we miss some analysis.

LIMITATIONS:
Limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a framework for understanding the theoretical properties of generator-reranker systems for language generation. It relates the reranking process to error correction during the decoding of messages in noisy channels, a concept that has been well-studied in communication theory. Explicitly, the paper conceptualizes an LLM generator as a sender transmitting messages through noisy channels, with the reranker acting as the receiver decoding these messages. This framing explains why 1) things like redundancy in the set of generated strings are helpful in generator-reranker systems, actually increasing the likelihood of an acceptable output and 2) increasing the number of options from the generator in the reranking process generally increases system performance. The paper makes several theoretical contributions, showing that when generator-reranker systems meet certain theoretical requirements, there is a guarantee of “error-free” performance. This property holds even when channel distributions are not independent, i.e., when the same model is used to generate possible solutions to the input. The paper provides some empirical verification of their proposed laws.\n\nSTRENGTHS:\nThe paper is very well written. The math is clearly explained and sound; it provides a nice theoretical justification of why generator-reranker systems work well  The topic is also very relevant, since LLM reliability (which is improved by the generator-reranker paradigm) is of utmost concern. The laws proposed in this paper also have practical use: they would allow practitioners to decide the number of strings needed from the generator system to achieve a certain accuracy, without lots of trial and error.\n\nWEAKNESSES:\nThe applicability of the communication system framework to generator-reranker systems is somewhat questionable given that there is not the same binary notion of acceptable/unacceptable for language generation systems. Rather, we’re dealing with a continuous spectrum of quality and the appeal of the generator-reranker system is its ability to increase quality (perhaps amongst “acceptable” solutions) rather than move from the realm of unacceptable to acceptable answers. The impact of this difference between the theoretical framework and the evaluation of generation systems in practice isn’t really discussed. \nThere are a few points that could be addressed to improve readability:\n* Some aspects of the abstract/intro are confusing because terms have not been defined and their equivalences in an LLM generator-reranker system have not yet been specified. For example, the reader won’t know what the implications of “channel distributions being statistically dependent” are (mentioned in the abstract) until after they’ve read through much of the paper.\n* There isn’t much intuition about what the R.V. X corresponds to in the generator-reranker system \n* Some more intuition behind the scale parameter (other than just the settings that it gives for its extreme values) would be helpful\n\n\nThe computational experiments are not very comprehensive, exploring only two generation tasks (one generator model for each task). It is thus unclear how general their results are.\n\nQUESTIONS:\n* I didn't understand the (bolded) comment in lines 148-9. It makes it sound as though the quality of the reranker depends on the quality of the generator. Could this be clarified?\n* Minor style recommendation: Perhaps move the first sentence of 3.3 to right after providing the expression for the partition function. That feels like a more natural place to me. In footnote 1: equivalent class -> equivalence class\n\nLIMITATIONS:\nThe authors discuss most of the limitations present in their work. I would like to see an additional discussion of the implications of their results for continuous evaluation metrics (rather than binary acceptable/unacceptable)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}