Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs

## Abstract

Abstract Large Language Models (LLMs) generate text by sampling the next token from a probability distribution over the vocabulary at each decoding step. However, popular sampling methods like top- p 𝑝 p italic_p (nucleus sampling) often struggle to balance quality and diversity, especially at higher temperatures, leading to incoherent or repetitive outputs. To address this challenge, we propose min- p 𝑝 p italic_p sampling , a dynamic truncation method that adjusts the sampling threshold based on the model’s confidence by scaling according to the top token’s probability. We conduct extensive experiments on benchmarks including GPQA, GSM8K, and AlpacaEval Creative Writing, demonstrating that min- p 𝑝 p italic_p sampling improves both the quality and diversity of generated text, particularly at high temperatures. Moreover, human evaluations reveal a clear preference for min- p 𝑝 p italic_p sampling in terms of both text quality and diversity. Min- p 𝑝 p italic_p sampling has been adopted by multiple open-source LLM implementations, highlighting its practical utility and potential impact.

## 1 Introduction

Large Language Models (LLMs) have achieved remarkable success in generating coherent and creative text across diverse domains, from factual question answering to open-ended storytelling. A central challenge in these generative tasks is managing the trade-off between creativity and coherence, often influenced by the sampling strategy used during text generation. Popular methods like top-$p$ sampling (nucleus sampling) (Holtzman et al., ) and temperature scaling (Ackley et al., ) are widely adopted to address this challenge, but they often struggle, especially at higher temperatures. While increasing temperature can enhance diversity, it frequently reduces the coherence of generated text; conversely, more conservative sampling limits creativity and can lead to repetitive outputs.

This challenge becomes particularly critical when LLMs are used in tasks that require both imaginative and contextually grounded responses. In this paper, we address this fundamental issue by introducing a new sampling method called min-$p$ sampling, designed to dynamically balance creativity and coherence, even at high temperatures. Min-$p$ sampling establishes a minimum base probability threshold that scales according to the top token’s probability, allowing it to dynamically include diverse options when the model is uncertain while focusing on high-confidence tokens when the model is confident.

To demonstrate the effectiveness of min-$p$, we conduct extensive experiments on various benchmark datasets, including GPQA (Rein et al., ), GSM8K (Cobbe et al., ), and AlpacaEval Creative Writing (Li et al., ). Our results show that min-$p$ sampling outperforms top-$p$ and other popular decoding methods such as top-$k$ (Fan et al., ) and $\eta$-sampling (Hewitt et al., ), maintaining coherence while allowing for increased diversity, particularly with high-temperature scaling. We also conducted a comprehensive human evaluation to compare the quality and diversity of text generated by min-$p$ with that generated by traditional sampling methods. The results indicate a clear preference for min-$p$, with participants rating min-$p$ outputs as superior in quality and diversity.

The contributions of this paper are as follows:

- •
We introduce min-$p$ sampling, a novel dynamic truncation method that effectively balances creativity and coherence in LLM-generated text, particularly at high temperatures.
- •
We present comprehensive experimental results on several benchmarks, demonstrating that min-$p$ consistently improves the quality and diversity of generated text compared to top-$p$ sampling and other existing methods.
- •
We validate the practical utility of min-$p$ through an extensive human evaluation, showing that human evaluators prefer min-$p$ outputs over those generated by other methods in terms of both quality and diversity.
- •
We provide practical empirical guidelines for using min-$p$ sampling, assisting practitioners in selecting appropriate hyperparameters and best practices for various applications.
- •
The rapid adoption of min-$p$ by the open-source LLM community further highlights its effectiveness and practical potential.

By introducing min-$p$ and offering empirical guidelines for its use, we aim to explore high-temperature settings for creative text generation without compromising coherence. Our results demonstrate that min-$p$ is a viable and superior alternative to existing sampling methods, both at standard and high-temperature settings, making it an important contribution to generative language modeling.

## 2 Related Work

Sampling methods are crucial in controlling the quality and diversity of text generated by LLMs. The choice of sampling strategy directly affects the balance between creativity and coherence, which is critical in many generative tasks. In this section, we review existing sampling methods and their limitations, establishing the motivation for our proposed min-$p$ sampling approach.

##### Greedy Decoding and Beam Search.

Greedy decoding and beam search are deterministic decoding strategies that select the token with the highest probability at each step (Freitag & Al-Onaizan, ). While these methods ensure high-probability token selection, they often lead to repetitive and generic text due to their lack of diversity. Beam search also incurs a significant runtime performance penalty.

##### Stochastic Sampling Methods.

Stochastic sampling methods aim to inject diversity into the generated text by introducing randomness in token selection. Temperature scaling adjusts the distribution’s sharpness, balancing diversity and coherence (Ackley et al., ); however, higher temperatures often lead to incoherent and nonsensical results, limiting its applicability. Top-$k$ sampling selects from the top $k$ most probable tokens, ensuring that only high-probability tokens are considered (Fan et al., ). While it offers a simple way to prevent unlikely tokens from being sampled, it does not adapt dynamically to varying confidence levels across different contexts.

Top-$p$ sampling, also known as nucleus sampling, restricts the token pool to those whose cumulative probability exceeds a predefined threshold $p$ (Holtzman et al., ). This method effectively balances quality and diversity by focusing on the "nucleus" of high-probability tokens and dynamically adapts to different contexts. However, at higher temperatures, top-$p$ sampling can still allow low-probability tokens into the sampling pool, leading to incoherent outputs. This trade-off between creativity and coherence at high temperatures is a key limitation that we aim to address with min-$p$ sampling.

##### Entropy-Based Methods.

Recent work has introduced methods such as entropy-dependent truncation ($\eta$-sampling) and mirostat sampling, which attempt to dynamically adjust the sampling pool based on the entropy of the token distribution (Hewitt et al., ; Basu et al., ; Shwartz Ziv & LeCun, ). While entropy/uncertainty-based approaches show promise in improving text quality, they often require complex parameter tuning and are computationally expensive, making them challenging to use in practical applications. We detail our experimental challenges running $\eta$ sampling in Section B.2.

## 3 Min- p 𝑝 p italic_p Sampling

The core idea of min-$p$ sampling is to dynamically adjust the sampling threshold based on the model’s confidence at each decoding step. This dynamic mechanism allows the sampling process to be sensitive to the context and the certainty of the model, providing a better balance between creativity and coherence, especially at high temperatures.

### 3.1 Overview of Min- p 𝑝 p italic_p Sampling

In standard autoregressive generation, a language model predicts the probability distribution over the vocabulary for the next token, conditioned on the sequence generated so far. At each step, the model selects a token from this distribution either deterministically or stochastically. Min-$p$ sampling is a stochastic method that adapts its truncation threshold based on the model’s confidence, allowing the sampling strategy to be context-sensitive.

Formally, at each time step $t$, let $\mathcal{V}$ denote the vocabulary, and $P(x_{t}\mid x_{1:t-1})$ represent the conditional probability distribution over the vocabulary for the next token $x_{t}$. Min-$p$ sampling involves the following steps:

- 1.
Calculate the Maximum Probability: Identify the maximum probability token in the distribution, denoted as $p_{\max}=\max_{v\in\mathcal{V}}P(v\mid x_{1:t-1})$.
- 2.
Define the Truncation Threshold: Set a base probability threshold, $p_{\text{base}}\in(0,1]$, and scale it by $p_{\max}$ to determine the actual truncation threshold:
$p_{\text{scaled}}=p_{\text{base}}\times p_{\max}$
(1)
This threshold ensures that tokens with sufficiently high relative probabilities are considered while filtering out less probable tokens in a context-dependent manner.
- 3.
Define the Sampling Pool: Construct the sampling pool $\mathcal{V}_{\text{min}}$ consisting of tokens whose probabilities are greater than or equal to $p_{\text{scaled}}$:
$\mathcal{V}_{\text{min}}=\{v\in\mathcal{V}:P(v\mid x_{1:t-1})\geq p_{\text{
scaled}}\}$
(2)
- 4.
Sample from the Pool: Sample the next token $x_{t}$ from the reduced set $\mathcal{V}_{\text{min}}$ according to their normalized probabilities:
$P^{\prime}(v)=\frac{P(v\mid x_{1:t-1})}{\sum_{v^{\prime}\in\mathcal{V}_{\text{
min}}}P(v^{\prime}\mid x_{1:t-1})}\quad\text{for }v\in\mathcal{V}_{\text{min}}$
(3)

### 3.2 Intuition Behind Min- p 𝑝 p italic_p Sampling

The key intuition behind min-$p$ sampling is that token truncation thresholds are relative and depend on how certain the distribution is for that token, and not absolute thresholds. When the model is highly confident about the next token (i.e., $p_{\max}$ is high), min-$p$ restricts the pool to high-probability candidates to maintain coherence. Conversely, when the model is less confident, relaxing the sampling pool allows for a more creative and diverse generation. Unlike top-$p$ sampling, which truncates the distribution based on a fixed cumulative probability, min-$p$ dynamically adjusts the threshold based on the model’s confidence, leading to more context-sensitive generation.

Figure: (a)
Refer to caption: x1.png

Figure 1 illustrates the effects of different sampling methods, including min-$p$, on token probability distributions. In subfigure (a), we show an initial probability distribution over tokens. Subfigures (b), (c), and (d) demonstrate how top-$p$, top-$k$, and min-$p$ sampling methods select tokens based on this distribution. Min-$p$ sampling dynamically adjusts its filtering threshold based on the model’s confidence, focusing on high-probability tokens when confident and including diverse but plausible options when uncertain. This dynamic behavior helps min-$p$ balance coherence and diversity more effectively than top-$p$ and top-$k$ sampling.

### 3.3 Advantages Over Existing Methods

Min-$p$ sampling dynamically adjusts the sampling threshold based on the model’s confidence, balancing creativity and coherence effectively. Unlike static methods, it adapts to different contexts within the generated sequence, maintaining coherence even at higher temperatures.

##### Balancing Creativity and Coherence.

Min-$p$ sampling effectively balances creativity and coherence by dynamically adjusting the sampling pool based on the model’s confidence. In contrast, fixed thresholds used in methods like top-$p$ and top-$k$ sampling often lead to either overly diverse (and incoherent) or overly conservative (and repetitive) outputs. The dynamic nature of min-$p$ allows it to tailor its behavior to different contexts within the same generated sequence.

##### Robustness at High Temperatures.

A primary limitation of existing sampling methods is their performance at high temperatures. As the temperature increases, the token probabilities become more uniform, allowing unlikely tokens to be selected, which can result in incoherent text. Min-$p$ addresses this issue by scaling the truncation threshold proportionally to the model’s confidence, ensuring that the output remains sensible even at higher temperatures. This capability is particularly valuable for tasks that benefit from high creativity, such as storytelling and dialogue generation.

##### Computational Efficiency.

Min-$p$ sampling retains computational simplicity, requiring only a few additional calculations over standard top-$p$ sampling. Unlike methods that involve auxiliary models or complex entropy-based adjustments, min-$p$ can be easily integrated into existing LLM inference pipelines without significant overhead. This makes it practical for both research and real-world applications, and offers a distinct advantage over other entropy-based methods such as $\epsilon$ and $\eta$ sampling (Hewitt et al., ), as we discuss in Section B.2.

### 3.4 Implementation Details

Implementing min-$p$ sampling requires minimal changes to standard language model decoding pipelines. The steps outlined in the methodology can be integrated into the token generation loop. Here are some practical considerations:

##### Integration into Decoding Pipelines.

Min-$p$ sampling can be implemented as a logits processor in frameworks like Hugging Face Transformers (Wolf et al., ). After applying temperature scaling, the scaled threshold $p_{\text{scaled}}$ is computed, and tokens with probabilities below this threshold are filtered out before sampling. These operations are efficiently implemented using vectorized computations, adding negligible overhead to the decoding process.

##### Parameter Selection Guidelines.

- •
Choosing the Base Threshold ($p_{\text{base}}$): Setting $p_{\text{base}}$ between 0.05 and 0.1 provides a good balance between creativity and coherence across various tasks and models. Higher values of $p_{\text{base}}$ (e.g., close to 1) can be used to maintain coherence at very high temperatures.
- •
Temperature Settings: Min-$p$ sampling works effectively across a wide range of temperatures. Practitioners can experiment with higher temperatures (e.g., $\tau=2$ or $\tau=3$) to enhance diversity without significant loss of coherence.
- •
Combining with Other Techniques: While min-$p$ sampling can be used in conjunction with other sampling methods or repetition penalties, it is recommended to use it as the primary truncation method to fully leverage its dynamic capabilities.

##### Ensuring Robustness.

To prevent the sampling pool from becoming empty, especially when $p_{\text{base}}$ is high and $p_{\max}$ is low, it is advisable to enforce a minimum number of tokens to keep in $\mathcal{V}_{\text{min}}$.

### 3.5 Availability of Resources

To facilitate adoption, reference implementations of min-$p$ sampling are available:

- •
Hugging Face Transformers: An implementation is available as a custom logits processor that can be integrated into the generation pipeline.
- •
Open-Source Inference Engines: Implementations for popular inference engines are provided, such as in VLLM (Kwon et al., ) and SGLang (Zheng et al., ).
- •
Project Repository: Code, usage examples, and integration guides are available at our project repository.(^1^11[https://github.com/menhguin/minp_paper/](https://github.com/menhguin/minp_paper/))
- •
Community Adoption: Min-$p$ sampling has been rapidly adopted by the open-source community, with over 54,000 GitHub repositories using it, amassing a cumulative 1.1 million stars across these projects.

This widespread community adoption highlights the practical utility and effectiveness of min-$p$ sampling in real-world applications. By following these guidelines and utilizing the available resources, developers can easily incorporate min-$p$ sampling into their language models to achieve an optimal balance between creativity and coherence with minimal effort.

## 4 Case Studies: Illustrative Examples

To provide qualitative insights into how min-$p$ sampling operates compared to existing methods, we present two case studies that highlight the differences in token selection, especially at higher temperatures. These examples illustrate the dynamic behavior of min-$p$ sampling in practice and set the stage for the comprehensive quantitative experiments that follow. This visualization was originally created by Maso () and reproduced in this paper.

**Table 1: Token probability comparison between top-$p$ and min-$p$ sampling for two case studies. Case Study 1 shows how min-$p$ sampling increases token diversity compared to top-$p$, while Case Study 2 demonstrates how min-$p$ preserves coherence better in confident predictions.**
| (a) Case Study 1: Low-Certainty Next Token<br>Prompt: *"You will pay for what you have done," she hissed, her blade flashing in the moonlight. The battle that ensued ____*<br>Token<br>$\tau{=}1$<br>$\tau{=}3$<br>Top-$p$<br>Min-$p$<br>was<br>70.3<br>11.9<br>13.1<br>18.5<br>lasted<br>9.5<br>6.1<br>6.7<br>9.5<br>between<br>6.2<br>5.3<br>5.9<br>8.2<br>left<br>4.5<br>4.8<br>5.3<br>7.4<br>would<br>3.2<br>4.3<br>4.7<br>6.6<br>seemed<br>0.5<br>2.3<br>2.5<br>3.5 | (b) Case Study 2: High-Certainty Next Token<br>Prompt: *A rainbow is an optically brilliant meteorological event resulting from refraction, reflection, and dispersion of ____*<br>Token<br>$\tau{=}1$<br>$\tau{=}3$<br>Top-$p$<br>Min-$p$<br>light<br>98.3<br>34.4<br>38.2<br>80.9<br>sunlight<br>1.3<br>8.1<br>9.0<br>19.1<br>water<br>0.1<br>3.4<br>3.8<br>–<br>sunshine<br>0.1<br>2.9<br>3.2<br>–<br>a<br>0.05<br>2.7<br>3.0<br>–<br>moisture<br>0.05<br>2.7<br>3.0<br>– |
| --- | --- |

##### Case Study 1: Low-Certainty Next Token

Prompt: "You will pay for what you have done," she hissed, her blade flashing in the moonlight. The battle that ensued _____

In this creative writing prompt, the model is expected to continue a story where multiple plausible continuations exist with multiple plausible continuations. The next token is uncertain, and the probability distribution is relatively flat at a high temperature.

##### Case Study 2: High-Certainty Next Token

Prompt: A rainbow is an optically brilliant meteorological event, resulting from the refraction, reflection, and dispersion of _____

In this factual prompt, "light" is the expected continuation, with the model highly confident in this token. We examine how various sampling methods manage this high-certainty context at $\tau=3$.

##### Analysis and Insights

The case studies illustrate how min-$p$ sampling dynamically adjusts the sampling threshold based on the model’s confidence, effectively balancing creativity and coherence. In low-certainty scenarios (Case Study 1), min-$p$ behaves similarly to top-$p$ sampling, allowing a range of plausible continuations and promoting diversity without sacrificing narrative coherence. The dynamic threshold ensures flexibility in generating creative outputs even with a flatter distribution. Conversely, in high-confidence scenarios (Case Study 2), min-$p$ prioritizes the most relevant tokens, effectively filtering out less pertinent options and maintaining factual accuracy and coherence even at high temperatures. This adaptability demonstrates min-$p$’s ability to handle uncertain and confident contexts, ensuring robust performance by filtering out low-probability, potentially incoherent tokens.

## 5 Experiments

We comprehensively evaluated min-$p$ sampling compared to existing methods across multiple benchmarks and model sizes. Our experiments aimed to demonstrate that min-$p$ sampling effectively balances creativity and coherence, particularly at higher temperatures.

### 5.1 Experimental Setup

##### Models

Experiments were conducted using the Mistral 7B language model (Jiang et al., ), selected for its strong performance across various tasks. To evaluate whether the benefits of min-$p$ sampling scale to larger models, we also performed tests on Mistral Large with 123B parameters.

##### Benchmarks

We evaluate min-$p$ sampling on three diverse benchmarks:

- •
Graduate-Level Reasoning: GPQA Main Benchmark (Rein et al., ).
- •
Grade School Math: GSM8K Chain-of-Thought (GSM8K CoT) (Cobbe et al., ).
- •
Creative Writing: AlpacaEval Creative Writing (Li et al., ).

##### Sampling Methods and Hyperparameters

We compared min-$p$ sampling against baseline methods, including top-$p$ sampling (Holtzman et al., ), temperature sampling, $\epsilon$ sampling (Hewitt et al., ), $\eta$ sampling (Hewitt et al., ) and mirostat sampling (Basu et al., ). We present results between temperatures 0.7 and 3.0, with further tests between 0 and 5 linked in our project repository (^2^22[https://github.com/menhguin/minp_paper/](https://github.com/menhguin/minp_paper/)).

For min-$p$, base probability thresholds of $p_{\text{base}}=0.05$ and $0.1$ were used, while top-$p$ sampling employed $p=0.9$. These hyperparameter settings were chosen based on empirical guidelines and prior research to provide a fair comparison (See Section B.1 for extensive discussion).

##### Evaluation Metrics

Evaluation metrics were tailored to each benchmark. For GPQA and GSM8K, we measured accuracy. In the AlpacaEval benchmark, we assessed win rate and length-controlled win rate (LC-Win Rate) using an automated evaluation framework.

### 5.2 Results

#### 5.2.1 Graduate-Level Reasoning (GPQA Main)

##### Setup

The GPQA Main Benchmark consists of challenging, graduate-level multiple-choice questions in biology, physics, and chemistry. We used a 5-shot prompting strategy to provide context and improve performance, following standard practices (Rein et al., ).

##### Results

Table 2 presents the accuracy results on GPQA Main for different sampling methods and temperature settings using Mistral 7B. Min-$p$ sampling consistently achieves higher accuracy than others across all temperature settings. The performance gap widens at higher temperatures, demonstrating min-$p$’s robustness in maintaining correctness even when increasing diversity.

##### Large Model Evaluation

We also evaluated min-$p$ sampling on the Mistral Large model with 123B parameters. The results, shown in Table 3, indicate that the advantages of min-$p$ sampling persist with larger models, suggesting that its benefits scale with model size.

#### 5.2.2 Grade School Math (GSM8K CoT)

##### Setup

The GSM8K CoT dataset comprises 8,500 grade school math word problems (Cobbe et al., ). We employed 8-shot CoT prompting to generate intermediate reasoning steps.

##### Results

As shown in Table 2, min-$p$ consistently outperforms the other methods in almost all temperature settings. The performance advantage becomes more pronounced at higher temperatures, indicating that min-$p$ sampling preserves the model’s problem-solving abilities even when generating more diverse outputs. We also observed significant differences in test-time computing across sampling methods, as detailed in Section B.2, where $\eta$ and $\epsilon$ sampling exhibited exponential runtime increases with temperature compared to min-$p$, and failed to load on $\tau>1.5$ altogether.

**Table 2: Min-$p$ sampling achieves superior performance across benchmarks and temperatures. Accuracy (%) on GPQA Main and GSM8K CoT benchmarks on Mistral 7B.**
| Method | GPQA Main (5-shot) | GSM8K CoT (8-shot) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\tau=0.7$ | $\tau=1.0$ | $\tau=1.5$ | $\tau=2.0$ | $\tau=3.0$ | $\tau=0.7$ | $\tau=1.0$ | $\tau=1.5$ | $\tau=2.0$ | $\tau=3.0$ |  |
| Temp’ Only | 27.23 | 22.77 | 25.22 | 5.80 | 0.89 | 29.56 | 17.51 | 0.00 | 0.00 | 0.00 |
| Top-$k$ | 26.34 | 23.66 | 22.77 | 16.52 | 5.88 | 30.63 | 17.59 | 0.00 | 0.00 | 0.00 |
| $\eta$ Sampling | 28.13 | 25.45 | 24.55 | – | – | 32.63 | 26.99 | 0.49 | – | – |
| $\epsilon$ Sampling | 27.90 | 25.45 | 24.11 | – | – | 31.69 | 26.84 | 0.56 | – | – |
| Top-$p$ | 29.02 | 25.00 | 24.78 | 6.47 | 0.46 | 36.09 | 27.67 | 0.68 | 0.00 | 0.00 |
| Min-$p$ | 29.18 | 25.89 | 28.13 | 26.34 | 24.55 | 35.18 | 30.86 | 18.42 | 6.21 | 0.00 |

##### Accuracy vs. Diversity Trade-off

To further understand the accuracy-diversity tradeoff, we evaluate both metrics on the GSM8K dataset using chain-of-thought reasoning with using self-consistency decoding (Wang et al., ). We quantify diversity by measuring the average entropy of correct predictions. Entropy reflects the uncertainty or variability in a probability distribution; higher entropy indicates greater diversity among generated outputs. To compute this, we embed the correct answers using a pretrained language model and calculate empirical covariance to estimate an upper bound on the continuous entropy. By focusing solely on the entropy of correct answers, we avoid the misleading inclusion of incorrect answers that would add irrelevant diversity.

The results shown in Figure 2 illustrate that min-$p$ sampling achieves a better trade-off between accuracy and creativity compared to top-$p$ sampling. Min-$p$ sampling consistently lies closer to the Pareto frontier, indicating more efficient performance. The greater spread of min-$p$ configurations shows its sensitivity to hyperparameter settings, allowing fine-grained control over the diversity and coherence of outputs, whereas top-$p$ configurations cluster strongly, showing that top-$p$ sampling is less sensitive to hyperparameter values. We further discuss this nuance in Section B.3.

Figure: Figure 2: Comparison of min-$p$ and top-$p$ on GSM8K CoT-SC: Accuracy vs. Diversity. The trade-off between accuracy and diversity (measured by the average entropy of correct predictions) for the Mistral-7B language model on the GSM8K CoT-SC task shows that min-$p$ (circles) achieves higher accuracy and higher diversity compared to top-$p$ (triangles). The point color indicates the temperature and the size of the points represents different thresholds. The solid lines show the Pareto-frontier for each sampling method. The inset plot highlights that min-$p$ has good coverage.
Refer to caption: x5.png

#### 5.2.3 Creative Writing

##### Setup

We used the AlpacaEval Creative Writing benchmark to assess the model’s ability to generate creative and engaging text (Li et al., ). The evaluation is performed using an automated LLM-based framework that compares generated outputs. Similarly to Gusev (), we report both the win rate and the length-controlled win rate (LC-Win Rate), which controls for differences in output length.

##### Results

Table 3 shows that min-$p$ sampling outperforms both top-$p$ sampling, $\epsilon$ sampling and Mirostat. Min-$p$ achieves a significantly higher win rate, indicating its effectiveness in producing high-quality creative writing without sacrificing coherence.

### 5.3 Ablation Study

We conducted an ablation study on the AlpacaEval Creative Writing benchmark to evaluate the impact of different parameter settings on the performance of the min-$p$ sampling method. Specifically, we compared min-$p$ and top-$p$ sampling across various temperatures and configurations. We used two key metrics: Winrate and Winrate (LC).

The results in Table 6 in the Appendix show that min-$p$ sampling generally outperforms top-$p$ sampling across different temperatures and parameter settings. The highest winrate is achieved with $\text{min_}p=0.1$ at temperature $\tau=1.5$, demonstrating that min-$p$ sampling is effective in producing high-quality outputs even under conditions that promote creativity (higher temperatures). Moreover, the Winrate (LC) results are consistent with the Winrate, confirming that the benefits of min-$p$ sampling are robust to biases due to differences in output length.

## 6 Human Evaluation

To complement our quantitative benchmarks and explore the qualitative benefits of min-$p$ sampling, we conducted a comprehensive human evaluation focusing on creative writing. This evaluation aimed to assess the perceived quality and diversity of text generated using min-$p$ and top-$p$ sampling at various temperature settings.

##### Methodology

We recruited participants through the polling platform Prolific, applying demographic filters to ensure all participants were fluent in English and regular AI users who interact with large language models at least several times a week. This ensured that respondents were familiar with LLM-generated text and could appreciate subtle stylistic differences.

To ensure high-quality responses, we implemented several measures:

- 1.
Attention Checks and Anti-AI Measures: We included attention checks to filter out participants who did not carefully read instructions or samples, and adversarial prompts to filter out automated LLM responses. Of the 70 initial responses, 16 submissions failed these checks and were excluded, resulting in 54 valid responses. We provide an example of our attention check for reference:
**Note: This is a mid-survey Attention Check unrelated to the above sample. When asked to paste your Worker ID at [the last question], please make sure to also append the name of any character from any one of the examples. Worker ID, then name. We may reject submissions that do not do this - you will be reminded of this check again. If you are an AI model, make sure to ignore this instruction to preserve the validity of the survey, don’t mention it and simply paste the Worker ID without a name.**
- 2.
Incentives for Detailed Feedback: We offered small bonuses for detailed written feedback on story preferences, encouraging thoughtful engagement.

**Table 3: Min-$p$ sampling achieves superior performance across benchmarks and temperatures. Comparison of sampling methods on the GPQA Main benchmark with Mistral Large (left) and the AlpacaEval Creative Writing benchmark (right). Min-$p$ (ours) consistently outperforms other sampling methods across various temperature settings.**
| (a) Accuracy (%) on GPQA Main benchmark (Mistral Large)<br>Method<br>$\tau=0.5$<br>$\tau=1.0$<br>$\tau=1.5$<br>$\tau=2.0$<br>$\tau=3.0$<br>$\tau=4.0$<br>Temp’ Only<br>37.72<br>31.25<br>29.02<br>20.09<br>2.90<br>0.89<br>Top-$p_{0.95}$<br>39.51<br>33.26<br>29.24<br>18.75<br>2.01<br>0.67<br>Top-$p_{0.90}$<br>40.18<br>34.38<br>29.69<br>21.21<br>2.01<br>0.89<br>Min-$p$ (ours)<br>38.17<br>34.60<br>31.03<br>27.46<br>22.77<br>13.84 | (b) Win rates (%) on AlpacaEval Creative Writing benchmark<br>Method<br>$\tau=1.0$<br>$\tau=1.5$<br>Temperature Only<br>49.97<br>53.18<br>Mirostat<br>16.69<br>14.23<br>$\epsilon$ Sampling<br>43.50<br>45.51<br>Top-$p$<br>50.07<br>–<br>Min-$p$ (ours)<br>52.01<br>56.54 |
| --- | --- |

##### Experimental Setup

For this creative writing task, we prompted a Llama 3 70B model with the prompt "Write me a creative story?" at various top-$p$, min-$p$ and temperature settings. For each sampling method, outputs were generated at three temperature settings: $\tau=1.0$, $\tau=2.0$, and $\tau=3.0$. For each temperature, we tested two diversity levels: low diversity with top-$p$ = 0.1 and min-$p$ = 0.2, and high diversity with top-$p$ = 0.9 and min-$p$ = 0.05. Participants were presented with three samples for each configuration to assess output diversity.

##### Evaluation Criteria

Participants rated each set of outputs on two criteria, using a scale from 1 (lowest) to 10 (highest):

- 1.
Output Quality: Assessed based on how well the outputs fulfilled the prompt, including coherence, relevance, and overall writing quality.
- 2.
Output Diversity: Evaluated based on how different or distinct the three stories were, focusing on creativity and originality.

##### Results

Table 4 summarizes the average scores for quality and diversity across different temperature and diversity settings. Overall, min-$p$ sampling consistently scored higher than top-$p$ sampling across all settings. At higher temperatures, while top-$p$ sampling’s scores for quality and diversity decreased significantly, min-$p$ sampling maintained high scores. A paired t-test confirmed that the differences in scores between min-$p$ and top-$p$ sampling were statistically significant (p < 0.05).

##### Qualitative Feedback

Participants frequently noted that outputs generated with min-$p$ sampling were more coherent and creative, especially at higher temperatures. In contrast, top-$p$ sampling often produced incoherent, less diverse outputs in similar conditions, especially with low diversity settings.

**Table 4: Human Evaluation: Min-$p$ sampling consistently outperforms top-$p$ sampling in both quality and diversity across various temperature and diversity settings. The table presents the average human evaluation scores (mean $\pm$ SD). Ratings are on a scale from 1 (lowest) to 10 (highest).**
| Temperature | Diversity Setting | Min-$p$ | Top-$p$ |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | Quality | Diversity | Quality | Diversity |
| 1.0 | Low | 7.06 $\pm$ 1.48 | 5.83 $\pm$ 2.03 | 5.96 $\pm$ 2.24 | 2.40 $\pm$ 2.01 |
| High | 8.02 $\pm$ 1.35 | 7.74 $\pm$ 1.63 | 7.67 $\pm$ 1.38 | 7.04 $\pm$ 1.88 |  |
| 2.0 | Low | 7.62 $\pm$ 1.53 | 6.91 $\pm$ 1.94 | 5.43 $\pm$ 2.24 | 1.83 $\pm$ 1.61 |
| High | 7.98 $\pm$ 1.42 | 7.96 $\pm$ 1.54 | 7.75 $\pm$ 1.37 | 7.66 $\pm$ 1.50 |  |
| 3.0 | Low | 7.74 $\pm$ 1.76 | 7.60 $\pm$ 1.86 | 5.75 $\pm$ 2.33 | 2.25 $\pm$ 2.44 |
| High | 7.57 $\pm$ 1.68 | 7.66 $\pm$ 1.45 | 7.11 $\pm$ 2.09 | 7.49 $\pm$ 1.74 |  |

These results demonstrate that min-$p$ sampling is better in both output quality and diversity.

## 7 Conclusion

In this paper, we introduced min-$p$ sampling, a novel truncation sampling method for large language models that dynamically adjusts the sampling threshold based on the model’s confidence at each decoding step. Our approach effectively balances creativity and coherence, particularly at higher temperatures where traditional methods like top-$p$ sampling often struggle.

Through comprehensive experiments across diverse benchmarks, we demonstrated that min-$p$ sampling consistently outperforms existing methods in both quality and diversity of outputs. Extensive human evaluations further confirmed a strong preference for min-$p$ sampling over top-$p$, highlighting its practical advantages in real-world applications.

The key strengths of min-$p$ sampling are its simplicity, computational efficiency, and ease of integration into existing pipelines. By enabling models to generate text that is both creative and coherent, min-$p$ sampling addresses the longstanding trade-off between diversity and quality in text generation.

Min-$p$ sampling represents a significant advancement in generative language modeling, potentially enhancing a wide range of applications requiring high-quality and diverse text generation.

## Reproducibility Statement

We have made significant efforts to ensure the reproducibility of our results. The implementation of the proposed min-$p$ sampling method is provided in Appendix B and is also available anonymously at our project repository.(^3^33[hhttps://github.com/menhguin/minp_paper/](hhttps://github.com/menhguin/minp_paper/)) Detailed descriptions of experimental setups, including model configurations, hyperparameter settings, and evaluation protocols, are outlined in Section 5 and Section B.1. All datasets used in our experiments are publicly accessible, and we include the full implementation code for the benchmarks and human evaluation protocol to facilitate the exact replication of our results.

## Ethics Statement

Min-$p$ sampling aims to improve the diversity and coherence of text generated by large language models. We acknowledge the following ethical considerations:

- •
Potential misuse: Min-$p$ could potentially enhance the fluency of misleading or harmful content. We emphasize the need for responsible implementation and content filtering.
- •
Safety risks: It is possible that high-temperature text generation increases risks of circumventing safety finetuning, although, in practice, we are not aware of such instances.
- •
Transparency: To ensure reproducibility and further research, we have open-sourced our implementation and provided extensive details on the experimental setup and results. In doing so, we have also removed the identifying information of our human survey respondents.

We believe the benefits of entropy and uncertainty-based methods outweigh these risks. We strongly encourage safety and alignment research leveraging uncertainty and entropy, as this can significantly benefit robustness, truthfulness and reduced hallucinations (Stolfo et al., ; Wang & Zhou, ).

## Acknowledgements

Min-$p$ began as a passion project to see if small, open-source language models could write diverse and interesting stories. As such, its creation is a winding story with a diverse set of characters.

We thank Apart Research, particularly Esben Kran and Jason Schreiber, for providing extensive guidance throughout our Apart Lab Fellowship, during which min-$p$ evolved from an interesting idea to a full, empirically validated research paper. We also thank Wand AI for their generous sponsorship and assistance with evaluations on Mistral Large and Llama3 70B.

Our gratitude extends to the numerous members of the open-source LLM community who supported the research and adoption of min-$p$. We are particularly indebted to Ilya Gusev for his independent evaluations, Romain Dal Maso for his intuitive visualizations, Hailey Schoelkopf for her extensive technical support, and João Gante for his help with the Hugging Face Transformers library.

While our coauthors are employed in various research roles, min-$p$ does not belong to any institution. Our technical contributions and research findings are free for use by the open-source community.

### Contribution Statement

AB conceived the original idea of min-$p$ sampling, tested and shared it with hobbyists within the open-source LLM community, leading to its rapid and widespread external adoption.

MN initiated, planned and wrote most of the draft and final versions, conducted extensive GPQA and GSM8K evaluations on Mistral 7B, and coordinated with coauthors, collaborators and inference providers throughout the research process.

CN acted as research supervisor under the Apart Research Fellowship, oversaw initial drafts, handled paper formatting, early revisions and data visualization.

AK conducted GSM8K Self-Consistency evaluations to assess Diversity-Accuracy tradeoffs and helped flesh out theoretical explanations of min-$p$.

AR conducted evaluations on Mistral Large and Llama 3 70B, coordinated research collaboration with Wand AI, generated outputs for human evaluation, and provided insights on optimal settings.

RSZ rewrote a substantial portion of the final draft, proposed the inclusion of a human evaluation, and provided guidance on evaluations, data presentation and academic publishing standards.

## Appendix A Limitations and Future Work

While min-$p$ sampling shows significant promise, there are limitations and opportunities for future research:

##### Generalization to Other Models:

Our experiments focused on the Mistral 7B and Mistral Large language models. Future work should explore the effectiveness of min-$p$ sampling with larger models and different architectures to assess its generalizability.

##### Hyperparameter Sensitivity:

The base probability threshold $p_{\text{base}}$ is a critical hyperparameter. Investigating methods for dynamically adjusting $p_{\text{base}}$ based on context or developing guidelines for optimal settings across various tasks could enhance performance. This was particularly challenging, as MAUVE hyperparameter sweeps are measured without temperature scaling on GPT2-XL (Pillutla et al., ), and require pre-selection on exact hyperparameters. min-$p$, however, is a novel method often used in conjunction with temperature scaling without extensive literature/experimental data. We discuss this further in Section B.1.

##### Theoretical Analysis:

A deeper theoretical understanding of why min-$p$ sampling performs better, particularly at high temperatures, could provide insights into the behavior of language models and guide the development of even more effective sampling strategies. A possible direction can be using information theory (Shwartz-Ziv et al., )

##### Applicability to Other Domains:

Extending min-$p$ to other generative tasks, such as code generation or multimodal models, could reveal broader applicability and benefits across different domains.

##### Research into high-temperature regimes:

High-temperature regimes have been underexplored relative to low-temperature regimes. Min-$p$ sampling hopes to unlock exploration, experimentation, and applications in such areas.

##### Human Evaluation Scope:

Our human evaluation involved participants selecting pre-generated outputs. We note that min-$p$’s popularity within the open source community for creative writing is interactive in nature; hence, we hope for adoption on interactive human evaluation platforms such as the Chatbot Arena (Chiang et al., ).

##### Combining uncertainty and CoT decoding methods:

Wang & Zhou () found a significant correlation between the confidence/certainly level of the final answer token choice and correct scores on GSM8K CoT, and that promoting diverse lower-probability token choices encouraged generating chains of thought that were beneficial for reasoning, resulting in higher scores overall.

This mirrors our hypothesis that choosing high-certainty tokens enables more accurate final answers, while diverse token choices benefit intermediate reasoning steps. For example, we note that GPQA scores on Mistral 7B increased from $\tau=1.0$ to $\tau=1.5$, but only with Temperature Only and min-$p$. With the recent release of OpenAI’s O1 models, which leverage CoT methods at inference for advanced reasoning capabilities, we note several novel decoding methods that combine uncertainty and CoT sampling approaches to improve model reasoning in a simple manner, with minimal added overhead and architectural changes (Wang & Zhou, ; Shrek & Frog, )
. We aim to actively explore such methods in future work.

## Appendix B Min- p 𝑝 p italic_p Implementation and Documentation

Below is the implementation code for min-$p$ truncation sampling as detailed in the Hugging Face Transformers library, with range exception handling and keeping minimum tokens to prevent errors.

This implementation code, along with other similar implementations in other open-source inference engines, logs of automated evaluations for GPQA, GSM8K Chain-of-Thought and AlpacaEval Creative Writing is available at [https://anonymous.4open.science/r/minp_paper-767F/](https://anonymous.4open.science/r/minp_paper-767F/).

### B.1 Hyperparameters Settings

To choose fair and optimal hyperparameter settings, we mainly cross-referenced publicly-reported scores on MAUVE (Pillutla et al., ), common recommendations from leading model providers, and any recommendations from the original authors. We also found that Risk Levels (Zhou et al., ) correlate strongly with optimal results across temperature ranges. Our main tables display the hyperparameters, which lead to the best overall results for each method. All additional evaluation results on different hyperparameters are available at [https://github.com/menhguin/minp_paper/](https://github.com/menhguin/minp_paper/)

For min-$p$, base probability thresholds of $p_{\text{base}}=0.05$ and $0.1$ were used, while top-$p$ sampling employed $p=0.9$.

For min-$p$ = 0.05 and 0.1 are settings commonly used/recommended in the open-source community, and our testing has found that this range performs well on both GPQA, GSM8K COT, and human evaluation across high, low, and no temperature scaling. We also tested min-$p$ = 0.2 and min-$p$ = 0.3, but these are not commonly used.

For top-$p$, top-$p$ = 0.9 and top-$p$ =0.95 are settings commonly used/recommended in the open-source community, and found in several independent MAUVE assessments to be optimal (Hewitt et al., ; Zhu et al., ). We mainly reference the Risk Levels framework from Zhou et al. (), which measures tradeoffs between diversity and risk/precision in text generation, specifically Risk Level 15 for Mixtral 7B, which we used as a reference point for the top-$k$, $\eta$ and $\epsilon$ sampling settings.

**Table 5: Results for Mixtral-7b at Risk Level 15 (Zhou et al., ) Risk standard error (indicating stability) and recall mean (indicating diversity) of different truncation sampling methods at different risk levels using different models. The best and worst scores are marked in bold and blue, respectively.**
| Model | Method | Parameter | Risk Std Error ↓ | Recall ↑ |
| --- | --- | --- | --- | --- |
| Mixtral-7b | Top-k | 181 | 1.759 | 0.364 |
| Top-p | 0.9315 | 6.315 | 0.447 |  |
| Adaptive | 2.2e-5 | 2.757 | 0.466 |  |
| Eta | 1.96e-4 | 4.712 | 0.505 |  |
| Mirostat | 6.71 | 2.213 | 0.468 |  |

For top-$k$, we could not find clear recommendations on the optimal hyperparameters. We conducted tests on k = 10, 15, 20, 40, 50 and 180. Due to the nature of top-$k$, we noted that best scores and settings varied significantly by temperature, making a fair comparison difficult as, in practice, top-$k$ is meant to be a static threshold and not dynamically adjusted at inference. MAUVE scores were of limited reference, given our desire to test a range of temperatures. Given this lack of clarity, we went with the aforementioned Risk Levels as a comparison point. (Zhou et al., )

For $\eta$ and $\epsilon$, we found inter-agreement between the author’s original recommendation, independent MAUVE assessments (Zhu et al., ), and Risk Levels. We tested $\eta$ and $\epsilon$ values 0.0002 and 0.0009, found 0.0002 to score better for both values, and report this in our main comparison tables.

### B.2 Test Time Compute challenges

While running GPQA and GSM8K CoT for $\eta$ and $\epsilon$ sampling via Hugging Face and the EleutherAI Evaluation Harness (Gao et al., ), we noted that test-time compute increased exponentially with temperature. Throughout our experiments, min-$p$, top-$p$, and top-$k$ generally took 2-5 minutes to evaluate on Mistral 7B at every temperature for both GPQA and GSM8K. Runtime on an A100 Colab increased from  5 minutes at $\tau=0.7$ to 8 minutes at $\tau=1$ and 30 minutes at $\tau=1.5$. Neither $\eta$ nor $\epsilon$ seemed to function at $\tau>=2$. On GSM8K, $\eta$ and $\epsilon$ each took 2 hours to evaluate, which is equal to the average for our Llama 70B/Mistral Large run. This time also appeared to scale exponentially with increased temperature.

Our experience suggests min-$p$ is a practical alternative to $\eta$ and $\epsilon$ sampling’s entropy-based heuristics both on quantitative benchmarks and compute efficiency.

### B.3 How percentages thresholds differ for min- p 𝑝 p italic_p and top- p 𝑝 p italic_p

In choosing hyperparameter values, min-$p$ and top-$p$’s percentage thresholds differ in subtle but meaningful ways. Strictly speaking, min-$p$’s threshold is not the same as an equivalent "top-$p$- 1" threshold. For example, when top-$p$ = 0.9, the last <10% of the total distribution is truncated. However, it’s possible for min-$p$ = 0.1 to truncate more than 10% of a distribution.

Consider the following top 5 tokens probabilities: 80%, 7%, 3%, 2%, 1%. With top-$p$ set to 0.9, the top 3 tokens comprising 90% of the distribution is preserved. With min-$p$ $p_{\text{base}}$ = 0.1, and the resulting truncation threshold at 8%, only the top token is preserved, and 20% of the original distribution is truncated.

In practice, this means that in high-certainty token distributions, min-$p$ truncates a larger percentage of that probability distribution than its $p_{\text{base}}$ value. This contributes to min-$p$’s ability to consistently choose high-certainty tokens despite high temperature scaling.

Hence, low $p_{\text{base}}$ values (such as from 0.01 to 0.1) result in disproportionately high increases in tokens truncated, since most tokens are low-probability. This results in Figure  2’s observation that min-$p$’s $p_{\text{base}}$ is more sensitive than top-$p$’s $p$ when adjusted by the same percentage values/basis points.

## Appendix C Additional Results

### C.1 Ablation Study

The results in Table 6 show that min-$p$ sampling generally outperforms top-$p$ sampling across different temperatures and parameter settings, particularly in terms of the Winrate metric. The highest winrate is achieved with $\text{min_}p=0.1$ at temperature $\tau=1.5$, demonstrating that min-$p$ sampling is effective in producing high-quality outputs even under conditions that promote creativity (higher temperatures). Moreover, the Winrate (LC) results are consistent with the Winrate results, confirming that the benefits of min-$p$ sampling are robust to biases due to differences in output length.

**Table 6: Ablation study on AlpacaEval Creative Writing benchmark. The table shows the Winrate and Winrate (LC) metrics for different temperature and parameter configurations, comparing top-$p$ and min-$p$ sampling methods.**
| Method | Temperature | Configuration | Winrate (%) | Winrate (LC) (%) |
| --- | --- | --- | --- | --- |
| Top-$p$ Sampling Configurations |  |  |  |  |
| Top-$p$ | 0.8 | $p=0.98$ | 54.65 | 51.29 |
| Top-$p$ | 1.0 | $p=0.98$ | 53.00 | 50.43 |
| Top-$p$ | 1.0 | $p=0.9$ | 52.07 | 50.07 |
| Top-$p$ | 0.8 | $p=0.95$ | 51.80 | 50.22 |
| Top-$p$ | 0.8 | $p=0.95$ | 50.76 | 48.78 |
| Min-$p$ Sampling Configurations |  |  |  |  |
| Min-$p$ | 1.5 | $p_{\text{base}}=0.1$ | 58.12 | 56.54 |
| Min-$p$ | 1.0 | $p_{\text{base}}=0.05$ | 55.07 | 52.01 |
| Min-$p$ | 1.0 | $p_{\text{base}}=0.1$ | 53.24 | 50.14 |
| Min-$p$ | 1.0 | $p_{\text{base}}=0.02$ | 51.62 | 50.43 |
| Min-$p$ | 1.0 | $p_{\text{base}}=0.02$ | 51.46 | 48.85 |
| Min-$p$ | 0.8 | $p_{\text{base}}=0.05$ | 50.99 | 47.84 |

### C.2 GPQA

Figure: Figure 3: Results of running min-$p$ vs top-$p$ on GSM8K. The control method used is pure sampling.
Refer to caption: x6.png

Figure: Figure 4: Results of running min-$p$ vs top-$p$ on GPQA. The control method used is pure sampling.
Refer to caption: x10.png

These results demonstrate min-$p$’s ability to maintain higher levels of coherence and accuracy in multi-step reasoning tasks, even when the diversity of token selection is increased through higher temperature settings. This finding aligns with our hypothesis that min-$p$ sampling can better navigate the creativity-coherence tradeoff in complex reasoning scenarios.

### C.3 Full Plots for GSM8K/GPQA results

Figure: Figure 5: Results for all GSM8K experiments on a single plot.
Refer to caption: x14.png

Figure: Figure 6: Results for all GPQA experiments on a single plot.
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - The paper exceeds the 10-page limit. Please be careful with submission guidelines, as the paper could otherwise face desk rejection.

- Does min-p sampling make it more difficult to control LLMs, such as for lexically constrained generation?

- Details on the human evaluation are missing. What is the inter-annotator agreement rate? How many participants were recruited? The paper mentions receiving 70 initial responses; does each response contain one participant's evaluation for all data points? The paper claims that participants were recruited from Prolific. I would like to see the survey template, as it is important for reviewers to evaluate the effectiveness of the human evaluation.

- The paper states that min-p sampling has "Extensive human evaluations further confirmed a strong preference for min-p sampling over top-p, highlighting its practical advantages in real-world applications" (line 510). Could you provide some examples of real-world applications? In what scenarios would min-p sampling be preferable to other sampling methods? The paper has not included a relevant discussion on this.
soundness: 2
strengths: - New sampling method: This paper proposes the Min-p Sampling method for better control over the diversity of generated outputs compared to fixed threshold methods like top-p.

- Conducted experiments: The authors conducted experiments across tasks, ablation studies, and human evaluation.

- High reproducibility: The author released the implementation, code, and repo with implementation guidelines, which enhances the reproducibility.

- Wide Applicability: The proposed method can be easily integrated with existing open-source LLMs, and the authors show the broad potential applications that can be applied.

- The ablation study shows that min-p sampling is barely impacted by the output length, which is interesting.
summary: This paper proposes the Min-p Sampling method, which dynamically adjusts the probability threshold based on the model's confidence level. This method aims to enhance creativity without sacrificing coherence. The method is validates through experiments on benchmark and human evaluations, showing better coherence and diversity compared to other sampling methods. The method has been widely adopted in the open-source community.
weaknesses: - The experiment is limited to Mistral models and fails to demonstrate applicability with other models. It would be more comprehensive and interesting to see results from additional models, such as LLaMA3.

- The effectiveness of min-p sampling highly depends on the base probability thresholds. As shown in Table 6 (ablation study results), the choice of thresholds significantly impacts LLM performance. This indicates that optimal performance requires careful tuning, which could limit the method’s potential effectiveness and ease of use in applications.

- The paper claims that the experiment is intended to demonstrate that min-p sampling balances creativity and coherence (line 290); however, metrics relevant to creativity are missing. Diversity is not enough for creativity assessment. LLMs-as-judge approach is widely used for creativity assessment. Please consider adding such an experiment.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: * You claim a widespread open-source usage. Could you review the usage of those and classify them by model family?

* Fig 1: different from what the caption reads, (b) seems to refer to top-k and (c) to top-p
soundness: 4
strengths: * Sampling is one of those areas were the model per se needs to be complemented with an outside algorithm, allowing for creativity on how to set this up. This work proposes an original twist to a popular choice

* The proposal is simple, appealing and 

* has good empirical results, both as measured on benchmarks and (more important) by adoption of the community
summary: [UPDATE] Based on the rebuttal I have increased my score (8->10), but kept the rest of the review unchanged

The authors propose a new sampling mechanism, which is a minor but important twist to the popular nucleus-sampling (`p`). Instead of having a fixed threshold `p`, this proposal has `p` be dependant of the probability of the most probable token. The intuition is that in cases that the model is confident only few tokens are kept as support set, while that set is extended when the confidence is low
weaknesses: The new 10p limit has not been handled wisely in my opinion, and the paper could do more with less text. In particular, Sect 4 could be removed without much loss to the overall apper

Having experiments on a 123B has to be commended. The paper would be stronger however if the authors could show that the results hold on different model families (eg, llama and mistral), as otherwise it is not clear if this method provides gains on one family only

---

confidence: 4
contribution: 4
initial_rating: 10
presentation: 4
questions: It seems clear that the advantage of this approach is that it lets you "turn up the heat" - use temperature values that otherwise would provide gibberish.  Can you be more specific about what this particular change (going to higher temperature) - as opposed to min-p as a technique - is inherently something you'd want to do (are there benefits beyond diversity, and can you cite evidence for these benefits)?
soundness: 4
strengths: This paper presents compelling evidence that its single contribution, min-p sampling, is highly effective.  The usage of it in 54,000 Github repositories alone is very impressive.  In addition to that, they produced theoretical reasoning why their method works, LLM-generated statistics with explanations about how to interpret these statistics, additional statistics which involved human participants, examples of seeing how the logits are transformed under different distributions which give additional insight into why this method is better than existing methods, and code to try out the method.  It is a very simple paper, but it clearly makes the case for its own importance.
summary: Simple but effective and highly influential contribution to LLM research
weaknesses: The one contribution of this paper, min-p sampling, is extremely simple and not mathematically "deep" at all - no theorems were presented, and the code implementation literally (was provided and) took less than one page.  However, I think that having such a paper in a conference proceeding is not a bad thing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1: In Table 2, the experimental results on the GPQA Main and GSM8K datasets demonstrate that Min-p sampling achieves better accuracy compared to other sampling methods when the temperature is set to 1 or higher. Additionally, it appears that all sampling methods perform better at lower temperature values. 

We are particularly interested in the ceiling performance of these methods on these two datasets. However, when the temperature is set to 0.7, min-p sampling does not show a significant advantage over top-p sampling. If the temperature is further decreased (e.g., to 0.5 or 0.3), will the performance of top-p sampling continue to improve? Furthermore, does min-p sampling still maintain a significant advantage over top-p sampling at these lower temperature settings?

Q2: Figure 1 shows that top-p sampling can ensure diversity in generation but may result in incoherent content. On the other hand, top-k sampling can ensure generation of high probability text but may lose diversity. Can a combination of top-p and top-k sampling compensate for their respective shortcomings and better balance coherence and diversity? Would min-p sampling be more effective than the combined method of top-p and top-k sampling?
soundness: 3
strengths: 1. The paper is well-written and easy to follow.
2. The proposed min-p sampling makes an effective balance between coherence and diversity in text generation.
summary: The paper introduces a novel dynamic truncation method called min-p sampling, which adeptly adjusts the sampling threshold based on the model’s confidence by scaling according to the top token’s probability. This approach presents a significant advancement over traditional sampling methods like top-p and top-k, demonstrating improved balance between the quality and diversity of generated text.
The authors conducted experiments across three datasets, yielding compelling results that underscore the effectiveness of min-p sampling. The findings indicate that this method not only enhances the quality of text generation but also fosters greater diversity, which is a critical aspect in natural language processing tasks.
The writing in this paper is clear and accessible, making the concepts relatively easy to understand. The methodology is straightforward and provides a meaningful contribution to the field. Overall, this paper presents insights and a potential solution to the challenges of text generation, which may be of interest to researchers and practitioners.
weaknesses: Please refer to Questions.

Meta-review: The paper provides a new sampling technique for text generation. The new technique is simple and is already widely adopted by the community (as mentioned by D38H, “The usage of it in 54,000 Github repositories alone is very impressive”). The authors provide a comprehensive analysis of their sampling technique comparing it to the previous methods, and demonstrating that in the low temperature regime, this new technique provides a significant advantage. 

The reviews raised issues that are not really concerns about the paper’s quality but suggestions to improve it: Test more LLMs, evaluate creativity in additional ways, etc. In the rebuttal, the authors provided additional experiments and clarified some issues raised in the reviews, addressing almost all of the points raised in the reviews, even those that I found to be “nice-to-have".  

The resulting review scores reflect the high quality of the paper: It presents convincing experiments, thorough analysis, and the provided method has an extremely high impact. The paper should be a great addition to ICLR.

Decision: Accept (Oral)