Kati8/ICLR_2023_2025 huggingface dataset:
authors: Nguyen Nhat Minh
authors: Andrew Baker
authors: Clement Neo
authors: Allen G Roush
authors: Andreas Kirsch
authors: Ravid Shwartz-Ziv
reviews: 
Comments: The paper introduces a novel dynamic truncation method called min-p sampling, which adeptly adjusts the sampling threshold based on the model’s confidence by scaling according to the top token’s probability. This approach presents a significant advancement over traditional sampling methods like top-p and top-k, demonstrating improved balance between the quality and diversity of generated text.
The authors conducted experiments across three datasets, yielding compelling results that underscore the effectiveness of min-p sampling. The findings indicate that this method not only enhances the quality of text generation but also fosters greater diversity, which is a critical aspect in natural language processing tasks.
The writing in this paper is clear and accessible, making the concepts relatively easy to understand. The methodology is straightforward and provides a meaningful contribution to the field. Overall, this paper presents insights and a potential solution to the challenges of text generation, which may be of interest to researchers and practitioners.
Strengths: 1. The paper is well-written and easy to follow.
2. The proposed min-p sampling makes an effective balance between coherence and diversity in text generation.
Weaknesses: Please refer to Questions.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: Simple but effective and highly influential contribution to LLM research
Strengths: This paper presents compelling evidence that its single contribution, min-p sampling, is highly effective.  The usage of it in 54,000 Github repositories alone is very impressive.  In addition to that, they produced theoretical reasoning why their method works, LLM-generated statistics with explanations about how to interpret these statistics, additional statistics which involved human participants, examples of seeing how the logits are transformed under different distributions which give additional insight into why this method is better than existing methods, and code to try out the method.  It is a very simple paper, but it clearly makes the case for its own importance.
Weaknesses: The one contribution of this paper, min-p sampling, is extremely simple and not mathematically "deep" at all - no theorems were presented, and the code implementation literally (was provided and) took less than one page.  However, I think that having such a paper in a conference proceeding is not a bad thing.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 4
reviews: 
Comments: [UPDATE] Based on the rebuttal I have increased my score (8->10), but kept the rest of the review unchanged

The authors propose a new sampling mechanism, which is a minor but important twist to the popular nucleus-sampling (`p`). Instead of having a fixed threshold `p`, this proposal has `p` be dependant of the probability of the most probable token. The intuition is that in cases that the model is confident only few tokens are kept as support set, while that set is extended when the confidence is low
Strengths: * Sampling is one of those areas were the model per se needs to be complemented with an outside algorithm, allowing for creativity on how to set this up. This work proposes an original twist to a popular choice

* The proposal is simple, appealing and 

* has good empirical results, both as measured on benchmarks and (more important) by adoption of the community
Weaknesses: The new 10p limit has not been handled wisely in my opinion, and the paper could do more with less text. In particular, Sect 4 could be removed without much loss to the overall apper

Having experiments on a 123B has to be commended. The paper would be stronger however if the authors could show that the results hold on different model families (eg, llama and mistral), as otherwise it is not clear if this method provides gains on one family only
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 3
reviews: 
Comments: This paper proposes the Min-p Sampling method, which dynamically adjusts the probability threshold based on the model's confidence level. This method aims to enhance creativity without sacrificing coherence. The method is validates through experiments on benchmark and human evaluations, showing better coherence and diversity compared to other sampling methods. The method has been widely adopted in the open-source community.
Strengths: - New sampling method: This paper proposes the Min-p Sampling method for better control over the diversity of generated outputs compared to fixed threshold methods like top-p.

- Conducted experiments: The authors conducted experiments across tasks, ablation studies, and human evaluation.

- High reproducibility: The author released the implementation, code, and repo with implementation guidelines, which enhances the reproducibility.

- Wide Applicability: The proposed method can be easily integrated with existing open-source LLMs, and the authors show the broad potential applications that can be applied.

- The ablation study shows that min-p sampling is barely impacted by the output length, which is interesting.
Weaknesses: - The experiment is limited to Mistral models and fails to demonstrate applicability with other models. It would be more comprehensive and interesting to see results from additional models, such as LLaMA3.

- The effectiveness of min-p sampling highly depends on the base probability thresholds. As shown in Table 6 (ablation study results), the choice of thresholds significantly impacts LLM performance. This indicates that optimal performance requires careful tuning, which could limit the method’s potential effectiveness and ease of use in applications.

- The paper claims that the experiment is intended to demonstrate that min-p sampling balances creativity and coherence (line 290); however, metrics relevant to creativity are missing. Diversity is not enough for creativity assessment. LLMs-as-judge approach is widely used for creativity assessment. Please consider adding such an experiment.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: Following the author rebuttal, I have revised my assessment upward (from 8 to 10), though the remainder of my evaluation remains consistent. The paper introduces an elegant modification to the standard nucleus-sampling approach (`p`), wherein the threshold becomes adaptive rather than fixed—determined by the confidence level of the model's highest-probability token. The central idea is intuitive: when model confidence is high, fewer tokens are selected from the candidate pool; conversely, when confidence is low, the pool expands accordingly.
Strengths: * Sampling represents a domain where algorithmic innovation can meaningfully augment base model capabilities, creating opportunities for creative methodological approaches. This contribution offers a novel variation on an established technique. * The proposed method is straightforward, elegant, and demonstrates solid empirical performance across both benchmark evaluations and—more notably—real-world adoption metrics from the research community.
Weaknesses: * The introduction of the 10p constraint appears suboptimal in execution, and the manuscript would benefit from greater conciseness overall. Specifically, Section 4 could be omitted without substantially compromising the paper's contributions. * While the evaluation on a 123B-parameter model deserves recognition, the work would be significantly strengthened by demonstrating generalization across diverse model architectures (such as Llama and Mistral variants). Without such evidence, it remains uncertain whether the proposed method yields improvements exclusively within a particular model family.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 4
Clarity: 3