Improving Diversity in Language Models: When Temperature Fails, Change the Loss
Authors: Alexandre Verine
Florian Le Bronnec
Kunhao Zheng
Alexandre Allauzen
Yann Chevaleyre
benjamin negrevergne
Venue: ICML 2025 poster (2025)
Keywords: Language Models
Diversity
Precision
Recall
Temperature
Primary area: deep_learning->large_language_models
TL;DR: We propose losses to improve diversity in language models to compensate for temperature failing.

Abstract: Increasing diversity in language models is a challenging yet essential objective. A common approach is to raise the decoding temperature. In this work, we investigate this approach through a simplistic yet common case to provide insights into why decreasing temperature can improve quality (Precision), while increasing it often fails to boost coverage (Recall). Our analysis reveals that for a model to be effectively tunable through temperature adjustments, it must be trained toward coverage. To address this, we propose rethinking loss functions in language models by leveraging the Precision-Recall framework. Our results demonstrate that this approach achieves a substantially better trade-off between Precision and Recall than merely combining negative log-likelihood training with temperature scaling. These findings offer a pathway toward more versatile and robust language modeling techniques.

Improving Diversity in Language Models: When Temperature Fails, Change the Loss
Alexandre Verine". 1 Florian Le Bronnec 23
Kunhao Zheng" 24
Alexandre Allauzen Yann Chevaleyre? Benjamin Negrevergne"
Abstract
models. Namely, several works have proposed to study two kind of errors these models can make (Pillutla et al., 2021; Shao et al., 2017; Le Bronnec et al., 2024):
Increasing diversity in language models is chal- lenging yet essential objective. A common ap- proach is to raise the decoding temperature. In this work, we investigate this approach through a simplistic yet common case to provide insights into why decreasing temperature can improve quality (Precision), while increasing it often fails to boost coverage (Recall). Our analysis reveals that for model be effectively tunable through temperature adjustments, it must be trained to- ward coverage. To address this, we propose re- thinking loss functions in language models by leveraging the Precision-Recall framework. Our results demonstrate that this approach achieves substantially better trade-off between Preci- sion and Recall than merely combining negative logelikel training with temperature scaling. These findings offer pathway toward more ver- satile and robust language modeling techniques.
1. The LM generates texts unlikely under the true distribu- tion of language, leading to outputs of limited quality. 2. The LM focuses on producing limited set of patterns or phrases, thereby reducing diversity in the outputs.
The analysis of these errors in generative models has been formalized through the Precision-Rech (P&R) framework, introduced by Sajjadi et al. (2018). Precision measures the proportion of generated samples that are plausible under a reference distribution, thereby assessing the quality of the model. Recall quantifies the proportion of the reference distribution that is covered by the generated samples, reflect- ing the model's coverage. Recall captures more than just sample diversity, it evaluates the diversity of samples that are likely under the reference distribution. This contrasts with metrics that operate independently of the reference distribution, such as those based on entropy (Theis et al., 2016; Friedman & Dieng, 2023) or vocabulary distinctive- ness (Zhu et al., 2018), which may indicate high diversity even for random-liks samples.
1. Introduction
The P&R trade-off is crucial and varies by application do- main. In code generation, high Precision is essential for producing runnable code, whereas in open-ended creative tasks, high Recall is key generating diverse and engaging content. Independently of the P&R framework, the need to tune this trade-off has motivated the development of variety of post-training corrective methods (Holtzman et al., 2020a). Among them, one of the simplest and most widely used methods is to adjust the decoding temperature (Fan et al., 2018: Zheng et al., 2024). Increasing the temperature directly increases the entropy i.e., the diversity of the model, however its impact the Recall is not well understood. In this work, aim to answer the following question:
Autoregressive language models (LMs) (Bengio et al., 2000) have demonstrated impressive capabilities modeling nat- ural language. By scaling both the data and the number of parameters, current baadedd approaches (Touvron et al., 2023; DeepSeek-AI, 2024) are now able produce expressive language models, capable of generating texts close to human-written ones. With these advances, evalua- tion has become a critical issue, and new metrics have been developed to better assess the generative abilities of these
'Equal contribution 'École Normale Supéricure, Université PSL, DIENS, Paris, France Miles,, LAMSADE, Université Paris- Dauphine-PSi. Paris, France 'Sorbonne Université, CNRS, ISIR, Paris, France 'Meta FAIR, Paris, France. Conresoondence to: Alexandre Verine <alexandre, verine @ @n,,.fr>, Florian Le Bron- nec <fforian.le- bronnne dauphine, dauphine.psl.eu>.
Question 1: What is the impact of adjusting the temperature of a LM in terms of Precision and Recall?
We address this question by showing that temperature ad- justments have a mixed impact on the P&R trade-off. While lowering the temperature can improve Precision, increasing
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

When Temperature Fails, Change the Loss
often reduces Recall, contrary the common intuition that higher temperature improves diversity. Based on this observation, argue that achieving a better P&R trade- off through temperature scaling requires training the model to prioritize Recall. This naturally leads to the following question:
Notations:
For clarity, we omit the dependency on the parameters 0 when it is clear from the context and use Q instead of Qs. When the parameters are fixed, meaning the gradient is detached, we denote this by Q, implying that
v VoQ = O. Probabilities conditioned on the context a:e are denoted as Pa(+) = P(- x<a) and Qa(-) = Q(-1 xxx).
Question 2: How can we train models for a higher Recall?
2.2. Temperature and sampling
We address Questions 1 and 2 by making the following contributions:
Once the model has been trained, it can be used to generate new sequences by repeatedly sampling each token žt from Qet using Øet as the context (i.e., xt ~ Q(- (<()). How- ever, instead of sampling directly from Qet. practitioners generally introduce new distribution Qu which is based on Qet. but features an adjustable temperature parameter t used to increase the entropy of Q'a and generate more diverse samples when necessary. For a given context Let. the temperature-adjuser distribution Q is defined as:
Impact of Temperature Scaling: Section 4 presents The- orem 4.2, which analyzes the effect of temperature scaling on the P&R trade-off. We further refine this analysis in Proposition 4.3 by examining specific artificial cases. Recall-Oriented Loss Functions: In Section 5 we show that three existing loss functions fit in the P&R framework and optimize for Precision. We build upon this and intro- duce modified versions of these losses to enhance Recall. Empirical Trade-off Evaluation: In Section 6, we em- pirically validate our theoretical findings. Notably, we demonstrate that in most experiments, our proposed losses improve Recall and can lead to a model that achieves better P&R trade-off through temperature scaling.
Qa(x) Σ Qa(a;) 1EEVL
(z(())=
(3)
Note that setting the temperature recovers the original distribution Qet. Setting the temperature any value >> increases the entropy of Qt (compared to Qa), and Q' becomes increasingly close to uniform distribution over V as t + co. Conversely, setting the temperature to any value << 1 decreases the entropy of Q' leading to a deterministic distribution Q%2 for t =
Our study suggests that focusing coverage at the training stage can produce language models that are more versatile with temperature scaling.
2. Background
2.1. Generative Models
2.3. Precision and Recall for Generative Models
In language models, consider sequences of L tokens, (Z1,..., xL) VL where V is the vocabulary set of cardinality V. We denote P()") the set of all probability distributions over the sequence of L tokens. The objective of training a generative model is to approximate the true data distribution P by learning a parameterized distribution Qa P P(V2), modeled as a product of probabilities conditional to preceding tokens Vcl == (x1, .1,,,......1)..
In practice, increasing the temperature of the model distri- bution Q' increases the entropy of the generated samples, but it does not always lead to better coverage of the target distribution P. To properly assess the effect of temperature scaling, we adopt Precision and Recall (P&R) metrics for generative models. Inspired by classification, an intuitive definition of these metrics for generative models was intro- duced by Kynkäänniemi et al. (2019), based on comparing the supports of the two distributions, as follows.
- = xa), 1=1
(1)
Definition 2.1 (Precision and Recall (Kynkäänniemí et al., 2019)). Let P, Q E P(VZ). The Precision ā and Recall 3 of Q with respect to P are defined as:
where Qz( IZ4 denotes the conditional probability of token 27 given the preceding tokens Vet. modeled as a soft- max distribution over the output of a neural network Fo. parameterized by O. This network is trained to minimize the negative log- likelihoooo (NLL) of the data:
= Q(Supp(P)) and 3= P(uppp(()))
(4)
Computing these metrics requires estimating the supports of the distributions, which can be achieved through sampling followed by k-nearest neighbors support estimation. This approach is practical and has been widely used to assess the quality and coverage of both image generative models and language models (Kynkäänniemi et al., 2019; DeVries et al., 2020; Song et al., 2023; Le Bronnec al., 2024).
L (f) = EEg- Σ log Qo (Z1 @0 (aal
(2)
The NLL optimization problem is equivalent to minimizing DKL (p) the Bullback-Leible divergence between the true distribution P and the learned distribution Qo.
2

---

When Temperature Fails, Change the Loss
Distributions
this trade-off, some studies have explored modifications to the sampling process (Brock et al., 2019). In the case of lan- guage models, adjusting the decoding process is a common strategy (Fan et al., 2018), with some methods explicitly targeting improved diversity (Chang et al., 2023). We focus specifically on the temperature parameter, whose limitations have been highlighted in multiple studies (Peeperkorn et al., 2024; Holtzman al., 2020b), mainly from an empirical perspective.
P
0.4
02
A -0.1
0.0
0.2
0.1
0.
8x
Tokens
Beyond inference, various training methods have been de- veloped to address this trade-off. For instance, f-divergence minimization has shown promising results in image genera- tion (Nowozin et al., 2016; Grover et al., 2018), with Verine et al. (2023) explicitly framing their approach within the P&R paradigm. In text generation, several studies have ex- amined the role of different f f-divergences in RLHF (Wang et al., 2023; Go et al., 2023; Sun & Schaar, 2024), either for reward modeling or regularization. For autoregressive mod- els, various works have proposed training with modified loss functions (Ji et al., 2023; Kang & Hashimoto, 2020; Pang & He, 2021). While introduced with different motivations, we later unify these methods within the P&R framework and show that they all effectively maximize surrogates of Precision.
Figure 1: Examples of PR-Curves. Q1 has a higher Pre- cision than Q2 but a lower Recall. Values of (x),8x) are plotted for X E 1,
While convenient compute, these metries have theoretical limitations, particularly in the case of tempered distribu- tions, where adjusting the temperature does not alter the support of the model distribution. To alleviate these limita- tions, Sajjadi et al. (2018) have introduced an extension of Precision and Recall that compares the densities of P and Q instead of simply comparing their support. To achieve this, they introduce new parameter À [0, oo), which helps identify regions where Q is at least À times denser than P. When À is high, the focus is on regions where Q assigns probability mass while P does not, which characterizes loss of Precision. Conversely, when À is low, the focus shifts to regions where Q assigns very little mass while P does, which reduces the Recall, The parameter À thus controls the trade-off between Precision and Recall, allowing the definition of the PR-Curve.
4. Temperature and PR-Curves
In this section, we analyze the impact of temperature on P&R. First we express general bound on Precision and Recall function of temperature. Then, in order to gain deeper insights, we craft simplified realistic distribution, and characterize how Precision and Recall change as we increase the temperature of this distribution.
Definition 2.2 (PR-Curve-(Sapppp (Sajaadi et al., 2018)). Let P.Q P(V²). The PR-Curve is the set PRD(P, Q) de- fined as:
PRD(P.Q) (0((P:))=((a,,3)11))) ((ax,33))Ae((,ooo)
(5)
General case: Our first result relies on the concept of sparsity defined as follows.
where:
Definition 4.1 (Sparsity of distribution). Given context Del E 1 the sparsity of a distribution Pei is defined as Supp Pa)l/V. We say that distribution is sparse when Suupp Pa)l/V << 1. More generally, the sparsity of a distri- bution P over sequences in V is defined as [Supp( P))//**
ax(1)()) = Σ min (AP(x),
æept
and BA(P]QQ=
Σ min min (iix)))) ztpτ.
Theorem 4.2 (Impact of sparsity on P&R). Let P, P,Qu= e
The two definitions are related: a.. = ā and Bo- = B. The values of ary for high X captures the quality, while the values of Bx for low À captures the diversity with respect to the true distribution. Figure shows examples of PR curves.
P(V"), then for any temperature t, we have:
ax(1))) < [Supp( (P) 21/e
(8)
VI.
and 3x(P1Q)) 5 |Supp(P) e
2Lft
(9)
3. Related Works
VL
where Z = max le(1...),, i.jev Fo(xa)) Fe (x(2))) the highest difference between the logits of the model.
With the rise of models exhibiting remarkable generative capabilities and the need to assess both their fidelity and diversity, several works have demonstrated the versatility of P&R metrics for developing and evaluating generative models (Sajjadi et al., 2018; Song et al., 2023). To balance
The proof of this theorem is provided in Appendix A. We observe that the sparser the target distribution, the harder it is for Qo to achieve good Precision and Recall.
3

---

When Temperature Fails, Change the Loss
a/pk- 1/K
P(-(m)))
1/K (1 - c)/K*
P(-lass))
b/pk
Qa(-(aria)
</(V K)
20
10
60
80
100
20
10
60
80
100
(a) Example of P( mre,) ) and @- m.,))
(b) Example of P( sxa,)) and Qo(: met,))
Figure 2: Example P and Qo for V = 100, K = 20 and a/p- = 1.45, and c= 0.15. K/V represent the sparsity of the target distribution, p the proportion of tokens underrepresented the level b/p Qo 4, is the noise level at 12-
In practice, the true distribution Par is sparse for most con- texts because of the number of grammatical, syntactic, or semantic rules that apply and force Pat to be null for most tokens. In contrast, the model distribution Qui is not sparse due to the softmax normalization, thus it will assign nonzero probabilities even for unacceptable tokens. Intuitively, in- creasing the temperature will increase the probability of these tokens and introduce a disproportionate number of unacceptable generations, effectively unlearning the lan- guage rules, and reducing both Precision and Recall. We empirically estimate the sparsity of reference distributions in Section 6.
Proposition 4.3 (Informal-P&R with temperature). Let P.0. P(V") as described in the previous paragraph.
For high values of X, the Precision ox decreases the temperature increases.
For low values of X, the Recall (3x decreases for tem- peratures 2 to provided that any one of the following mild conditions is met:
the vocabulary size V is much larger than the target support size K (K << V), Qo good approximation of P, (i.e., a 1,b 1= c< 1).
To gain deeper insight the im- pact of increasing the model temperature on the PR- FR-Curve, we analyze a specific case. We choose a target distribu- tion P where all Pei are sparse uniform distributions over small subset of K tokens, and model distribution Qo that matches P everywhere except the two specific positions in the sequence and 12, i.e, Va e VL, Vig (4, 12), Qo(-) a P. æt.)) For position I and 12, the conditional distributions are defined as follows:
Artificial Case Analysis:
Further details about To The formal theorem, its proof and details on to are available in Appendix A.2.
For Recall, a common behavior an initial increase with rising temperature, peaking at some value to Beyond this point, Recall tends to decline and eventually converges to level that is consistently lower than the value at t = 1. To ob- tain a model capable of achieving both higher Precision and higher Recall through temperature scaling, it is beneficial (10) to train the model with a strong emphasis on Recall. Pre- cision may then be improved by lowering the temperature. This motivates the objective of the next section: developing training strategies that produce diverse set of models.
if < <K,,
pk:
if pK S K, otherwise, if P.),
pk
1.
I-c
(x xel1))
(11)
otherwise.
V-K
5. Training model for diverse output
This is illustrated in Fig. 2 (a) and (b) respectively. This simple setting allows full characterization of the PR- Curve across different temperatures while remaining the- oretically meaningful. Notably, we can show that the inequalities (8) and (9) are tight for high temperatures f Z + +log (oo)-oo demonstrating that Qa achieves the optimal Precision-Reddd exact expressions for the PR-Curve and its rate of change are provided in Appendix A, Prop. A.2 and A.4. Connec- tions to world behavior are discussed in Section 6. The results can summarized the following proposition:
Training models for specific Precision-Recall trade-offs has been explored in image generation, notably by minimizing alternative f-divergences (Verine et al., 2023). However, these methods depend on assumptions that are incompatible with the causal structure of language generation, rendering them unsuitable for direct use in language models. Instead, existing methods adapt the training loss to weight samples differently from the standard negative log-likelihood (NLL) loss. In this section, we focus on three representative meth- ods: Trunc (Kang & Hashimoto, 2020), GOLD (Pang
tradeoff in this regime. The

---

When Temperature Fails, Change the Loss
following loss:
0.14
Trunc
1.0
c- c-Dio weigg
C C (0) = -Be. Σ (Q())28) log R (12)
L
A-PR
High
0.8
GOLD by Pang & He (2021). GOLD (Generation by Off-policy Learning from Demonstrations) is a method that leverages off-policy learning to improve the quality of gener- ated samples. One specific variant of this method, GOLD-6, has been shown be equivalent to reweighting the gradients of the training samples' log- oggliiklihhod, thus further promot- ing highly probable tokens (Li et al., 2022). The authors propose the following loss:
0.6
TruncR
0.2
Low
High a
Low
0.0
Bx
0.2
0.4
LGOLD( -Eg-- Σ Qu(m)) log log Qultt)) (13)
Figure 3: Illustration trade-offs targeted by each method over the model's PR curve (black). The highlighted parts represents regions where two c-Div loss functions of oppo- site effects place more emphasis, red indicates high empha- sis, blue low. Trunc and TrancR target specific point the PR-Curve, either maximizing it vertically (Precision) or horizontally (Recall), while X-PR focuses on improving a point along the = Ar line.
Tailr by Ji et al. (2023). Tailr (Total Variation Guided Language Generation) method that aims to minimize the TV distance between the model distribution and the target distribution. To do so, the authors first propose an upper bound on the TV distance based the total variation of the conditional distributions. They then approximate the unknown conditional target distribution Pe as mixture of the model's 's own distribution and one-hot distribution:
Definition 5,1 (7--rrxy distribution). Let 2ę Pat be to- ken sampled from the conditional target distribution. Given E [0, 1]. the 7-proxy distribution is defined as:
He, 2021), and TaiLr (Ji et al., 2023). Although originally introduced with different motivations, we show that these methods consistently prioritize Precision over Recall. To address this imbalance, we propose alternative loss func- tions designed to enhance Recall. All theoretical proofs are provided in Appendix B. In the next section, we experimen- tally evaluate which methods are most effective in making models (1) more diverse and, most importantly, (2) more versatile.
P ( (1-)==)=))
(14)
This 7-proxy is used to derive the following loss:
Σ Qa(())
L
LTaLe (0) -R
log (z)
[i-] 7+ (1-7)Qa(x;)
(15)
All losses discussed in this section can be viewed as reweighted variants of the NLL., with weight computations detailed in Appendix C. Figure 3 illustrates the trade-off's each method targets along the model's 's PR curve.
5.2. From Trunc to TruncR
The Trunc method considers only subset of the target distribution, meaning it does not penalize the model for missing samples outside this subset. As a result, it inherently allows for a loss of Recall. More precisely:
5.1. Baseline Methods
Trunc by Kang & Hashimoto (2020). This approach de- fines a target distribution ptrune, obtained by renormalizing the original distribution P over a subset Xtrune X, satis- fying P(Kerune) = 1 - for a given constant A e [0, 1]. The method minimizes the NLL between this truncated dis- tribution and the model distribution, resulting in a tighter upper bound on the total variation distance. In practice, the samples with the highest
Proposition 5.2 (Trunc optimizes Precision at a given Re- call). Optimizing using the A loss is equivalent to optimizing Precision for fixed value of Recall B = 1- A.
This result is derived by minimizing the NLL between the truncated P and Q. To reverse the trade-off, one might con- sider inverting P and Q in the loss. However, this approach is infeasible requires sampling from Q and evaluating P(-(az)). To address this, we introduce a loss function that approximates minimizing the NLL between P and Qtrune.
model is trained exclusively fng-likelihood values. This is done by selecting samples within the top 1 - A quantile of the log-likelihood distribu- tion, i.e., dynamically identifying the threshold 8 R such that Emz. (Q.(=))2)] 1 A and then minimizing the
C CrrnER (0) =- -Ex-P
Σ 1 (Q.(a))6) log Qa(ar) 1-1
(16)
5

---

When Temperature Fails, Change the Loss
such that P (Qu(=)ss)] proach requires the same quantile approach to estimate the threshold 5. We can show that this proposed loss achieves the opposite effect of the Trunc method:
A. Note that this ap-
of less probable tokens, likely increasing Precision. We pro- pose to generalize this reasoning and propose the following non-proper loss function:
Proposition 5.3 (TruncR optimizes Recall at given Preci- LI-A sion). Optimizing 0 using TruneR is equivalent to optimiz- ing Recall for a fixed value of Precision a = - A.
(0) -Bag-P
Σ log (19)
where
Training for a limited subset of the model distribution is not new idea. In fact, Verine al. (2024) train image generative models on limited subset of generated samples and also observe that this leads to increased Recall.
Qc((11)
(Qu(o))s (1) ( y+ (1 (1-7)Qa((1))
and 6AME I-(1-))AVE. We show that under the same assumptions than Ji et al. (2023), this loss can be used to optimize any point (x) on the PR-Curve for À 1:
5.3. From GOLD to c-Div
By reweighting the loglleilihood of training samples, the GOLD method increases the likelihood of highly probable tokens. However, this reweighting can be generalized:
Theorem 5.6 (A-PR optimizes the PR-Curve A). Opti- mizing 0 using the LX-PR loss is equivalent maximizing lower bound on Bx) on the PR-Curve for X S 1.
Theorem 5.4 (Conditional Tsallis a- D-Divergence Minimiza- tion). Using Definition 5.1 with y 1, minimizing the conditional a-divergence between Pa and Qei for all LE (L.....,L) and for all x ~ P is equivalent to minimizing
This loss generalizes Fast. any point on the PR-Curve for À s 1. Notably, for X 1, the loss is equivalent to the TaiLr loss, as 8^ = 1 and À = 1. For X < 1, the A- PR loss retains the effect described in Proposition 5,5 but additionally enforces the likelihood to remain below 8xM/.
CeDiv( =- - EED-P Σ
1/
lol log Qa((i)) (17)
6. Experiments
In particular, GOLD minimizes L.Div (0) with a= = 1/2. It is well established that adjusting the a parameter in the min- imization of a-divergence significantly influences the trade- off between quality and diversity (Minka, 2005; Labeau & Cohen, 2019; Go al., 2023). The relationship between a-divergence and the PR-Curve has been analyzed in Verine (2024). Higher values of a lead to divergences that are more mass-covering, favoring Recall. In particular, optimizing a divergence with a > 1 results in a more Recall-oriented approach compared to NLL.
In this section, we empirically assess the theoretical insights on temperature scaling (Section 4) and training methods (Section 5). Our experiments aim to address the following questions:
To what extent do our simplified theoretical settings align with real-world language modeling scenarios?
What is the impact of temperature scaling on the Precision-
Recall trade-off in language models?
How do the proposed training methods affect Recall?
5.4. From TaiLs to X-PR
To answer these questions, we conduct experiments on mul- tiple tasks at different scales.
The TaiLr method aims to optimize the Total Variation (TV) distance, which corresponds to the point (PIQ)/2 for A 1. However, due to practical constraints, the proposed loss is not a proper loss function and does not directly opti- mize the TV distance. Instead, it solves a different problem: Proposition 5.5. The optimal distribution Qø- using the Tail.r method, ie., CTaIL (0) with 7 > O, is given for every context Jet 1-- by:
6.1. Evaluation Tasks and Metrics
We consider four tasks where quality and coverage can be measured in a meaningful way: two tasks of code gener- ation, integer multiplication, and open ended generation. Full details regarding the different evaluation methods are detailed in Appendix D.2.
(2(==((1-+VV))
V V, (x|
(18)
ConeContests (Li et al., 2022). We use the test set com- prising 165 challenging problems. We propose to evaluate P&R using the widely used @ metrics, (Chen et al., 2021). which is the expectation that at least one code sample is correct given a budget of k samples. In this setup, we naturally consider pass@1 1 as a proxy for Precision, since it measures exactly the portion of the outputs generated by
1-7
In other words, Qa(x) > Pa(x) if Pa(x)) > 1/V, and @x(xx) & Pa(x)) otherwise.
The loss CisaLo is not a proper loss since its optimal distribu- tion is not P. However, it effectively reduces the likelihood
6

---

When Temperature Fails, Change the Loss
Precision
.... Liama 3.1 8D Instruct
0:30
RLEF Llama 3.1 8D Instruct
Precismm
1.0
Llana 3.1 70B Instruct
RLEF Llama 3.1
68
34-
00
0.-
0.15
6:
0.20
6.7
0.30
0.40
Recall
10
10
700
0.6G
0.30
Figure 4: Effect of tuning& & between 0.2 (smaller dots) and 2 (larger dots) on P&R for Llama3.1 models on the CodeContests dataset.
Figure 5: Effect of the temperature on the P&R for the integers multi- plication task, at different levels of underrepresentation in the training data. Recall increases with the temperature, then drops.
the model that correct. Then, the boost from pass @ I to pass@ k depends largely on the diversity of the code sam- ples, which is why we use pass@ 100- pass@) 1 as Recall.
When training models over this dataset, simulate un- derrepresented integers by adjusting their frequency in the training data, mimicking imperfect learning by Qa. Specifi- cally, the first digit of the first token is drawn with proportion from the first five digits and 1 b from the last five. This setup closely matches the artificial cases, allowing us to validate their relevance empirically. As the reference distri- bution is known, we can compute the P&R metrics directly.
MathQA-Python (Chen et al., 2021). We evaluate the P&R trade-off on the MathQA-Python dataset, which con- sists of simple Python code generation tasks. We use the same evaluation as in CodeContests, using pass metrics.
Integers multiplication. The goal is to generate pairs of positive two-digit integers along with their product modulo 97, in the format az x b, C2 (Papadopoulos et al., 2024). The reference distribution uniform over the in- put integers and deterministic over the result. The model Qe trained on synthetically sampled examples from P. Although this example is simple, we believe it models key characteristics of real-world patterns in natural language, where certain words exhibit spread probability distribution or highly skewed distributions depending on the position.
Writing Prompts (Fan et al., 2018). This task involves generating creative stories from given prompts. We analyze how different training losses and temperature settings im- pact the learned distribution Qp in generating text. Using the P&R metries from Le Bronnec et al. (2024), we assess the trade-offs between fidelity and diversity in text generation.
6.2. Models
We use the following models in our experiments: Llama3.1- 8B/70B Instruct/RL.E (Grattafiori et al., 2024) general instruction-tuned models. These models are used for the CodeContests generation tasks and for the support sparsity estimation. Olmo-1B (Groeneveld et al., 2024), is smaller pre-trained model. We finetune this model on the Writing- Prompts and MathQA-Python datasets using the proposed losses. Llama3.2-3B is finetuned on the WritingPrompts dataset. Llama-Alpaca is Llama3.1-88 instruction tuned the Alpaca dataset (Taori et al., 2023). This model is used to WritingPrompts and MathQA-Python tasks. For Llama3. 1.......BBBBBB 1-8B and Llama3.2-3B we omit Trunc and TruncR results due to incompatibility of the original code with multi- GPU parallelism, which is required for training. Further training implementations are detailed in Appendix D.
Precision
2.00
==
1.50
1.25
0.5-
1.00
0.75
0.50
Recall
0.00
0.5
6.3. Assumptions for the Theoretical Limits of P&R
Figure 6: P&R for e [0.1,5] 5] for the integer multiplication task for models trained with cDiv for different a. By tuning t on models with high Recall, we improve the P&R trade-off. The PR-Curves for high dominate the curves for lower
Sparsity of P. Theorem 4.2, we establish the limit of the PR-Curve as a function of the sparsity of P. To gain a practical insight on the actual sparsity of P. we compute an
7

---

When Temperature Fails, Change the Loss
Table I: P&R of Olmo-1B trained on WritingPrompts dataset. Values higher than NLL. are highlighted in bold. We report the MAUVE (Pillutla al., 2021) score for reference.
Prectsion
Proxssion
Protsion
Method
MAUVE
P
R
NLL
0.104
81.4
8.9
Than-R 1223
Thac. 3 0:25
eDat
GOLD
J.B Ary 021
Tust
Trunc
0.074
85.5
8.9
Recall
Recall
Recall
(2 Tunne
0.073
85.5
9.3
D0C
1.10
600
023
1000
RE
(a) NLL vs. TruncR
(b) NLL vs. C-Div
(C) NLL A-PR
GOLD
0.005
99.3
0.2
Figure 7: P&R trade-off for t E [0.1, 1.8] for various training methods on the WritingPrompts dataset with Olmo-1B Existing losses such as Trunc, GOLD and TaiLr tend to favor Precision over Recall. In contrast, our proposed losses improve Recall while maintaining comparable Precision when adjusting the temperature.
la 1.40
0.068
54.2
11.9
TaiLr
0.087
83,9
9.3
7-100
X-PR 7-
0.096
81.3
12.6
estimate of the sparsity of the distribution for two datasets: CodeContests and WritingPrompts. Our estimation is based number of unique tokens within the support of the condi- tional distribution P(- 22))). We approximate this by con- sidering the truncated conditional distribution obtained from strong pretrained model Pa. specifically counting the to- kens that constitute the top-p probability mass of (- xz1). Full details are provided in Appendix D.2. Our experiments reveal significant sparsity in the conditional distributions relative to the vocabulary size V. Specifically, for Code- Contests, the estimated support size less than 0.1% of V when p= 0.9, and remains under 8% when p 0.99. These observations confirm the high sparsity of the target distribution, empirically validating that the PR-Curve of Qo deteriorates significantly when >>>> 1.
Lowering the temperature improves Precision but re- duces Recall. In Figure 4, Llama3.1-8B RLEF is the only set-up when the temperature has no effect on the Precision. Across all other tasks, decreasing the temperature (1 < 1) leads to higher Precision, but always at the cost of lower Recall. This aligns with our analysis in Section 4.
Increasing the temperature has limited or negative impact on Recall.
As the temperature increases, we observe the expected be- havior in the toy multiplicative task: Recall initially im- proves, but beyond a certain threshold, it begins to decline and eventually drops to zero. In the more complex Writ- ingPrompts and CodeContests tasks, this effect is less pro- nounced: Recall exhibits little or no improvement before ultimately decreasing. These observations are consistent with the theoretical findings in Section 4, and in particular with Proposition 4.3, which shows that while higher tem- peratures may temporarily enhance Recall, they eventually cause it to decline to zero..
Connection with artificial case. These observations sug- gest that the artificial case in Theorem 4.2 captures features relevant to practice: the target distribution P is highly sparse, and the pretrained model, viewed as Qe. concentrates mass on few tokens, with low residual spread across the rest, reflecting the theoretical structure.
6.5. Training for Recall
Figure 8 presents the P&R of the integer multiplication task for different training methods at different parameter settings. Table and 3 presents the P&R of various models trained with different losses on the WritingPrompts and MathQA dataset. As predicted, baseline losses favor Precision at the expense of Recall, whereas our propused losses success- fully improve Recall. This is consistent with our theoretical analysis in Section 5.
6.4. Effect of Temperature on P&R
Results are shown in Figures 4, 5, 6 and 7. Results with other decoding methods are discussed in Appendix D.3.
Table 2: Sparsity of the target distribution (approximated upper bound on (Supp(P)//V for various top-p truncation thresholds of the approximate distribution, the CodeCon- tests and WritingPrompts datasets.
6.6. Enhaneing the Temperature P&R Trade-off
0.9
0.95
0.99
Recall-optimal point. On all tasks, when the baseline NLL is temperature-tune to maximize Recall, we can tune our methods to reach superior Recall level with the same Precision. This is verified on Figure 6 for integers multipli-
Task
CodeContests
0.03%
(.08%
8.08%
WritisgPrompts
3.86%
7.80%
24.9%
8

---

When Temperature Fails, Change the Loss
Pression
Precision
Precióon
Tlmc
Al APR
015
Rocall
Tecall
Rocall
000
(a) Trunc & TruncR
(b) GOLD & C-viv
(c) TaiLr & A-PR
Figure 8: P&R trade-off at t= 1 with the proposed losses on the integer multiplication task. Baselines Trunc, GOLD) and TaiLr improve Precision. In contrast, our proposed losses improve Recall.
cation, on Figure 7 for the WritingPrompts dataset. This also verified on MathQA dataset on Table 4a, where we report ---ii-trained models achieving a higher Recall than NLL at the same Precision level.
perature scaling more effective in balancing P&R. However, we observe that these loss functions can be less stable than NLL in practice.
Precision-optimal point. We identify scenarios in which our approach consistently achieves higher Recall across the entire spectrum of Precision levels attainable by NI.L. This demonstrated on Figures 6 and 7b, and for Llama-Alpaca, with the c-Div loss in Table 4b.
7. Conclusion
Our study highlights the limitations of temperature scaling as a mean to improve diversity in language models. While lowering the temperature enhances Precision, increasing often fails significantly boost Recall. Through theoretical analysis and empirical evidence, show that models must be trained explicitly for Recall to make temperature tuning more effective. To this end, we propose alternative loss func- tions that achieve a better trade-off between Precision and Recall. Our results suggest that refining training objectives is more effective approach than relying solely on decoding strategies, and can make models more versatile.
Figures 6 and 7 illustrate the evolution of P&R as tem- perature varies. Beyond improving Recall at = 1, our proposed losses achieve better overall P&R trade-off. Notably, they allow for slightly better Precision at the same Recall level as NLL by adjusting the temperature, but also allow for significantly better Recall at the same Precision level. This suggests that training for Recall make
Table 3: Comparison of training methods on the WritingPrompts on MathQA-Python datasets. All generations were sampled with temperature / = 1, except for Llama-Alpaca, where a lower temperature / = 0.5 was used to mitigate degenerate outputs.
Task Model
WritingPrompts Alama-Alpace
MathQA-Python
Olmo-1B IB
Llama3.2-3B
Olmo- IB
Llama-All Hama-Alpaca
Method
R
R
NLL
81.4
8.9
83.3
4.4
77.4
8.1
42.0
36.6
8.8
39.0
Trunc-R
A 0.25
85.5
9.3
A-01
29.8
43.3
e-Div
1.4
54.2
11.9
a-14
82.2
12.6
1-3
72.5
17.5
a 1.4
30.1
46.1
a 1.4
8.2
43.4
A-PR
0=0.1
81.3
12.6
0=
57.1
26.9
0
59.9
19.0
0=0.1
6.4
48.1
X= 0.1
8.3
42.1
1000
20
Table 4: c-Diy trained models achieving better tradeoffs than NLL on MathQA-Python.
(a) Achieving higher Precision than NLL highest Recall
(b) Achieving higher Recall than NLL highest Precision
Model
Olmo-1B IB
Alama-Alpace
Model
Llama- Llama-Alpaca Alpaca
Method
P
R
P
R
Method
P
R
NLL
1-6
0.20
0.47
t-1
0.067
0.49
NLL
t 1.6
0.10
0.10
c-Div
1,4,-1
0.21
0.50
o- 1.4, 0.8
0.087
0.49
c-Div
1,4,-11
0.10
0.14

---

When Temperature Fails, Change the Loss
Acknowledgements
Friedman, D. and Dieng, A. B. The Vendi Score:
A Diversity Evaluation Metric for Machine Learning, July 2023. URL http:/ /arxiv...... org/ wttg/bsg/////ooo/oeet...ot..eere /210.. 02410. arXiv:2210.02410 [cond-mat, stat].
This work has been partly funded through project ACDC ANR-21-CE23-0007. This work was granted ac- cess to the HPC resources of IDRIS under the alloca- 10145333,,,,, 2025-
tions 2025-A0181016155 A0171014638, 2025-AD01 101402200 made by GENCI.
Gehring, J., Zheng, K., Copet, J., Mella, V., Cohen, T., and Synnaeve, G. Rlef: Grounding code Ilms in execution feedback with reinforcement learning. arXiv preprint arXiv:2410.02089 2024.
Impact Statement
This paper presents work whose goal is to advance funda- mental algorithmic development. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning Language Models with Preferences through f-divergence Minimization, June 2023. URL http: /twpiv.iivvv.///ii///e/////aavv/ ortg/bs//32...2../// 08215. arXiv:2302.06225 [cs, stat].
References
Grattafiori, A., Dubey, A., Jauhri, A., and Pandey, A. The llama 3 herd of models, 2024. URL https: //arxiv. org/ org/abs/ 2200.447.22 2447722183.
Bengio, Y., Ducharme, R., and Vincent, P. A neural probabilistic language model. In Leen, T., Dietterich, T., and Tresp, V. (eds.), Advances in Neural Infor- mation Processing Systems, volume 13. MIT Press, 2000. URL
abs
21783.
Groeneveld, D., Beltagy, I,, Walsh, E., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Flazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naík, A., Nam, C., Peters, M., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell. E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Sol- daini, I.,, Smith, N., and Hajishirzi, H. OL Acceler- ating the science of language models. In Ku, L.-.., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Com- putational Linguisties. doi: 10.18653/w1/202..usm 841. URL htt.httss///ccaattttett ogy wwc.lanntholooy.orr/00000 acl loog..ng....... .ooo..81/.........
cc/papper_flleespppprr///ooooo 728 22842066222001555525544000088880 pdf.
Brock, A., Donahue, J., and Simonyan, K. Large Scale GAN Training for High Fidelity Natural Image Synthe- sis. February 2019. URL http:h/tt//cc /tt:://axxivv org/o///loo.oo. abs/ 1809. 1096. arXiv: 11096 [cs, stat].
Chang, C.-C., Reitter, D., Aksitov, R., and Sung, Y.-H. KI- divergence guided temperature sampling, 2023. URL httgg////////// https://aaat org/ abs 2306 01286.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained code. arXiv preprint arXiv: 107.03374 2021.
DeepSeek-AI. Deepseek Ilm: Scaling open-source lan- guage models with longtermism, 2024. URL https: //arxiv.org./eeee............ hrtp:/oog///sse.. ab////a1s/2441 2401 02954.
Grover, A., Dhar, M., and Ermon, S. Flow-GAN: Com- bining Maximum Likelihood and Adversarial Learning in Generative Models, January 2018. URL http: arxiv. www.vv.og//eee. b 170111100000 08868. 1nXiv:1705.082 [cs, stat].
DeVries, T., Drozdzal, M., and Taylor, G. W. Instance Selec- tion for GANs, October 2020. URL http:://aaiii hoggabs/2007 15255. arXiv:2007.15255 [cs, stat].
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020a. URL htRhst/::: /ttp://///p//eeeeeee///s//ee/eee net.eewwwwwwweeeeeeeeeee ntt.:/wumete///r forum? id=rygGdy=Fvh.
Fan, A., Lewis. M., and Dauphin. Hierarchical neu- ral story generation. In Gurevych, and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 889-898, Melbourne, Aus- tralia, July 2018. Association for Computational Lin- guistics. doi: 10.18653//1/118-1000 1082. URL ht https: //aclanttie aocllanhhoooy...o www.hhhooooo.or//---000220
Holtzman, A., Buys, J.. Du, L., Forbes, M., and Choi, Y. The Curious Case of Neural Text Degeneration, Febru- ary 2020b. URL http: /arxivi..v. org/ abs/ abs/1904. 09751.
[es].
10

---

When Temperature Fails, Change the Loss
Ji, H., Ke, P., Hu, Z.,, Zhang, R., and Huang, M. Tailor- ing Language Generation Models under Total Variation Distance. February 2023. URL http: www.iaavvvvvooo/.../eoaa org/ abs/2302, 13344. urXiv:230.11334 [cs].
1 1126/science.ab URL hhtp::://axivvorgp Abs/2203.07814 arXiv:2203.07814 [es].
Minka, T. Divergence measures and message passing. pp. 17, 2005.
Kang, D. and Hashimoto, T. B. Improved Natural Lan- guage Generation via Loss Truncation. In Jurafsky, D., Chai, J.. Schluter, N., and Tetreault, J. (eds.), Pro- ceedings of the 58th Anmual Meeting of the Associa- tion for Computational Linguistics, pp. 718-731, On- line, July 2020. Association for Computational Linguis- URL https: 66.
Murphy, K. P. Machine learning a probabilistic per- spective. MIT Press, 2013. ISBN 978-0-262-01800-99 0-262-01802-0.
Nowozin, S., Cseke, B., and Tomioka, R. f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, June 2016. URL http: /atpvvarvvvvv/////////////////v. org/ abs 606 00709. arXiv:1606.00709 [cs, stat].
tics. doi: 10. 10.18665311/20222a---maam //aclantholog": ptg//020
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Ham- bro, E.,, Grefenstette, E., and Raileanu, R. Understanding the Effects of RLHF on LLM Generalisation and Diver- sity. In ICLR, 2024.
Pang, R. Y. and He, H. TEXT GENERATION BY LEARN- ING FROM DEMONSTRA- 2021.
Papadopoulos, V., Wenger, J., and Hongler, C. Arrows of time for large language models. arXiv:2401.17505, 2024.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved Precision and Recall Metric for Assess- ing Generative Models. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). Vancou- arXiv: 1904.06991.
Peeperkorn, M., Kouwenhoven, T., Brown, D., and Jor- danous, A. Is temperature the creativity parameter of large language models?, 2024. URL https: //ar:iv/./.... org/ abs / 2205 00492.
ver, Canada., October 2019.
Labeau, M. and Cohen, S. B. Experimenting with Power Divergences for Language Modeling. In Inui, K., Jiang, J.. Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4104-4114 14, Hong Kong, China, November 2019. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ D19-1421. URL https: http///aaaaaatcoooeee/ee org/ 119-/421/.
Pillutla, K., Swayamdipta, S., Zellers, R., Thick- stun, J., Welleck, S., Choi, Y., and Harchaoui, Z. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. In Advances in Neural Information Processing Systems, volume 34, pp. 4816 4828. Curran Associates, Inc., 2021. URL https: sttps://proceedings.. neurips. wttpcc/ppppp/////paeeaa
0 c3 0dadc0 -asstttta....
html.
Le Bronnec, F., Verine, A., Negrevergne, B.. Chevaleyre, Y., and Allauzen, A. Exploring precision and recall to assess the quality and diversity of LLMs. In Ku, L.-W., Mar- tins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp. 11418-11 1141111444,, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10.18653/ 616. URL https: https: http:::://ccaantooooy.ooooo acl-long 616/.
Sajjadi, M. S. Bachem, O., Lucic, M., Bousquet, O., and Gelly, S. Assessing Generative Models via Precision and Recall. In 32nd Conference on Neural Information Pro- cessing Systems (NewrIPS 2018), Montréal, Canada, Oc- tober 2018 URL http: //aaxivv...// wwt//rrg//////ooo/o//...ot..ssre abs/ 1806. 00035. arXiv: 1806.00035.
Shao, Y., Gouws, S., Britz, D.,. Goldie, A., Strope, B., and Kurzweil, R. Generating high-quality and informative conversation responses with sequence-to-sequence mod- els. In Palmer, M., Hwa, R., and Riedel, S. (eds.), Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2210-221111 Copen- hagen, Denmark, September 2017. Association for Com- putational Linguisties. doi: 10.18653/1//D1--II/ URL https: http://aaaaattooooooooo www.:/ogyyorggggg//
Li, Y., Choi, D., Chung, J., Kushman, N., Schritwieser,, J. Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., d' Autume, C. d. M., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D. J., Rob- son, E. S., Kohli, P., Freitas, N. d., Kavukcuoglu, K., and Vinyals, O. Competition-Level Code Generation with AlphaCode. Science, 378(6624):1092-111 De- cember 2022. ISSN 0036-8075, 1095-9203. doi: 10.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Con- sistency Models, March 2023. URL http::ttttattta 010 arXiv:2303.0166) [es, stat].

---

When Temperature Fails, Change the Loss
Sun, H. and Schaar, M. v. d. Inverse-RLignment Inverse Reinforcement Learning from Demonstrations for LLM Alignment, May 2024, URL http::/arxiv..r////// Nos/2405.15622 arXiv:2405.15624.
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models, SIGIR, 2018.
Taori, R., Guirajani. I., Zhang. T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-follovingg llama model. https: setps://githhe /ttp:///gtttttttt commtatsu-lab// stanford 2023.
Theis, L.,, Oord, A. d., and Bethge, M. note the evaluation of generative models. arXiv, April 2016. URL http: http:://wwww..../aa//o/o//s/ 01 01844. 1rX1v:1511.01844 [cs, stat].
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez. A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https::p//// htttpp://arxiv.oo/ abs/ 20t5/2302.110 13971.
Verine, A. Quality and Diversity in Generative Models through the lens of f-dieeggences. PhD thesis, January 2024. URL
Verine, A., Negrevergne, B., Pydi, M. S., and Cheva- leyre, Y. Precision-Recal Divergence Optimization for Generative Modeling with GANs and Nor- malizing Flows. Advances in Neural Information Processing Systems, 36:32539-32573, December 2023. URL https://proeddiigs..e... httppaper_iilllesppppr////..2.2. 1111FF:cocablaa33c/77/4888388000 -t html.
Verine, A.. Pydi, M. S., Negrevergne, B., and Cheva- leyre, Y. Optimal Budgeted Rejection Sampling for Generative Models. Proceedings of The 27th Interna- tional Conference on Artificial Intelligence and Statis- tics, March 2024. URL http://////// arxiv.orgg org/ abs/ 2311. 00400046600 60. arXiv:2311 1..0040 [es].
2311.00460
Wang, C., Jiang, Y., Yang, C., Liu, H., and Chen, Y. Be- yond Reverse KL: Generalizing Direct Preference Opti- mization with Diverse Divergence Constraints, Septem- ber 2023. URL https: arwiv. www.wiiwooo///////////// . 15624, 4. 112Vv
Zheng, K., Decugis, J., Gehring, J., Cohen, T., Negrevergne, B., and Synnaeve, G. What Makes Large Language Models Reason in (Multi-Tum Code Generation?, Oc- tober 2024. URL http:////. axxiv. wttpiv...ggab/////////// 08105. 15X.v:2410008105 [cs].
12

---

When Temperature Fails, Change the Loss
A. Proof of results in Section 4
A.I. Proof of Theorem 4.2
For improved clarity and readability, we introduce the concept of acceptable tokens.
Definition A.1 (Set acceptable tokens), Given context Vel E V'-1 the set of acceptable tokens is S(x2)) Supp(Pa)). i... the of tokens with non-zero probabilities in Pet.
Let us first recall the theorem:
Theorem. Let 20 e P(X)), then for any temperature we have:
ax(P1Q)) s [Supp(P) ZL/t
(20)
VL
1
[Supp P)]
and Bx(P)(Q)) 5 X
Z1/1
(21)
VL
where m maxie(l,....) maxi Fo(xx)); - Fo((xa)) the highest difference between the logits of the model and [Supp(P)) Ez,eS(e) ExzeS(ar) Exges(mer) 1.
Proof. Recall that Bx ax/A therefore we will study the behavior of an with I and X:
(x(1)% Σ
min (A)()),
(22)
xEXt
I 2x(2 wes))
Σ z
min
AP(x),
(23)
[=1
Since P(x)= = 0f x Supp(Pa) by definition then we can restrict the sum to x Supp Po):
(x(P))) - Σ min (AP(x). A (h a.))
(24)
zeSupp(Pb)
1-1
we denote At, = max; Fo(xt.);; - (x we can write the conditional probability as:
exp(Fa (xa)tt
Q2(2, xaa)
(25)
ΣV exp( F' Fo( (xxx wa)/t) exp(- exp(-A1,eana/1))
(26)
Σk- (xxp(-A,mma.//))))
Thus, we can upper- upper-bound bound the conditional probability by the probability for the most probable token:
(x xxa)= S ΣV sap(-A,,aat)/E
(27)
We denote Z = ma max.e(1.....) maxi,jey (o( - (x(a)) the highest difference between the logits of the upper-bound bound the conditional probability by:
model on the target distribution. We can then upper-
1
exp(Z/1)
Qb(zy (aa)) S Σl, exp(-Z/t)
(28)
V Vexp(-Z/t)
13

---

When Temperature Fails, Change the Loss
Therefore, upper-bound the Precision by:
L exp(2
(x(P))())
Σ
min AP(x).
I
(29)
ssSupp(P)
11
exp(2
Σ #eSupp(P)
min AP(x))
(30)
exp (xp(ZL/t)
Σ #cSupp(P)
(31)
VL
exp(ZT.11)
(P)
(32)
Vi.
which concludes the proof for the Precision.
A.2. Artificial case: Proof of Proposition A.2 and Proposition A.4
We recall the artificial case presented in Section where we consider the distributions P and Qo defined as follows.
We choose target distribution P= I Per where all factors Pes sparse uniform distributions over small subset of K tokens, and model distribution Qo that matches P everywhere except at the two specific positions in the sequence I and l2, i.e, Væ e U4, Vi (1, Q- xaa) P(. w<e)) For position 1 and l2, the conditional distrbutions are defined as follows:
if pK: pf sos K, otherwise,
pk:
Qo(x (eee,))
b
pk:
0,
if Pa, O,
Φo (x) (xat))
K
otherwise,
V-K
where e [o, 1] controls the number of tokens that will have extra-mass assigned under To, 0 [pK] controls the excess mass assigned to the these tokens, and b 2 0 is chosen to ensure normalization. E [0, 1/2] determines the level of noise introduced outside the support of P.tz-
We first present the proposition that fully characterizes the PR-Curve of Qo. along with its evolution under temperature scaling. We drop the notation for simplicity, and we denote P= P and Qo = Qo-
Proposition A.2 (PR-Curve under Temperature Scaling), Lei P0Qu E P(x;) respectively defined in the artifical case in Section 4. Then, for any temperature t E R, there exists trade-off Amin and Amaxo with μ = p (11110 p):
(1 (1-
(1-
A min
(33)
I p1 1/ 11-1/' (1- (1- (V/K (VIK-1)--\/edl)
and
1/t
4/2
(1-c)1/e
X
(34)
1- (1- + 1 (1 (1-6) + (V/K 1)1-1/1,2/0
such that the Precision a (P)l))) and Recall 3x can be written as:
For all X 2 Amaxi
(1-e)²/t
(x(l()))
(35)
11/t + +V/K - -11-K-)
(1 (1-6)
(x(1)(())
(36)
À (1 11 -1)-//11-1/////
14

---

When Temperature Fails, Change the Loss
01
0.5
0.5
0.0
0.3
1.0
84
0.0
0.5
1.0
84
0.0
0.5
10
8x
(a) 1 0.1
(b) [ 1
(c) ( 10
Figure 9: PR-Curve for different temperatures with 100, K 50, a/p = 1.45, = 0.15 and p = 0.5. The PR-Curve at t 1 represented in dashed line.
03
0.5
0.0
0.
1.0
84
0.0
0.5
1.0
81
10
0.5
1.0
8x
(a) L 0.1
(b) [ = 1
(c) L = 10
Figure PR-Curve for different temperatures with V 100, K 10, a/p 1.05, e 0.05 and p = 0.5.
For all Amax X 2 Amin:
(1- -)
(1 e1
ax(PIQ)) = pd + (1 1/t 11-1/1a (1 e) (V/K - 11)1-1/t,1/8
(37)
1
(1 a1
(1 (1-c)
Ba(PIQ0) p+ À (1 - 1/ + 1 1/1//////1 (1 + (V/K -1)1-1/1211
(38)
For all X S Amin.
(x(()) X. Bx (p(()) 1.
(39)
(40)
We can visualize the PR-Cus PR-Curve for different temperatures in Figures and 10 with different parameters V, K, a, and p.
Proof. We begin by computing the PR-Curves for the original distribution To at temperature = 1. We then apply temperature scaling to obtain the tempered distributions and compute the corresponding PR curve.
15

---

When Temperature Fails, Change the Loss
PR-Curve for == 1. Let's first compute Precision and Recall for extreme values of the trade-off parameter X, i.e. A +00
and À 0.
A +000
O. (PIQQs) Qo(Supp(P))
(41)
Σ zeSupp(P) P)
Qo(x)
(42)
Σ
II Qo xa)
(43)
( ((ppp(()
Σ
I Qo(as xx1)x x
E
Qo(an, [eee))
(44)
mnt5( domll 1eS(ma
i=1
oh (5(c)))))
1-
x
Σ
II Q6(xi x<)) x
Σ
Q0(213 xel1z)
won 13 1) [+++1
212 (5(()))))))))))))))))))
Σ
π Q0(21 x<s))
x
4-11
Σ
I
P(z1 xa)x x
Σ
Q0(xn, Ize1)
(45)
[=1
mi (5(c)))))))
b-1
x
Σ
P(x) xa) x
Σ
Q0(21s Deta))
[1++1
212 (108))))22)))
x
Σ Bi 11 (m (m5 1=1211
I
P(ri xeS)
Σ
Il
P(i xc)) x
C
where er is either a or b
(46)
1= pk
01
[=]
Σ 1-e
x
Σ
II 1h++1
P(m
xa)x
K
21, (S
Σ
FI P(z, Ia))
1ig+1
K Ci (1-c)
Σ
(47)
I= pk
=1- 1
(48)
Equation (45) is obtained under the assumption that Qo perfectly matches all conditional distributions P except P(. maa,) and P(. (ett)) Equation (46) is derived by substituting the expression Qa. Equation (47) is obtained by iteratively marginalizing over the values of . 11 Ft, +l, -12-1, and Tlatl.
A 0:
Bo(P)Qa) P(Supp(Qe)) =
(49)
since Sup)(Qo) c Supp( P).
A =]0,+ +oo[:
In the following, since aA = XB3, we focus our analysis on oxx-
16

---

When Temperature Fails, Change the Loss
ax(P)())= Σ
min (AP(x)), Qo(z))
(50)
xext
1/ P Dai. I1 QIQa Ra))
1
Σ
min
(51)
#1 P1ont
1-1
i-1
Σ
P( ma) min (AP(21, Fet ) P(1 xe12)) Qo Fet, (Q0 (ett))
(52)
01 11 E=1
1el2
b-11 P(x aaa)
Σ
(53)
(mr, (k) (
min
C pk
(21 skY
(M)o)) )}
1-
e + 1l (x)
K
where is either a or b depending on the value of 21, When 21, S(xet,), the minimum is reached for X 1l (mgeS(m..)) = 0. Therefore, the only terms that contribute the sum are those for which Ti2 {S(xaa), i.e. the acceptable tokens position 12.
(P((0))
Σ
h-
II P(o1 æa) min
A sx)
1l
a
1 sk) (1 s))
(54)
K2
pk²
[=1
Σ 21
Il
P(z xaa Σ min
K
d
a
(1 (1-s))
(55)
K'
pk
1=1
And finally by marginalizing over 21,
T4 we have:
ax(PlQ0) Σ min (A, q(1 - -))p))
(56)
Three regimes can be distinguished:
1. For A a(1 - c)/p, we have:
1 K ax - [<((--)=(1-o))
K
R
1-
(57)
pk
In this regime, that is, for large values of X, ax reflects the model's quality, as it converges to Precision when À + +00.
2. For - b(1-c)/p s A a(1 c)/,e)/p, we have:
ax 1 pk
K
Σ
Σ
c)
(58)
K
1=1
l=pK
1-p
pd b( 1 - c)
(59)
P
We will denote μ p/11-1111) p) in the following, and since pk
+ (1 - p) b
= 1, we have:
pk
pk
1 p
b
(60)
Thus, we have:
ax(1QQ) pd (1-c)(1-a)
(61)
17

---

When Temperature Fails, Change the Loss
3. For x < 6(1 p)/p, we have:
1 K Σ A A,
(P(Q0)
(62)
K i-1
and therefore:
(P 1.
(63)
In that regime, Recall is maximal since the model generates all tokens.
PR-Curve for Tempered distributions.
Tempered distribution Q:: To simplify the notation, we define the inverse temperature r = 1/t and set μ = p/(1 p).
We define the tempered distribution Q as follows:
Q | (Dete) )= =
pk
where a is a for s pk and b otherwise,
(64)
Σ==1
K
Cj Kp
2+=K+1 (0)"
c
pk
(65)
DeK
jj=1
pk|
+
Σj-pK
ΣK
ppk
br
1
of
(66)
K
pa
+(1 -D)
1
67
(67)
(1 p) pai + .b¹
Moreover, since pk x pk + (1 p) x p
1, we have:
b p P
a).
(68)
1-p p
1- p
Thus:
1
(2( (eet)) =
(69)
(1 p) K ua² + 1r7 11 (1-a))
1
(70)
(1- (1-p)K u 11(11 a)T + pa
cf
(71)
(1- (1 (1- a) + +1-t
Therefore, we have:
if $ pK,
Q2(21, 1 [xes))== =
(I-p)Kp² (I-a) (1-0) (I-p)K (1-0)--p1-r)
(72)
otherwise.
For 2i e S {((rrtt))
(
26(111 [<ez))
(73)
K (L))
+ +V K) V-R
(1) (1-6)
1
(74)
K (+)) (1- (V/K) - 1) (<
(1-6)²
(75)
K (1-6)=++ + (V/K- (6)
18

---

When Temperature Fails, Change the Loss
PR-Curve for P and Q% With these expressions, we can compute the PR-Curve for different values of r. By analogy with the previous computations at f 1, we marginalize over all tokens for which Pt ar2)) oo( ma)) yielding:
(x(())
Σ
min (AP(zi), (b(2))
(76)
x
K min A
1
G
(1-c)² (V/K -1)1-TE"
Σ 1-1
(77)
K
(1- (2= (1 a) + 1 a +V//)K
1
6j
(1-c)²
min
(1- p) (1 - a1= + a A;
(78)
K
(1-e)* + -1)--E-
Let define:
1 (1-a)
(1-c)²
min
(79)
1- (1 - a)" + a= a² (1-6)² + (V/K 1)1-Te
and
1
a_
(1-c)
(80)
1- 11p(1-1 a1 + a (1-c)* + (V/K - 1)
We can show that there exists three regimes for the Precision and Recall dependent on the value of À compared to X and Maxi
1. For À 2 X we have:
min (>. >
1
&
(1-6)²
(1 - p) (1 a) + + az (1-c)*+ (V-c))(VVKKK))))) (V/K-1)1-7-
of
(1-t)-
(1 (1- p)pp a1- + + + a (1-c)² + (V/K-1)l-r
Therefore:
(P((()) 1 k
of
(1-6)T
K E
(81)
pa (1-p)b (1-6)* + (V/K-V - -1)2-5e
1
(1-c)-
K
of
(82)
K (1 - + (VIK 1)
pa + - (1-p)br
(1-6)²
(83)
(1-c)=+ -VV/K-1)1-
In particular:
(1-c)r
a6 (PI()) (1- + +V/K - 1) T-
We observe that both:
1
a_
a_ pa + (1 - p)b
p (1 a)T 4 a
ar
and
(1-6)7
(V-
are strictly decreasing functions of t = 1/T, since a > b and 1 1 c. Therefore, in this regime, (p(()) is strictly decreasing with t. Moreover, the range values defining this regime is also strictly increasing, as Xe X decreases with f.
19

---

When Temperature Fails, Change the Loss
We also observe that:
lim ax (PIQ)) K (-++00
lim ai =1 and C-U
(84)
V
Therefore, the range of À for which the Precision is constant is increases as the temperature increasing and the constant value the o decreasing. Thus, the Precision is strictly decreasing from 1 to K/V as the temperature increases.
For E [Mmmn maxl, we have:
pk Σ
(x(1)))
(85)
1-1 K
(1- p) (1-a)²
1
(1-c)T
Σ
(1-e)*+
+ (V/K - 1)
(1- p)K (1-a)²
(1-()²
=pp+
(86)
(1 p)K (1-0)² + -1l-Ta (1 (1-6)T+
(V/K
(1-
(1-c)T
-p
(87)
(1 a)* + al a (1 c) + (V/K-1)1-
We note that:
ax (P(())) = pd + p))
(88)
In that regime, to know the behavior of the PR-Curve, we need to know if miin (1-a)* (1-e)'
(1 -p (V- 1))/K- is increasing decreasing with f. However, can observe that:
lima(P)(Q)) = pd and lim a(P))))) = pd + (1- (1 1++00
(89)
t-0
3. For À Xmin have:
K
(x(P)(())= =
A= = À and thus Bx (PIQ)) = 11
(90)
[=] K
This concludes the proof.
Analysis of the PR-Curve.
To analyze the behavior of the PR-curve, we study the dependence of the expression
(1 a1- (1 + al (1-6)*+ (V/K- 1)2-rer
(1-
(91)
on the temperature parameter t. We first introduce the function:
(1-7)²
(n = u2-rqz + (1 ())
(92)
which allows us to rewrite the above expression as fa (T) -c./K--(T))
Therefore, focus analyzing the behavior of the function g(T) := fa,p fe.//K-(T) (T) varies.
Lemma A.3 (Rate for change of g). Let K, L e No with K L, c c]0, 1[ E00 1[ p/(1 ([(1)) p). be [0. 1] and a 1 - -b/u Then, we can define:
V/K-1
(93)
V/K 1 (2) (-RiI))
such that:
20

---

When Temperature Fails, Change the Loss
For all eo. 9 is stricily decreasing with T.
For all e > 9 is first decreasing then increasing with T.
Proof. First, show that:
(T) = log(1 - 7) 7) [ + (1-7)-
(94)
(ut-Igr + (1 - 7))
(1-7)" [log(>) t ""
lolg log(v)u -log(11 - (1((1-))(1-))
(ul-ey- (1 (1-
ul-rgT(1 - (1- (ul-rar + (1 - 7)
")olog log 7/6
(95)
(96)
By noting that:
7"
f1-92/((T)
(97)
7 (1/1)) 1-7 (1-
7,
(98)
(1/1)= t- (1 71
(99)
-2-TzT+ + - 7)
we can write the derivative of g as:
(T) log (1-9)
(100)
7/1
Then:
g'(7) Sa, (Teevvv))//)) (a (T)
(101)
1-a
log
(102)
a/u
+ (T <((V/K fe.TKK--))_.I..f( J..IK--1))ii-avvv---
1-E
fa (1f (1-a.//((T) (1 (100(111))))
(1-a)
(103)
a/P
+ +1-.u///K-n((>) log c/(V/K-1) -
1-E
By observing that 1/f1-y1//(TT) (p'-Tn" + +11-) 1+
we can show that g'(r) can be rewritten
d'(7) (1oo(())--o.jp((((((()))))) (7))1--11///K--(
(104)
1 1+ 1-1 (11-o) a) ) log
1-E
[((V/K-1)
- -VIK - 11 1-c [1-)] log
(1(1 (u(1-a) (-1--a)
h(1/7).
(105)
21

---

When Temperature Fails, Change the Loss
As
need to study the sign of h(1/T) h(C)) with 4. We can show
that:
1/c
a
1-6
h(t)=
1
log c/(V/K-1)
(106)
a/P
1
1-c
1 +
log M(
V c/(V/K- -1)
where ((1 p/1-a)/a b/a < and (1- (1-c)/eVV/KKD 1/(V/K 1) > 1. We will show that h is strictly increasing. If
denote (1))(1) ((1 7))((///))/ we have:
1-7 7/1/ (1)
(1) f2
log
(107)
Therefore, we can compute the derivative of l::
1
M - a)
1--
(1)
log
(1) log
(108)
82
c/(V/ -1)
1-c c/(V/K- -1)
log
TeV/K-1(t) log 11(1 aa
(V/K - 1)
1 log
1-6
log
a
Te./K-1(1)
V-1 + taa (1)
(109)
12
e/(V/K-1)
1M aa)-a))
>0
>0
Therefore, h is strictly increasing. Considering the limits of h when / goes to 0 and +o0, we can show that:
1/t
lim
((1-a)
0,
lim htt)= 1-0
since
(110)
1/r
lim
<[(V/-II)
1-c
+00
and
1 11 log
1-6
V
1(
lim h(t)
log
(111)
</(V/K-
11)
V
K
lim
((1-a)
1/t
since
1/t
lim
<(V-1)
To study the variation of g()), we need to study the sign of lim htt).
We can show that:
1 1
1 1
1-c
V
b
lim h(t) t0
0
log (og(V/K- 1)
log
log
2
(112)
1 + log (og(//-1))
V
log
a
log(V V/KK -K-1)
(113)
V K
(TIRRV-R
< 1/c 166-2-
(114)
v/K-1
V/K-1
3 05
(115)
V/K-1+ (g) KJ
by noting that //(1+ /4) Therefore, by defining
V/K-1
(116)
V/K 1+ (g) 1.
We can identify two cases for the rate of variation g:
22

---

When Temperature Fails, Change the Loss
b = 0.01
b = 0.05
= 0.5
10
109
108
1.0
0.8
0.6
8 10-1
10
10
0.4
10-2
10"2
10
0.0
10
10a
10a
10
10a
10a
10
10a
10a
V/K
V/K
V/K
Figure 11: Values of cn for different values of V / b, and p. We consider different levels of token underrepressentation by varying b and p. Low values indicate that the model assigns very low probability to the relevant tokens, while low values of p mean that a large proportion of tokens are underrepresented. We observe that even for small values of V/K, when b is low and p high, the value of 6n remains relatively large, often above 0.1. We recall that when co, Recall starts to decrease some point temperature increases.
First when e 2 Eu. then h(t) s 0 for all and therefore is increasing with t.
On the other side, if eo, then there exists such that h((o) and thus such that h(t) 0 for / to and h(() for t < to. Therefore, is first increasing then decreasing.
can now analyze behavior of the PR-Curve based on Lemma A.3: Proposition A.4 (Variation of PR- PR-Curves with Temperature). Let P, Qo E P(x be the distributions defined in the artificial setting of Section Then, as the temperature increases:
The threshold Nmax decreases strictly with t, satisfying:
K
lim X Mmx +o0 t00
and lim X t+000
(117)
For the lower threshold Xmin
- cIfc> co. then Nm increases strictly with t.
-If S co. then Noin increases for small then decreases after a certain point.
In both cases:
K
lim X 0 and
lim X
Amin
V
(118)
The behavior of the PR-Curve with respect to temperature depends on the regime of X in relation to Xuin and Nunsei
High-X regime (A 2 Xims)): The interval (x2 X widens grows. In this case, both Precision ax (PI)Qt) and Recall 3x 3x(P1Q) Q1) decrease strictly with t, converging respectively to K/V and K/(V)) X). More precisely, for any fixed X, there exists temperature to such that for all to. Precision and Recall decrease strictly with temperature. The larger the value of X, the smaller the corresponding to-
Low-> regime (A N N:
-Ifc this regime expands with increasing t.
-If 6 $ co. the regime first expands, then contracts.
23

---

When Temperature Fails, Change the Loss
In this regime, both Precision and Recall constant with respect lemperature, taking values og X and Bx 1. Intermediate-X regime (A E [Xmino Mmaa)): As L - co, this range collapses to = K/V. In this regime:
-Ife > eo, both Precision and Recall increase strictly with temperature.
If 5 tg. both Precision and Recall increase initially with I, then decrease. In all cases, they converge to B/V, 1.
Moreover, if V (1=a)(1-1)) cc), then there exists temperature such that for all t to:
3x(P)1Q) < (x(()))
(119)
Proof. The different regimes follow directly from Lemma A.3 and the characterizations in Theorem A.2:
The behavior in the high-A regime is derived from the asymptotic expressions in Eq. 84.
For the intermediate regime, the PR-Curve values follow Eq. 88, and Lemma A.3 determines the qualitative behavior based on En-
For the regime of low X, only the Recall and Precision are fixed. Only the range of À defined by X < X varies and follows the same variations observed Lemma A.3.
24

---

When Temperature Fails, Change the Loss
B. Proof of results in Section 5
B.I. Proofs for Subsection 5.2: Trunc
Trunc loss. Define pTrone (-) (- &Trune), ).une).ae and let H.) denote the entropy. Informally, the following proposition states
that if frunc gets arbitrarily close to its minimum value (1 Δ) (Trunc) then for a recall arbitrarily close A, the precision will be arbitrarily close to 1.
Proposition. (Trunc optimizes the precision given recall) For any if L
Trume (0) sct (1 (PT())) then
there is precision-recal pair (a, (3) such that 3 2 (1 - A) vt, and a 2 1- V 2(1-A)
Proof. Let Xrane
and A 1 P (Nmmne). Let us first rewrite the objective function:
Trunc (0) = Exg- Σ 1l Q())2))) log Q<((zi)
=- EE P (Q((0))=)) log Q(x)]
(Nrnn E mm)) [log Q())]
=- A) prom () [log (x))
= (1 - A)LKL (P Trune 1Q) + (I Trunc)
Combining the inequality Ci (0) SEt (1 H- with Pinsker inequality, we have
DKL. (PTrnnc 11Q)
DrV (ptrumc .Q)
2(1 A)
Choosing A= 1_ we are now ready to compute lower bounds on Precision-Recall
as (PIQ))
Σ
min
I-A P(x).
Σ
min
1- P(x).
xe ion:
Σ min ppTune (x). (())((x))))))))) Dr Druunun Q)
2 ((1-A)
E(1-A) A)
8* (P = T (PIIQ) (1-1))((111))
2
TruncR loss. We proove Proposition 5.3 using characterization of the optimal distribution under the Li-A loss.
We first characterize the optimal distribution minimizing the Ct loss. This result will serve as a foundational lemma for proving the main proposition relating TruncR optimization to Recall at fixed Precision.
Lemma B.1 (Optimal TruncR distribution). Let P (P,...,,Py) be discrete reference distribution. Consider the loss
C (Q) Σ P. 1Q log Q..
where > is chosen so that Eiques cs P; 1-11-14 and Q = (Q .Q..... Qn.
...)..)).)) ranges over the simplex. Then the unique minimizer
Q = min(5, pP:):
25

---

When Temperature Fails, Change the Loss
where y O and ó satisfy
E -1,
P.-1-A PrPis
i-1
Proof. Introduce Lagrange multiplier À for the constraint Σ,Qi=1 The Lagrangian is
I(Q.)) = - Σ - (Q,10) lo XE
Stationarity with respect to Q; gives
P. Q
=0 = Qi f X <0 Q; 6.
(q + x
Hence Q min(6, Pt/A). Setting 7 = 1/A yields Q PP)) Finally, choose and á so that Σ, Q, and
EiyP,e8 P-=1-A. Uniqueness follows from the strict concavity of Σ; P.logg Q..
Remark B.2 (Lower bound on 7)) Let
{= <0) S = [1,....,, S.
The quantile constraint gives Eies 1-
Normalization gives
1 ΣT.,
At the boundary S, ô P ()) Thus
1= - A) P m)) [59] = 7= (1-A) (s) SSc
2 1.
since (1 - A) + (s) s ( (1-A) Eiss 1,
We now state and prove the main proposition linking the minimization of the L1-5 fixed Precision constraint.
Truncht loss to Recall maximization under a
Proposition B.3 (TruncR minimization favors Recall at fixed Precision). Let P = (PI....,Pay Pa) be reference distribution and Q' min be the optimal distribution minimizing the TruncR loss Lt from Lemma B.l. Then, for all A 0, any À O such that oxx (xx(P)QQ)) = A satisfies
2(PIQ))21-1
In particular, minimizing the TruncR loss with constraint Eiq.e8 Pi = A leads to distribution achieving Recall at least 1 at Precision level A.
Proof. We compute the Precision for Q" For any A> 0, define min(A,7). Then
ax(P(Q)) = Σ min(AP, = Σ min(AP, miin(AP,,in(77P,00) = Σ min((P,,)))
(120)
(121)
First, observe that if A> 7, then 7= and a = Σ, Qi 1, by the normalization of Q". So for any A > 0, a solution with ay A must necessarily satisfy À 7.
26

---

When Temperature Fails, Change the Loss
Since 7.2 by Remark B.2, in the regime where X 1 s 7, and thus v A. In particular, when À
31 Σ (i,,).
(122)
Σ
min Pt, Σ min(P.,6)
(123)
18
P.P min(P::8)
(124)
is
1 A
Σ
min( P(,
(125)
1 -1-28 A
(126)
where in Eq. (124) we used the fact that for i & S = (i: 1P:7P, < <6,, we have P. < 6, so min(P:,0) P..
Now, let À be such that ax Then from the above, S 1, and thus
1-A Bx(PQQ) a À 21-A
This shows that the Recall corresponding to Precision level necessarily at least 1-A, which concludes the proof.
B.2. Proofs for Subsection 5.3: GOLD
Let us recall the Theorem 5.4: Theorem (a-Divergence Minimization). Minimizing the a-divergences between Pat and Qei for all I e {1....,.) L} is equivalent to minimizing
CZ (0) -Eg-p
Σ Qu(z)(taa)) log Qe
(127)
Proof. Let us focus on the a-divergence D,, any distributions Pet and Qei and ot Pat
D. Qa(
Pa(x))
(128)
LeeV
[Qu((z))-a
(129)
by assuming that Pat Pat with 7 = 1. We can write the gradient of the a-divergence with respect to the model parameters
0:
VeDa(P
Vo ==
1-a
(130)
(0(x)) (Coai))-ooo00 voQo(xi) by the chain rule,
(131)
(o(ar)
Qo(m) voQa(>;)
(132)
-Q((z)) Qo since V, log =
(133)
Qo(x:) Vo log (m)).
(134)
Therefore, minimizing the a-divergence is equivalent to maximizing the loss:
Q(x1) 1-a |- log Q(x:))
(135)
27

---

When Temperature Fails, Change the Loss
B.3. Proofs for Subsection 5.4: Tail.
First let recall proposition 5.5:
Proposition. The optimal distribution Qo using the TaiLr method with y > satisfies the following property:
Væ VL, Vl L.e,,.... I,..,,,,, L, Q0(x)(aa)
(136)
1-7
In others words, Q(](aa)) > ((n((aa) if P(æi(ara) 1/V and Q(ai)aa) s P(n)(ae)) otherwise.
Proof. The loss minimized by the TaiLr method is:
Σ Qu((a)) log Q0(21) 1-1 7+ (1 Qu()))
TaiLi (0) = -Ex-P
(137)
Differentiating every term with respect to A, we have:
Qe(xii y (1 (1-7)Q(((T))
Rogg:)
V
log Qu(1n((re))
(138)
+ (1-7)Qa(Zz)
1 1-7 voQo(x)(
(139)
1 1 - -))
Volog(> (1 Qx(((((2)))
(140)
Minimizing HhaLe (0) equivalent to maximizing the following loss:
Eg- Σ
log(> + (1 (1=7) = Σ Po): (100)))0(((
(141)
2-1
L Exa-P
P(o((E log(7 + (1-7) LayeV
Σ
(142)
In others terms, this is equivalent maximize the following loss for all e {I,...; L) and Øett P. and denoting qi (0(2)) and pa P(n((aa):
((q) Σ pi log( (1-7)q))
(143)
By using the Lagrange multiplier method, to ensure that the sum of the probabilities is equal to one, we have the optimization problem:
pu log(y + (1-7)))) + ( 1
L
(xx, (1) = Σ
Σạ 1=1
(144)
1-1
Using KKT conditions, we have:
VI, Vx)( p1((1-) El=1 ===1111,
7+(1-7)9 1 = 0,
(145)
=1,
Which is equivalent to:
7
11.
p== p
+ 91
(146)
1-
By summing for all / (1...., L}, we have:
1-7
# =
(147)
1 1-7+Vy
28

---

When Temperature Fails, Change the Loss
and thus, for all I (I.......: L):
p1(1-7+Vy) V7)) -2 7
(148)
1
1-7
1-7
Finally, we can show that in that case:
S Pi + 11( -7 V7) -2 $ pt
(149)
1-7
+ p1 7 V+)) $ p((1-))+
(150)
2 m V
(151)
which concludes the proof.
First, us recall that precision di can be written as linear function of a divergence (x || Q) denoted PR-Divergence in Verine et al. (2023):
ax (P(Q)) min (A, 1) (P)II))
(152)
It can be shown, using Lemma 4.4.1 in Verine (2024) that:
1 Σ (A(())-Q(()) - 1/. 2 2
Dx(P Q)
(153)
Therefore, using similar arguments as Ji et (2023), we can bound the divergence between and Q by the divergence between the conditional distribution P(- wet)) and Q(: æ<:):
1 Σ
1
Dx(P
-(x)) 1A -1
(154)
2 20
1
L Q12 x211)
=
Σ
P (a)
- = - 1|
(155)
2
2
1-1
[-1
1
Σ
(
P(z (())(())))) xa))
1 [1 Q(1 xaa)
1 -/1x- 1/.
(156)
Zin
Using the triangular inequality, it can be shown that for every at, by I::
L
L x
[-]
L
Σ
bil
I
aj
π
6/
(157)
1=1
[=1
j-1
v=|+1
Thus, by taking a; b P(at wa) and b, = Q(1i æa). the PR-Divergence can be bounded by:
1 1 Σ Σ H((tt(zz) mes) (x xa) - Q(x1 ma))
A (Q(5)) (ee)))
Σ
2 1=1 1=10)
2110...1 j=1=1
(158)
| 1111/.
2
By marginalizing over the V/, P..1.... IL, the bound can be expressed as:
L
Σ
4
1 P(zs ma) Q( maa) 2 11 - 1/.
(159)
F=
21
Finally, we can show that:
L 1-l
D x (P(-1 (P(:1xa)) Q(- mmn)))
1 1
- -
Dx (PI
Q) $ E
P.p-
Σ
- 1
(160)
29

---

When Temperature Fails, Change the Loss
Now need to compute:
min X, Qe((z1) Pa (x1)
(P TTI) || Q(11 raa) min(A, 1) P((xa(
(161)
(162)
Now the challenge is that we do not have access to the ground truth distribution P. Thus, we can sample from the ø from the true distribution Par and approximate the expectation density using Definition 5.1:
Pa(x)) Pal (mi) = (mv=m)) (1 (1- (1-7))x((1))
(163)
where Q is the model distribution detached from the computation graph. In other words, voQ= = 0. However, to estimate the expectation, we can either use a one-step Monte Carlo approximation similarly to Ji et al. (2023):
E P((oe) min A, Qa(m))
]
min
A
(164)
Pei (x))
Duy (mi)
Qa(x1)
min
A y+ (1 Qx1
(165)
Qa((1))
(Q(((e))< q+ (1 Qa)))
(Qu((1126))) X
(166)
Ay with 6x 1-(1-7)X*
By differentiating the expectation, we can show that the gradient of the expectation is:
Qa((z)) Pai (Ii)
Vo min A Qu((xr)
V Em,- P( (mmm)
min X
(167)
Pun (a1) (
voQu((x))
(Qu(())<
(168)
+ (1 - - Qa(((()))
Qe log (cl
((Qu(s))os (1=)))((m;)))
(169)
Thus, using the Monte Carlo approximation and Assumption 5.1, we can show that minimizing the PR-Divergence equivalent maximizing the following loss:
Q0(x1(Z2) log Qo (0(2)/zz (0(2)(zee)
CPR-A-Mc(() -Eg
(170)
( + (1 (00))))((zzza))
22/9 where 8 1-(1-7)A477
30

---

When Temperature Fails, Change the Loss
C. Training Algorithms details
this section, we describe the training algorithm used to minimize a weighted Negative Log-Likelihood (NLL) loss. While the optimization loop remains the same across all methods, the weight computation w(x () varies depending on the chosen criterion: Trunc, TruncR, c-Div, X-PR.
The core idea is compute importance weights based on detached estimate of likelihood, denoted Q. This value is treated as constant during backpropagation, ensuring the weight computation does not interfere with gradient updates.
In Python, this is implemented using the .detach () method. For instance, if logp model (x), then logp. det.eetchh...coo () returns a tensor with the same values but without gradient tracking. This means Q can be used for weighting without contributing to the gradient computation itself.
Algorithm 1 Weighted NLL Training Algorithm
Require:
Training dataset D (x) fi-1, model Qo. method method (NLL, Trunc, TruncR, c-Div, A-PR), hyperparame- ters A, K for Trunc, a for c-Div, y and for - PR, learning rate rj
1: for each training step do
2: Sample batch
3: if' method NLL then
4: for each B and 7 do
5: w(x,1)
6: end for
7: else if method Trunc or method = TruneR then
8: Compute log- likellodd (xx) for all æ B
9: Add (=x) for all B to rolling list Q of size K
if method Trunc then
Compute threshold 5 for the highest x K values in Q for each B and do w(x, - 11 (Q(=)28)
12:
13:
14:
end for
15:
else if method TruncR then
16:
Compute threshold 5 for the lowest x K values in Q for each æ é B and do
17:
18:
wwx,1) - 11 (Q(w)s8)
19:
end for end if
20:
21:
else if method c-Div then
22: for each æ 8 and do
Compute (21)
w(x11) )) (x; 1/2
25:
end for
26:
else if method X-PR then
27:
for each æ B and / do Compute Qx (z))
28:
w(x,1) 7+ Q(l)))
Qu(()))
30:
end for
end if
32:
Compute gradient:
33:
g Vo (- E-Eers Σl= w(x 1) log Qa(()))
34:
Update parameters: 0 + 79
35: end for
31

---

When Temperature Fails, Change the Loss
D. Experiments
All experiments were conducted using Pytorch and HuggingFace Transformers. For MathQA-Python generation, we used vLLM library to speed up the generation process. We used both A100-80GB and H100-80GB GPUs for the experiments.
D.I. Models and training details
CodeContests. We benchmarked 1 8B/70B Instruct model before/after the RLEF, RLEF (Gehring et al., 2024) uses rule-based reward to fine-tune LLM with reinforcement learning. An interesting point, is that the increase of the Precision the cost of the drop of the Recall after RLEF in Figure echos the finding of Le Bronnec et al. (2024); Kirk al. (2024) that RL training could reduce the diversity of the model output.
Integer multiplication. We trained a small replica of a Llama transformer model, with hidden layers, an embedding dimension of 32, 4 attention heads and a feedforward dimension of 128. We trained the model using 25 000 samples. We used the Adam optimizer, with learning rate of 0.001, weight decay of 1, 500 epochs, and batch size of 512 sequences.
WritingPrompts & MathQA-Python. We fine-tuned models based on the pre-trained Olmo-1B and Llama3.2-31 models, All models, regardless of the dataset and the loss function used, were trained for 3 epochs. For non-NLL losses, fine-tuning started from checkpoint obtained after epoch of NLL training. For all training, used the Adam optimizer, with constant learning rate of 1e-6, with 1000 linear warmup steps. batch size of 8.
Instruction tuning on Alpaca. We fine-tuned Llama3.1 on the Alpaca dataset, to get a basic instruction-tune tuned model, capable of generalization. We the same training setup as for the other datasets. For Alpaca generation, we used reference temperature of 0.5 on most experiments, since we observed some degeneracies in the generation with a temperature of 1.0.
D.2. Evaluation Methods
In the evaluation phase in the Section 6, evaluate the model using different methods.
Number of tokens the Top-p
Max K top P
10a
10
p= 0.0
Max sumppe
Top 0.9
10
p
10
0.99
Top 0.99
10
10x
10a
10a
0.6
0.8
0.88
0.52
0.34
0.98
1.00
1.02
Proportion of samples
Top
(a) Repartition of the support of the conditional distribution.
(b) Quantiles the size of the support.
Figure 12: Support size of the conditional distribution P(x: x<) estimated using a reference model Llama3-88 B-------s-tteeee on the CodeContests dataset. The left plot shows the distribution of the number of tokens needed to cover different mass p (e.g., 0.9, 0.95, 0.99) for each position The right plot shows this numbers and the 10%, 50% and 90% quantiles.
Estimating the Sparsity of P via Token-Level Conditionals. In this experiment, we aim to provide an empirical proxy for the sparsity of the true target distribution P. By quantifying how many tokens carry most of the probability mass under strong reference model, we can validate the assumption that P is indeed sparse in real-world settings. We approximate the sparsity the target distribution P using a reference model Q assumed to approximate P reasonably well. Specifically, we assume that Q fits P correctly in structure but remains noisy in estimation. This reflects realistic modeling conditions, where Q captures the general shape of the target distribution but with limited precision due to model size or training noise. An
32

---

When Temperature Fails, Change the Loss
illustrative histogram of support sizes across examples is shown in Figure 12. We refer to this estimate as upper bound on the true sparsity of P, the method only considers local conditionals ( (ee1) and may include spurious high-entropy predictions not representative of the true support.
Regarding the use of the geometric mean to estimate sparsity: we rely on it because it is well suited to multiplicative or exponential behaviors, which naturally arise in token-level conditional probabilities language models. At each position sequence, we compute the number of tokens needed to cover fixed portion (e.g., 90%) of the total probability mass. Taking the maximum per sample, and then aggregating across the dataset using the geometric mean, allows us to capture robust notion of sparsity that penalizes extremely large values less than the arithmetic mean would (which is desirable since these rare), preserves icale-invariance (if all values are scaled by constant factor, the geometric mean scales accordingly), reflects the multiplicative nature of uncertainty across positions in sequence models. This approach aligns with best practices in probabilistic modeling, such as those discussed in Murphy (2013) and other works on log-loss and information content.
Algorithm 2 Estimating the Support Size of the Target Distribution
Require: Dataset D = ) reference model Qa (c.g., Llama3.1), I), coverage threshold e (0, 1)
for each sample E 'D do
2: for each position in the target sequence do
3: Compute conditional distribution Qo x<4)
4: Sort vocabulary tokens by decreasing probability
5: Compute minimal number k; of top tokens such that cumulative mass 2
6: end for
7: Let (m)x(x)) + max ky
8: end for
9: Compute geometrie mean of (xmax()))
10: Return geometric mean upper bound estimate of [Supp(P)
Precision and Recall for the Integers Multiplication Task. In this experiment, we evaluate the model using the Precision and Recall metrics. We compute these metrics using the following algorithm:
Algorithm Precision and Recall for Integer Multiplication
Require: Number of samples N, model Qo trained to generate sequence of digits 0102 bibb = 0102, where a; dy, C: E (0, 1..,...., 9} such that 0142 6,6y%%%% cj
1: Initialize unique- Ø
2: Initialize correct + 0
3: for each sample i 1.....N. NN do
4: Sample a sequence (1) from the model
5: if the sequence has the right format then
6: alaz x 6162%97 = 0102 then
7: correct correct +
8: if (2) & unique then unique unique (ajaz, 6162)
9:
10: end if
11: end if
12: end if
13:
end for
14: Compute Precision Precision
correct
N
15: Compute Recall Recall juniquel 99x99
16: Return Precision, Recall
Precision and Recall for the WritingPrompts Task. In this experiment, we evaluate the model using Precision and Recall, computed following the algorithm proposed by Le Bronnec al. (2024), described below. We used the same
33

---

When Temperature Fails, Change the Loss
hyperparameter than the original paper, with two differences: instead of using the embedding of the last token to featurize texts, we averaged the embeddings, and used 2,000 samples for the support estimation.
Algorithm 4 Precision and Recall for WritingPrompts
Require: Number of samples 2000, model Qe. dataset D (x(()) Ji=1 number neighbors & embedding model
④ GPT2-Large.
1: Estimate the support of the distribution P with D using k--N based on Kynkäänniemi et al. (2019) and Le Bronnec et al. (2024) with embedding model ø.
2: Initialize Q Ø
3: while Q < N do
4: Sample sequence (1) from the model and add it to
5: end while
6: Estimate the support of the distribution Qo with Q using k--N based on Kynkäänniemi al. (2019) and Le Bronnec
et al. (2024) with embedding model ø.
7: Count the number of samples Niae of Q that are in SuppP.
8: Count the number of samples Nreal of ID that are in SuppQa.
9: Compute Precision Ninic
N
10: Compute Recall Neae
N.cal 11: Return Precision, Recall
Precision and Recall for the Code Generation Task and MathQA-Python. We use the widely adopted pass@k metric tasks in CodeContests and MathQA-Python. Formally, pass@k estimates the probability that at least one of sampled completions is correct, which directly corresponds to Precision when A 1, and Recall as pass 100 pass @ 1. This accounts for diversity in model outputs under multiple draws.
to evaluate
The computation uses the unbiased estimator for pass (Chen al., 2021): given 2 k samples per prompt, let e be the number of correct completions. The unbiased estimate is:
if n
pass @ ks
otherwise
Algorithm 5 Precision and Recall for Code Generation
Require: Dataset D = (x) number of samples per prompt n., evaluation budget A:
1: Initialize sum pass 1 + 0
2: Initialize sum-pass.k + 0
3: for each prompt (+) D do
4: Generate completions y (6.1) ..... (f.m) using Qo
5: Let number of correct completions among the
6: Compute unbiased estimate:
pass k (0)
if n k
otherwise
sum_pass_1 + hti.ssummppp. sww.pass.l c/n
8: sum-pass.k + suw.pass.k k (
9:
end for
10: Precision sum_psss_1
11: Recall sum.pass.k Precision
Return Precision, Recall
34

---

When Temperature Fails, Change the Loss
D.3. Comparison with decoding methods.
To complement our analysis temperature-baser sampling, we report results for two alternative decoding strategies on the W PRITINGPROPPT dataset: Top-p sampling, with varying p and KL-guided temperature sampling (Chang al., 2023), reported to yield better diversity in question-anwering and summarization tasks.
We tested Top-p for E (0.1,0.5, 0.8, 0.9), and re-implemented KL-guided sampling, using guidance parameter or e 3.0,, 5.0, 10.0) (same range as in the original paper). All experiments used the same model and evaluation setup as in the main paper.
Table 5: Precision (P) and Recall (R) on WRITINGPROMPTS for different decoding methods. Note that for as a increases, KL-guided approaches standard sampling. For all methods we used a temperature of = 1.
Method
P
R
Base sampling,
0.848
0.086
Top-p. p= 0.1
0.997
0.001
Top-p. p= 0.5
0.996
0.001
Top-p, p= 0.8
0.886
0.033
Top-p. p = 0.9
0.805
0.058
KL, = 1.0
0.757
0.061
KL, = 3.0
0,800
0.068
KI., a = 5.0
0.831
0.069
KL, a = 10.0
0.844
0.086
From Table 5, we observe that Top-y sampling increases Precision at the expense of Recall. This aligns with intuition, as Top-p discards low-probability tokens, favoring more likely completions and thus leading to higher Precision.
In contrast, KL-guided sampling reduces both Precision and Recall overall. This may be due the fact that the method originally designed for conditional generation tasks such as summarization and question answering, which likely have different characteristics than our open-ended generation setting.
These results reinforce the need for Recall-oriented training losses, which offer a more effective way to improve the Precision-Reee Recaal trade-off than decoding- based methods alone.
35


Reviews:
[{"review_id": "0R6UlOWLtJ", "reviewer": "Reviewer_pQ4a", "summary": "Increasing diversity in language models requires careful tuning of decoding temperature. This paper shows that lowering temperature improves precision, but raising it often fails to enhance recall and effective tunability demands training models focusing on coverage. This paper provides two settings where the precision would fail to improve and recall can also go down provably. Then it provides a series of loss functions that can be used to perform fine-tuning on LLMs that claim to improve the recall. Then results are provided for three tasks, for all the models considered and proposed loss functions.\n\n\n\n## update after rebuttal\n\nI've updated my ratings reflecting my satisfaction with the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer pQ4a for the thoughtful feedback and careful review. We are grateful for the time and effort.\nFirst, regarding the algorithmic questions raised by several reviewers: we will include detailed algorithm blocks for all computations in the appendix.\n\nWe address explicit questions below and will incorporate all clarity improvements in the next revision.\n\n**Additional experiments**\n\nFollowing your suggestion, we added experiments on the MathQA-Python dataset (details in response to reviewer DUE2). We also trained a general-purpose instruction model on the Alpaca dataset, and observed a similar trend as in our initial experiments. We hope these results further support our method's generalizability.\n\n\n\n**Computing the support**\n- In practice, the support size of $P$ is indeed intractable. However, we can still obtain a rough characterization of the sparsity of $P$ (introduced in Thm 4.2) by leveraging the sparsity of the token-level conditional distributions $P(\\cdot \\mid x_{<l})$. To this end, we approximate the reference $P$ using a strong model $Q_{\\theta}$ (Llama3.1-8B in our experiments). For each sample and each token position $i$ in the solution, we compute the conditional distribution $Q_{\\theta}(\\cdot \\mid x_{<i}) \\in \\mathbb{R}^V$, where $V$ is the vocabulary size. At each position, we then determine the smallest number of tokens $n_{\\mathrm{top_p}}$ that account for $p \\in {0.9, 0.95, 0.99}$ of the total probability mass. For each sample, we take the maximum of these $n_{\\mathrm{top_p}}$ values across all positions, and finally, we compute the geometric mean of these maxima across the dataset to estimate the sparsity of $P$.\n\n- We refer to this estimate as an \"upper bound\" since the true sparsity of $P$ is likely lower than what we obtain from the conditional distributions alone.\n\n- We will make sure to include this explanation in the final version of the paper.\n\n**Confusing explanations**\n- Thanks for the feedback, we will better distinguish between PR over the full distribution and conditional PR.\n  \n**Conditions for an increasing Recall**\n- Yes, all conditions mentioned in Prop. 4.3 should hold simultaneously. In the case $\\rho \\approx 1$, the temperature does not help, as the model is already close to the target distribution.\n\n**Eq.18 and TruncR loss.**\n- We indeed differentiate only the terms outside the indicator function. The notation $\\bar{Q}$ indicates that we use the value of $Q$ in the implementation, but no gradient flows through it (achieved via `detach()` in PyTorch). In Eq. (18), this implies that the gradient of the sum is zero when the condition is not satisfied. We will add an algorithm block to clarify this in the final version.\n- Regarding $\\delta$, it characterizes the proportion of samples to keep. So, yes your interpretation is correct. During training, samples below the $\\delta$ threshold will have their likelihood increased. To maintain a fixed proportion of selected samples, $\\delta$ will gradually increase.\n\n**Prop 5.2 + teacher model**\n- Proposition 5.2 is proved in Appendix B.1, but is not clearly referenced in the main part. We will fix this. By \"difficult,\" we refer to the fact that both of both sampling from $Q$ and computing $P(\\cdot | x_{<l})$, since $P$ is unknown.\n- In a different setup, you're right that $P$ could be approximated by a teacher model. But in that case, the objective would be to match the Recall of the target model to that of the teacher, not to the true distribution. This, along with the idea of using RLHF to improve Recall, are very interesting directions for future work, but we believe they deviate somewhat from the main focus of our paper and theoretical analysis. We will nonetheless add a discussion on this in the final version of the paper.\n\n**Fig 4 PR**\n- Unlike Figures 5, 6, and 7, which display empirical values of Precision and Recall (computed using the metrics from Le Bronnec et al., 2024), Figure 4 is a theoretical illustration of the effect of each loss on the PR-curve, as in Verine et al., 2023. It is parametrized by $\\lambda$, and showing both $\\alpha_\\lambda$ and $\\beta_\\lambda$ is standard in that context. Our goal was to illustrate that optimizing the $\\lambda$-PR objective increases the corresponding tradeoff value for a given $\\lambda_0$. In particular, for $\\lambda_0 < 1$ (i.e., below the line $y = x$), Recall increases more than Precision.\n\n**Fig 6 $\\lambda$**\n- The three points corresponds to models trained with $\\lambda$-PR for $\\lambda=1$ (matching TailR loss), and some $\\lambda<1$. We observe that $\\lambda < 1$, Precision decreases while Recall increases, as expected.\n  \n**Tasks sparsity**\n- In addition to CodeContest, we evaluated the sparsity of WritingPrompts (plotted in the table below), a creative task. The sparsity is lower than for CodeContest, which could be expected as this is a less constrained task.\n  \nTop-p  | 0.9   | 0.95  | 0.99  \n- | - | - | - |\n|Supp(P)\\|/V | 3.86% | 7.80% | 24.9%", "claims_and_evidence": "The mentioned proofs and assumptions are correct. The evidence is clear and convincing and the claims aren't problematic. I do have certain questions on how certain quantities are computed, which I've deferred for the later section.", "methods_and_evaluation_criteria": "Need more datasets, such as MATH, AIME2024, MathQA-Python.", "theoretical_claims": "The theoretical claims are correct. I've read through the appendix and examined the two artificial cases closely. I think assumptions are reasonable, and the two cases span the most practical settings. Overall I like the math aspect quite a lot.", "experimental_designs_or_analyses": "The experimental aspect seems correct. However, the presentation for some of the graphics can be improved to make it widely appreciable by people with visual impairment. \n\nI think more datasets can be used where it is often desirable to generate multiple solutions and then used in conjunction with a verifier (see -- Generative Verifiers: Reward Modeling as Next-Token Prediction). Such datasets may be -- MATH, AIME 2024.", "relation_to_broader_scientific_literature": "Improving on PR is an important task, and increasing temperature is something people think can lead to nice and diverse solutions. Therefore, I think this paper is a good contribution to a variety of sub-community in LLM, including synthetic data generation to verification.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Following are my questions --  \n\n1. I am not fully able to understand how the support is computed in practice, or how M can be computed. Why can one find an upper bound on the cardinality of support set by the geometric mean of the $\\max_{\\ell}\\mathcal{S}(x_{\\ell})$?? Shouldn't cardinality be monotonic with $\\ell$ and with that how does this max operation over $\\ell$ not have a trivial solution?\n2. How is the top-p region estimated? The writing is not very clear, and I am not sure what is being done algorithmically. I'd appreciate an algorithmic block for every computation done in the paper.\n3. For the toy example, it would be good to remind the readers that the actual PR is computed over the entire distribution till length L and not on the conditionals. I got very confused initially since I was thinking about conditional distribution PR, and it was clear only when I read the proof. \n4. Do all mentioned conditions to have a strictly increasing recall have to hold simultaneously? Moreover, what happens when $\\rho \\approx 1$ , that is, Q is very close to P? \n5. I am quite a bit confused about the turnc loss function, and its practical implementation aspect. For the starters, what is $\\bar{Q}$? When we are training, initially the model I am assuming should have a lower $\\delta$ but over time $\\delta$ should increase, right? \n6. Can authors add proof to Proposition 5.2? Below the same prop, why is a sampling from Q difficult? Moreover, if someone swaps the P and Q, isn't it somewhat similar to PPO, assuming the log-likelihood under the true distribution is the reward model (which we can replace with a teacher model's likelihood function)? This makes me feel that there should be a natural baseline like this in the work (formulating and using RLHF to improve recall) \n7. Equation 18 seems non-differentiable as it has a parameter inside an indicator function. This needs clarification and again an algorithmic block. \n8. Why is $\\lambda$-PR separately plotted (which increases as y=x line) as another method to improve PR in Fig4? Isn't it the case that all the PR plots when varying temperatures are at fixed \\lambda? What is even the use case of having a plot between $\\alpha_{\\lambda}$ and $\\beta_{\\lambda}$ if truly one always varies $\\lambda$ in the entire range [0, $\\infty$]\n9. For Fig 6, what is the $\\lambda$, otherwise if it is plotted for all lambda then it kind of conflates the trend with different hyperparameters or loss functions. \n10. Which of the considered tasks have the sparsest distribution?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVIlezvGZD", "reviewer": "Reviewer_DUE2", "summary": "The paper provides a detailed analysis of the relationship between temperature, precision, and recall, offering insights into why lowering the temperature improves quality (precision), while increasing the temperature usually does not enhance coverage (recall). The paper primarily addresses two key questions: the impact of temperature adjustment on the precision-recall trade-off in language models, and how to train models to improve recall. By proposing recall-oriented loss functions, it presents a method to achieve a better P&R trade-off through temperature scaling, and validates this approach experimentally. The main contributions of the paper are as follows:\n\n1. Analysis of the impact of temperature scaling on the P&R trade-off\n2. Proposal of recall-oriented loss functions\n3. Experimental validation of theoretical findings", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer DUE2 for the review and constructive feedback. We have added further experiments with more datasets and models; please refer to our response to Reviewer 2owJ for details. While the first question has only one relevant scenario, we believe the second and third questions require multiple scenarios to answer meaningfully.\n\n**The authors provide three scenarios to answer the three proposed questions, yet in the subsequent analysis, not all of these questions seem to be fully addressed by the three given scenarios. Therefore, additional experiments may be needed to ensure that the issues claimed in the three scenarios are adequately answered (e.g., in Section 6.4, only the WritingPrompts dataset task is discussed).**\n\n\nFollowing your suggestion, as well as those from other reviewers, we conducted additional experiments within the limited available time to further investigate the effect of our Recall-oriented losses.  (Similar response as to Reviewer 2owJ)\nThe new experiments include:\n\n- **New dataset: MathQA-Python.**  \n  We trained a model on the MathQA-Python dataset. Using the same proxies for evaluating Precision and Recall as in CodeContest.\n\n- **New model: Llama3.2-3b WritingPrompts.**  \n  We trained an additional model on the WritingPrompts dataset.\n\n- **General-purpose instruction model.**  \n  We trained a general-purpose instruction model on the Alpaca dataset using our proposed losses, $\\lambda$-PR and $c$-Div, to demonstrate their effectiveness on broader instruction tuning tasks. We then evaluated this model on both the WritingPrompts and MathQA-Python datasets.\n\nWe report the results in the tables below and will extend Table 1 accordingly in the final version of the paper.\n\n\n### Olmo1b MathQA-Python\n\n| Alpha                                            | P    | R    |\n| ------------------------------------------------ | ---- | ---- |\n| NLL                                              | 0.42 | 0.36 |\n| c-Div (c=1.4)                                    | 0.30 | 0.46 |\n| $\\lambda$-PR ($\\lambda = 0.1, \\gamma = 1e^{-7}$) | 0.06 | 0.48 |\n| TruncR  ($\\Delta = 0.1$)                         | 0.29 | 0.43 |\n\nAn interesting observation is that $\\lambda$-PR impacts Precision more severely than the other losses, but ultimately achieves a high Recall. This suggests that the resulting model offers better coverage of the target distribution.  \n\n\n### Llama3.2-3b WritingPrompts\n| Method                                           | P    | R    |\n| ------------------------------------------------ | ---- | ---- |\n| NLL (MLE)                                        | 0.77 | 0.08 |\n| c-Div ($c = 1.3$)                                | 0.72 | 0.17 |\n| $\\lambda$-PR ($\\lambda = 0.9, \\gamma = 1e^{-5}$) | 0.59 | 0.19 |\n\nNote that for this model, which is larger than Olmo1b, we did not benchmark the TruncR loss, as the current implementation is not compatible with distributed training.  \nWe also observed that the model tuned with $\\lambda$-PR was much more sensitive to the sampling temperature. We used $t=0.5$, as higher values led to some degeneracies. However, due to limited time, we could not investigate this further.\n\n### Alpaca WritingPrompts\n| Method                                           | P    | R    |\n| ------------------------------------------------ | ---- | ---- |\n| NLL                                              | 0.83 | 0.04 |\n| c-Div ($c = 1.4$)                                | 0.82 | 0.12 |\n| $\\lambda$-PR ($\\lambda = 0.5, \\gamma = 1e^{-7}$) | 0.57 | 0.26 |\n\n### Alpaca MathQA-Python\n| Method                                           | P    | R    |\n| ------------------------------------------------ | ---- | ---- |\n| NLL                                              | 0.09 | 0.39 |\n| c-Div ($c = 1.4$)                                | 0.08 | 0.43 |\n| $\\lambda$-PR ($\\lambda = 0.1, \\gamma = 1e^{-5}$) | 0.08 | 0.42 |\n\nWe started from a pre-trained Llama3.1-8B model and used the same training setup as described in the paper. This yields a basic instruction-tuned model, capable of generalization (but still with limited capacity compared to SOTA models).  \nNote that for all experiments conducted on Alpaca, we used a temperature of $t=0.5$ to avoid degeneracies. As with previous experiments, we could not benchmark the TruncR loss due to its incompatibility with distributed training.\n\n### Analysis\n\nWe observe the same pattern as in the initial experiments. This confirms that our losses can achieve higher Recall than NLL, which we believe strengthens the claims and findings presented in the paper.", "claims_and_evidence": "The author’s experiments and theoretical demonstrations extensively validate their claims:\n\n1. The analysis of the impact of temperature on P&R is thorough and detailed, and the improvements to the loss function for the P&R trade-off have achieved the stated effects.\n2. The experimental section sufficiently validates the relevant theoretical findings through three different scenarios.", "methods_and_evaluation_criteria": "1. The method designed by the author analyzes the relevant impact of temperature on P&R, and the method design is reasonable.\n2. The author compares a sufficient number of baselines.", "theoretical_claims": "The theoretical proof results in the paper are clear and well-presented, with complete and detailed proofs provided, although I did not check all the details.", "experimental_designs_or_analyses": "The experimental design by the authors is adequate, but some additional experiments might be necessary:\n\n1. The authors provide three scenarios to answer the three proposed questions, yet in the subsequent analysis, not all of these questions seem to be fully addressed by the three given scenarios. Therefore, additional experiments may be needed to ensure that the issues claimed in the three scenarios are adequately answered (e.g., in Section 6.4, only the WritingPrompts dataset task is discussed).\n2. The analysis of the experiments in Section 6 is somewhat confusing, and the appearance of some figures is not clearly related to the problems they aim to address.", "relation_to_broader_scientific_literature": "I haven't found it yet.", "essential_references_not_discussed": "I haven't found it yet.", "other_strengths_and_weaknesses": "I haven't found it yet.", "other_comments_or_suggestions": "Some images appear in the main text but are not mentioned in the text.", "questions_for_authors": "Please refer to the above parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iveKwEoi0I", "reviewer": "Reviewer_2owJ", "summary": "This paper studies how recall and precision can be effectively traded off in language models. First, they study formal definitions of precision and recall in simplified settings and show cases where decreasing the temperature improves precision at the cost of recall, but increasing the temperature hurts both precision and recall. Motivated by the fact that it seems easier to improve precision than recall via temperature adjustment, they then propose recall-oriented loss functions. Empirically, (1) they confirm that it is difficult to improve recall by increasing temperature, and (2) they show that fine-tuning with recall-oriented loss functions and then decreasing temperature leads to a better precision-recall tradeoff than starting with a normally trained model and increasing temperature.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer 2owJ for the detailed review and suggestions. We are glad you appreciate the paper.\n\nFollowing your suggestion, as well as those from other reviewers, we conducted additional experiments:\n\n**New dataset: MathQA-Python.**  \nWe trained a model on MathQA-Python and used the same evaluation as for CodeContest: Precision as pass@1, and Recall as the gain obtained from sampling (pass@100 - pass@1). We chose MathQA-Python over CodeContest because it offers more training data and is better suited to the capabilities of our models.\n\n**New model: Llama3.2-3b WritingPrompts.**  \n  We trained an additional model on the WritingPrompts dataset.\n\n**General-purpose instruction model.**  \n  We trained a general-purpose instruction model on the Alpaca dataset using our proposed losses, $\\lambda$-PR and $c$-Div, to demonstrate their effectiveness on broader instruction tuning tasks and more expressive models. We then evaluated this model on both the WritingPrompts and MathQA-Python datasets. \n\nWe report the results below and will update Table 1 in the final version. While time constraints limited the scope, we believe these additional experiments provide a strong indication the effectiveness of our losses on more tasks, models and metrics.\n\n\n### Olmo1b MathQA-Python\n\n| Alpha                                            | P    | R    |\n| - | - | - |\n| NLL                                              | 0.42 | 0.36 |\n| c-Div (c=1.4)                                    | 0.30 | 0.46 |\n| $\\lambda$-PR ($\\lambda = 0.1, \\gamma = 1e^{-7}$) | 0.06 | 0.48 |\n| TruncR  ($\\Delta = 0.1$)                         | 0.29 | 0.43 |\n\nAn interesting observation is that $\\lambda$-PR impacts Precision more severely than the other losses, but ultimately achieves a high Recall. This suggests that the resulting model offers better coverage of the target distribution.  \n\n\n### Llama3.2-3b WritingPrompts\n| Method                                           | P    | R    |\n| - | - | - |\n| NLL (MLE)                                        | 0.77 | 0.08 |\n| c-Div ($c = 1.3$)                                | 0.72 | 0.17 |\n| $\\lambda$-PR ($\\lambda = 0.9, \\gamma = 1e^{-5}$) | 0.59 | 0.19 |\n\nNote that for this larger model (compared to Olmo1b), we did not benchmark the TruncR loss due to its incompatibility with distributed training in the current implementation.\nWe also observed that the model tuned with $\\lambda$-PR was much more sensitive to the sampling temperature. We used $t=0.5$, as higher values led to some degeneracies. However, we could not investigate this further.\n\n### Alpaca WritingPrompts\n| Method                                           | P    | R    |\n| - | ---- | ---- |\n| NLL                                              | 0.83 | 0.04 |\n| c-Div ($c = 1.4$)                                | 0.82 | 0.12 |\n| $\\lambda$-PR ($\\lambda = 0.5, \\gamma = 1e^{-7}$) | 0.57 | 0.26 |\n\n### Alpaca MathQA-Python\n| Method                                           | P    | R    |\n| -- | ---- | ---- |\n| NLL                                              | 0.09 | 0.39 |\n| c-Div ($c = 1.4$)                                | 0.08 | 0.43 |\n| $\\lambda$-PR ($\\lambda = 0.1, \\gamma = 1e^{-5}$) | 0.08 | 0.42 |\n\nAll Alpaca experiments used a temperature of $t=0.5$ to avoid degeneracies.\n\n### Analysis\n\nWe observe the same pattern as in the initial experiments. This confirms that our losses can achieve higher Recall than NLL, which we believe strengthens the claims and findings presented in the paper.\n\n\n### Section 6.2\n- For Section 6.2, we will reformulate the text to clarify that the experiments are designed to verify the assumptions behind the theoretical analysis, not its results.\n\n### Experiments\n\n- We used hyperparameters very similar to those described in the original Olmo paper (we only used a smaller learning rate of 1e-6 instead of 2e-6). To ensure comparability, we used the same optimization parameters for all losses.\n- For the PR metrics, we used the exact same setup as in the original paper (Le Bronnec et al., 2024), i.e., GPT2-large as the embedding model.\n- We trained all models under the same conditions (same number of epochs, same batch size, same optimizer, etc.) on the training set and reported the metrics on the validation set.\n\n\nWe will incorporate these details in the appendix of the paper.\n\n### Suggestions\n- We will gladly incorporate the suggested reformulations, it will indeed improve the overall flow of the paper. We could indeed move some parts to the appendix and spend more time discussing the idea behind the model.\n\nFor a side note, in Theorem 4.2, the bounds rely on *almost* no assumption on the target distribution or the model, and the bound is not necessarily useful for large $t$, especially if $Z$ is low. For instance, when the model is uncertain (i.e., $Z$ is low), the bound may still be useful even for $t < 1$. But we agree that this is more a general theoretical result than a practical one.", "claims_and_evidence": "I really like the direction that the paper is going, as the claims are interesting and the problem is important. However, the experiments presented in the paper do not sufficiently support the claims, as discussed below. One other minor limitation of the paper is that I find the writing somewhat confusing (will provide concrete suggestions in a later section).\n\n- The first main claim is that lowering temperature improves precision at the cost of recall, but increasing temperature typically harms both after a certain point. This claim is supported by experiments.\n\n- The second main claim is that fine-tuning with recall-oriented loss and adjusting temperature attains a better precision-recall curve than doing normal NLL training and adjusting temperature. This claim is weakly supported by a single experiment on WritingPrompts. I find this experiment insufficient because:\n  - (a) Because there is no \"downstream\" measure of precision and recall, they instead rely on an automatic measure based on a previously proposed method involving embedding the texts and measuring precision and recall in the embedded space. This evaluation setup provides useful evidence, but automatic metrics have been found to be flawed (see, e.g., [Gehrmann, Clark, and Sellam 2023](https://arxiv.org/abs/2202.06935)). Therefore, I think this claim needs to be evaluated with more metrics. One good one would be looking at pass@k on CodeContests, which the authors use as a dataset for other parts of the paper.\n  - (b) I also think this claim needs to be evaluated on more than one model and more than one dataset.\n\n- Minor point: section 6.2 claims that it \"empirically confirms the theoretical insights from Theorem 4.2,\" but as far as I can tell it only measures support sparsity at various cutoffs, when Theorem 4.2 makes specific predictions about upper bounds for precision and recall measures. So the authors' claims that their experiments confirm their theory does not seem well-supported by evidence.", "methods_and_evaluation_criteria": "The methods and datasets make sense for the problem.", "theoretical_claims": "I read the theoretical claims in the main text and they seem reasonable. I did not check the proofs.", "experimental_designs_or_analyses": "- The paper provides some description of the experiments but does not describe them in enough detail for me to judge their soundness with confidence. Nonetheless, the experiments seem sound at first glance. Some details that I could not find:\n  - How hyperparameters were chosen\n  - Which model is used to produce embeddings for the automatic P&R metrics\n  - Whether there was a train-test-val split", "relation_to_broader_scientific_literature": "- The first contribution, studying how temperature affects precision and recall, seems well-studied. Nonetheless, the paper provides useful further experiments for this question.\n- The second contribution, training LMs for recall to attain a better precision-recall tradeoff, seems new and valuable.\n- Theory in section 4 (characterizing P&R tradeoffs in simplified settings): I found this analysis interesting but would characterize it more as a supplement and motivation for the other claims in the paper, rather than a standalone theoretical contribution.\n- Theory in section 5 (characterizing the proposed recall-oriented losses): I see this theory as supplementing the main contribution of proposing recall-oriented training.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Some writing suggestions:\n- Overall, I think the paper could be streamlined in terms of making it clearer how the theory motivates the proposed solution, as well as omitting tangential results or making it clearer that they are tangential. For example, I did not find theorem 4.2 to be particularly compelling because as far as I understand, the bound is only useful for large $t$, but in practice people do not take temperature to be very far from 1. \n- The second part of Section 4 is framed as using an artificial setting to make the prediction that very high temperature decreases both P&R. However, it's already well-known empirically that setting temperature very high is not effective. Nonetheless, I found the setting interesting because it could provide some intuition on the factors at play. I would suggest spending more time talking about the intuition behind the model, like how different factors affect the effect of temperature, rather than centering the section around the specific claim that high temperatures are harmful.\n- I found it confusing to refer to the previous methods as \"baselines\" because they are not solving the same problem as this paper. I think it would be clearer to just refer to these methods as the precision-focused versions of the recall-focused losses in this work.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjJh7d9EgG", "reviewer": "Reviewer_2yG3", "summary": "The paper investigates the impact of temperature scaling on the precision–recall (P&R) trade-off in language models. The authors provide a theoretical analysis showing that while lowering the temperature enhances precision, increasing it does not necessarily improve recall. They propose new loss functions (e.g., TruncR, c-Div, and λ-PR) to train models that emphasize recall, thereby allowing for a more balanced diversity–quality trade-off when temperature scaling is applied. Experimental evaluations on tasks such as code generation, integer multiplication, and writing prompts are used to validate the theoretical insights.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and your feedback.\n\n## Decoding-based methods\n\nIn our paper, we deliberately focused on the temperature parameter (commonly referred to as the \"diversity\" parameter) because its theoretical analysis already presented some complexity. For this reason, we chose to limit the scope of this study to ensure a focused exploration of the temperature parameter.\nHowever, we agree that comparing empirically with other decoding-based methods could bring other insights.\n\nAs suggested, we investigated the effect of the following methods on the WritingPrompts dataset:\n- top-p (or nucleus) sampling\n- KL-Divergence Guided Temperature Sampling\n\nWe re-implemented the KL-Guided sampling from scratch, since the original implementation was not compatible with our PyTorch models. We used the same range for $\\sigma$ as in the original paper.\n\n|     | P     | R     |\n| --- | ----- | ----- |\n| NLL | 0.848 | 0.086 |\n\n\n#### Top-p\n| p   | Precision | Recall |\n| --- | --------- | ------ |\n| 0.1 | 0.997     | 0.001  |\n| 0.5 | 0.996     | 0.001  |\n| 0.8 | 0.886     | 0.033  |\n| 0.9 | 0.805     | 0.058  |\n\n\n#### KL-Guided\n\n| $\\sigma$ | P     | R     |\n| -------- | ----- | ----- |\n| 1.0      | 0.757 | 0.061 |\n| 3.0      | 0.800 | 0.068 |\n| 5.0      | 0.831 | 0.069 |\n| 10.0     | 0.844 | 0.086 |\n\n(As $\\sigma$ increases, KL-Guided approaches standard decoding.)\n\nWe can conclude that Top-p increases Precision at the cost of Recall. Since Top-p removes the least probable tokens, we could intuitevely expect a higher Precision.\n\nKL-Guided seems to globally decrease both P&R. We believe that an explanation would be that this method has been mainly designed for conditional text generation, like summarization and question answering, which might exhibit specificites compared to our generation task.\n\nOverall, we believe that these results provide more grounding on the necessity of using Recall-oriented losses as opposed to simple decoding-based methods.\n\n## General models\n\nTo answer your question, we trained a general-purpose instruction model on the Alpaca dataset using our proposed losses, $\\lambda$-PR and $c$-Div, to demonstrate their effectiveness on broader instruction tuning tasks. We then evaluated this model on both the WritingPrompts and MathQA-Python datasets.\n\nFor MathQA-Python, which requires generating Python code from natural language questions, we used 3 in-context examples to prompt the model and evaluated it as in CodeContest.\nWe used the same evaluation as for CodeContest. We chose MathQA over CodeContest because it offers more training data and is better suited to the capabilities of our models.\n\n### Alpaca MathQA-Python\n| Method                                            | P     | R    |\n| -- | ----- | ---- |\n| NLL                                               | 0.088 | 0.39 |\n| c-Div ($c = 1.4$)                                 | 0.084 | 0.43 |\n| $\\lambda$-PR ($\\lambda = 0.75, \\gamma = 1e^{-5}$) | 0.083 | 0.46 |\n\nOn these results, we see that overall, our losses do not significantly impact Precision, meaning that generalization/in-context ability is not affected. However, we observe a significant increase in Recall, which is consistent with our previous findings. \n\n### Alpaca WritingPrompts\n\n| Method                                            | P    | R     |\n| -- | --- | ----- |\n| NLL                                               | 0.83 | 0.040 |\n| c-Div ($c = 1.4$)                                 | 0.82 | 0.12  |\n| $\\lambda$-PR ($\\lambda = 0.75, \\gamma = 1e^{-5}$) | 0.51 | 0.21  |\n\nOn the WritingPrompts dataset, we observe that general models trained with our losses on the Alpaca dataset exhibit a similar pattern to the specialized models. We see a significant increase in Recall, sometimes at the expense of Precision.\n\nThis suggests that our losses can consistently improve Recall even on more general models and tasks. We hope these additional experiments provide a more comprehensive view of the effectiveness of our proposed losses.\n\n### Suggestions\n- We will add more details about P&R in the appendix, this should help clarify the evaluation process.\n- We computed MAUVE metrics for the WritingPrompts dataset. We will add it to Table 1.\n  \n| Method  | MAUVE     |\n| -- | ----- |\n| Trunc   | 0.074 |\n| GOLD    | 0.005 |\n| Tailr   | 0.087 |\n| c-Div   | 0.068 |\n| Trunc-R | 0.073 |\n| λ-PR    | 0.096 |\n| NLL     | 0.104 |\n\n\n- For 243–244, we mean that image generation can be optimized for specific PR tradeoffs using f-divergences, as shown by Verine et al. (2023). However, these methods rely on assumptions that don’t hold for text due to its causal nature. We’ll clarify this in the final version.\n- In code generation, pass@100–pass@1 reflects the gain from sampling multiple candidates over one. In your example, the metric is relevant: if samples differ only slightly (e.g., variable names), structure diversity is low and pass@1 is likely high. We’ll clarify this with more details on the evaluation process.", "claims_and_evidence": "- The authors provide a theoretical investigation into how temperature affects P&R, providing analyses that explain the observed limitations of temperature scaling. However, Artificial case analysis is limited, as only a small set of cases are considered. Cannot give a general insight about the trends of P&R when varying temperature in general cases.", "methods_and_evaluation_criteria": "Strength\n- Viewing the quality-diversity trade-off using precision and recall is interesting (however, it’s not new). The alternative loss functions to train models for higher recall seem to be effective. By shifting the focus from decoding adjustments to training objectives, the authors provide a new angle on tackling the quality–diversity trade-off.\n\n\nWeakness\n- While the claim in the main paper is general (change the loss function when the temperature sampling is based), there is a lack of comparison with existing decoding-based works that promote diversity along with quality. This is essential to understand the effectiveness compared to simpler methods (no need to train with a different loss function). One example is Chang et al. KL-Divergence Guided Temperature Sampling.\n    \n- The effects of changing the loss function on other tasks are not investigated. For example, does changing the loss function affect the generalization/in-context learning ability of LMs? This affects the utility of the proposed methods for general use.", "theoretical_claims": "I have not closely verified the proofs.", "experimental_designs_or_analyses": "- The experiment setup and evaluation criteria have some concerns (see comments).", "relation_to_broader_scientific_literature": "Most of the existing work on prompting diversity or balancing diversity-quality trade-off in LLMs is in the decoding phase. This paper brings new insight and claim that we should adjust the loss function to promote the effectiveness of temperature-scaling in decoding phase.", "essential_references_not_discussed": "- Chang et al. KL-Divergence Guided Temperature Sampling.\n\n- Lu et al. Diver: Large Language Model Decoding with Span-Level Mutual Information Verification\n\n- Zhang et al. Trading Off Diversity and Quality in Natural Language Generation\n\n- Chung et al. Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions", "other_strengths_and_weaknesses": "See comments and questions", "other_comments_or_suggestions": "- For the completeness of the paper, I suggest to discuss the detailed computation of Precision and Recall metrics for writing prompts experiment in the appendix.\n    \n- MAUVE and Average Pairwise Cosine Similarity are more common metrics to evaluate the quality and diversity of LLM responses. It would be better if the authors can evaluate the proposed methods on those metrics to give diverse insights.", "questions_for_authors": "- Sentences 243 244 are vague to me and does not give enough context for the next drawback claim. What are specific trade-offs (is it about P&R only or about general measures? minimizing alternative f-divergences between what? Citations need for this sentence as well.)\n    \n- Why pass@100 - pass@1 measures the diversity of generated samples? if all generated samples are the most same with some different tokens. In this case, the pass@100 is high but diversity is low?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper addresses the challenge of improving diversity in language models by analyzing the limitations of adjusting decoding temperature, demonstrating theoretically and empirically why increased temperature often fails to enhance recall. The authors propose alternative recall-oriented loss functions (e.g., TruncR, c-Div, λ-PR) that, when combined with temperature scaling, yield better precision-recall trade-offs. Extensive theoretical analyses are supported by comprehensive experimental validation across multiple datasets, demonstrating generalizability and effectiveness.

Strengths:
- Theoretical insights clearly justify the inadequacy of traditional temperature scaling in enhancing recall.
- Novel loss functions provide a robust and effective method to achieve improved recall without severely compromising precision.
- Comprehensive empirical validation across diverse scenarios, models, and tasks strengthens the paper’s findings.

Weaknesses:
- Initial clarity issues regarding the computation of certain metrics and theoretical formulations, although largely resolved in rebuttal.
- Early experimental results somewhat limited, subsequently strengthened significantly through rebuttal phase additions.

Overall, the paper contributes valuable theoretical and empirical insights into a fundamental problem in language modeling. After incorporating additional clarifications and experiments, reviewers uniformly raised their scores, emphasizing the significance and thoroughness of the contributions. Therefore, this paper is recommended for acceptance.
Decision: Accept (poster)