Exploring Precision and Recall to assess the quality and diversity of LLMs (2024)

Abstract: We introduce a novel evaluation framework for Large Language Models (LLMs) such as LLAMA-2 and MISTRAL, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data1.

heading: 1 Introduction
text: In recent years and months, there has been a rapid democratization of Large Language Models (LLMs), exemplified by platforms such as ChatGPT (OpenAI et al., 2024) and HuggingChat (Hugging Face, 2024). These models are now widely accessible for a diverse range of tasks, from composing emails and application letters to performing multi-document summarization, generating medical prescriptions, and even crafting poetry and novels (Hosseini et al., 2023; Skjuve et al., 2023). The expanding spectrum of applications underscores the ubiquity of LLMs in our daily lives, necessitating the development of novel evaluation frameworks to accommodate their growing relevance.
*Authors contributed equally to this work. Corresponding authors: florian.le-bronnec@dauphine.psl.eu, alexandre.verine@dauphine.psl.eu.
1https://github.com/AlexVerine/pr-4-llm
Until recently, benchmarks were designed to target specific tasks such as machine translation, summarization, and question answering, among others. However, LLMs now encompass a wide range of tasks, prompting the community to reconsider methods for comparing and assessing these models. One proposed solution is to gather a diverse set of tasks to better evaluate the versatility of modern generative models. For example, the Open LLM Leaderboard (Beeching et al., 2023) presents a unified framework consisting of closed questions that focus on the model’s ability to provide concise and accurate answers. However, it is important to note that these evaluations are sample-based, relying on generated samples from the model compared against aligned references, written by humans.
In contrast, the recent line of work on distribution-based metrics greatly departs from the sampled based evaluation. By considering LLMs and datasets as empirical distributions, the new metrics attempt to quantify how they differ and how they overlap. This shift drastically changes the scope of the evaluation. Beyond performance
11418
measures based on human references, the objective is to estimate the disparity between some data distribution (P ) exhibited by human-written texts and the distribution learned by a LLM (Q), eliminating the need for aligned corpora. As exemplified by the development of MAUVE (Pillutla et al., 2021), this kind of approach opens new perspectives for comparing LLMs in terms of their generative abilities rather than for some peculiar tasks.
However, many factors take part in assessing open-ended text generation and focusing on a single measure may restrict the significance of the evaluation. This is well known in the field of Information Retrieval, for example, where Precision and Recall are at the core of evaluation. Although the usage of Precision and Recall for generative models has been previously used in image generation by Sajjadi et al. (2018) and Kynkäänniemi et al. (2019), we propose to adapt them to LLMs. With an extensive set of experiments involving prominent LLMs such as LLAMA-2 (Touvron et al., 2023) and Mistral (Jiang et al., 2023), we show that these two new metrics significantly improve the evaluation with a better understanding and characterization of the flaws in text generation. Precision and Recall allow us to clearly distinguish between samples quality or adequacy (for Precision) and a lack of diversity in the model outputs (for Recall). Empirical results show that these two measures are necessary for a in-depth comparison of LLMs. For instance, we are able to quantify that fine-tuning models on instruction sets with human feedback significantly improves sample quality, albeit at the expense of sample diversity, as evidenced in Figure 1 by the trade-off between Precision and Recall.
As a summary, our contributions are threefold:
• We adapt the concepts of Precision and Recall, traditionally used in image generation, to evaluate the performance of large language models. Our method offers a novel lens to assess the quality and diversity of text generation without the need for aligned corpora.
• We carry out a thorough evaluation of state-ofthe-art language models, such as LLAMA and MISTRAL, using our proposed framework. This analysis provides a detailed comparison of the performance of these models in terms of quality and diversity on challenging open-ended generation tasks that are out of the scope of traditional evaluation methods.
• Our investigation sheds light on the impact of
fine-tuning LLMs as chat models. We present empirical evidence that, while such fine-tuning can improve the quality of generated samples, it also reduce their diversity, highlighting a crucial trade-off between Precision and Recall.
In summary, our contributions advance the field of distribution-based NLP evaluation by introducing novel metrics tailored to the specific tasks of open-ended generation. Through empirical analysis and insights, we provide a deeper understanding of LLM generation capabilities.

heading: 2 Related works
text: Historically, assessing text generation tasks has been dependent on human resources, such as annotated texts or aligned corpora. This is particularly true in areas like machine translation and automatic summarization, where N-gram based metrics have been developed, notably BLEU (Papineni et al., 2001) and ROUGE (Lin, 2004). These metrics perform a rough comparison between the generated text and human-produced and aligned reference texts. More recent developments have introduced model-based metrics, such as BERTScore2 (Zhang* et al., 2020), which aim to capture semantic similarities more effectively. However, these evaluations still largely depend on "aligned references," meaning human-provided examples that constrain the expected range of generated outputs. The challenge becomes significantly greater with open-ended generation tasks, where the range of acceptable responses is vast and cannot be encapsulated by a single reference.

heading: 2.1 Standard evaluation of generative models
text: Perplexity is the historical measure for language modeling (Bahl et al., 1977). Easy to use and cheap to compute, practical motivations clearly explain its persistence for evaluation, despite its notable limitations. As demonstrated in the work of Pillutla et al. (2021), perplexity works at the token level, only considering the surface forms, hence missing semantic information. Moreover, the metric is calculated on a per-sample basis, thereby failing to capture the diversity of generated texts.
Nowadays, the prevalent method for evaluating recent LLMs involves benchmarks with many tasks with easily verifiable correct answers (Touvron
2BERTScore uses precision and recall scores that differ significantly from ours, as they evaluate the similarity between a single generated text and its aligned reference.
et al., 2023; Google, 2023; Beeching et al., 2023). Typically, this involves the generation of responses to short questions prompted in the instructions. The responses are then compared to a reference using metrics such as Exact-Match. While effective in assessing the comprehension and reasoning capabilities of LLMs, this kind of approach does not assess the generation skills of the models.
To overcome the limitations of automatic metrics, another trend considers human evaluation (Celikyilmaz et al., 2021), especially for complex tasks. Beyond its cost, the guidelines must be carefully designed, since various criteria could be necessary to rate texts and compare model outputs. In the end, human evaluation is prone to high variance and is difficult to reproduce. Another issue is that diversity is not evaluated is standard human evaluation protocols, since texts are rated individually or by pairs.

heading: 2.2 Evaluation of open-ended generation
text: Although the prevailing evaluations focus primarily on the quality of generated texts, some studies aim to characterize the diversity of these outputs. An approach involves computing the proportion of distinct N-grams generated, as demonstrated in (Li et al., 2016). Another notable proposition by (Zhu et al., 2018) introduces Self-BLEU which computes the BLEU score of each generated text against all other generated texts as a reference. However, N-gram matching cannot capture semantic features. For instance, a model generating random words achieves a perfect Self-BLEU score.

heading: 2.3 Quality/diversity tradeoff
text: The tradeoff between quality and diversity has been studied by Go et al. (2023) and Kirk et al. (2024) specifically during the human feedback alignment phase of LLMs. They draw their conclusions through traditional metrics and specifically through the summarization task. By introducing new metrics for such evaluation and focusing on extensive open-ended tasks, our methods bring new tools inspired by past research in image generation and backed by a solid theoretical framework. Our findings extend throughout the human feedback alignment phase to include both instruction tuned and pre-trained, highlighted significant differences between those types of models.
In the current landscape, most of the existing methods prioritize quality or rely on a simplistic characterization of diversity. Nevertheless, the recent introduction of MAUVE (Pillutla et al., 2021) makes a significant shift toward diversity. Drawing inspiration from the domain of image generation, this new metric compares generated texts to a reference distribution without requiring aligned texts. MAUVE consists of a divergence-based metric that captures certain properties of text and exhibits correlation with human judgment. Subsequent research by Pimentel et al. (2023) confirms the validity of this approach by leveraging common divergence measures. MAUVE and subsequent works propose a simple metric that allows us to summarize the evaluation of both quality and diversity in a single measure. Our work builds upon these efforts; we aim to demonstrate that using two distinct measures enables us to distinguish between lack of quality and lack of diversity. This distinc-
tion becomes necessary to enable a deeper and more precise understanding of generative models, as illustrated by the simple example of Figure 2.

heading: 3 Background on Precision and Recall
text: Nowadays generative models for images are capable of producing ultra-realistic samples, making classical metrics such as Fréchet Inception Distance (FID) (Heusel et al., 2017) too coarse to assess the full distributional properties of the models. This jump of performance has motivated the definition of precision and diversity for generative models. This need for a finer evaluation is exemplified on Figure 3. Q1 represents a model that produces realistic images but lacks diversity in the generated people’s appearance. Conversely, Q2 depicts a model generating a broad range of people with lower quality. Capturing this significant tradeoff is essential for comprehending generative model behavior. Inspired by the Precision and Recall metrics of binary classifications, the pioneering work of Sajjadi et al. (2018) proposed to redefine these two terms to assess the quality and diversity
of generative models. These metrics are based on a discretized estimation of the reference and model’s distribution. Kynkäänniemi et al. (2019) pointed out some limitations of those metrics and proposed a new way to compute Precision and Recall, leveraging more robust support estimation. This method is now widely adopted in the literature and is typically used along with the FID. Still, these metrics have some limitations, like the sensibility to outliers (Naeem et al., 2020). This motivates further ongoing work on this topic, including theoretical contributions (Simon et al., 2019; Naeem et al., 2020; Alaa et al., 2022; Cheema and Urner, 2023; Verine et al., 2023) or practical methods (Kim et al., 2023).
Precision and Recall definition. Given its importance in the field of image generation, we adopt the definition of Precision and Recall introduced by Kynkäänniemi et al. (2019). Alternative definitions and approaches are discussed in Appendix A.1.
Definition 1. Let P and Q be two distributions over a space X ∈ Rd. We denote Supp(P ) and Supp(Q) their support. The Precision and the Recall of Q with respect to P are defined as:
Precision = Q(Supp(P )) (1a)
Recall = P (Supp(Q)). (1b)
Precision and Recall in practice. Since Precision and Recall involve an estimation of the support, standard pipelines involve a projection of the samples into an latent space, using pre-trained models such as Inception-v3 (Szegedy et al., 2015) or VGG (Simonyan and Zisserman, 2015). Supports of the distributions are then estimated using a knearest neighbors algorithm. We present this algorithm in more detail in Section 4.
Quality and diversity in NLP. Although using metrics characterizing quality and diversity at a
distribution level is well-established in image generation (Dhariwal and Nichol, 2021; Sauer et al., 2022; Song et al., 2023), it is a very recent practice in the field of text generation. Pillutla et al. (2021) led the way in this field by introducing MAUVE, a pioneering metric that builds on the theoretical concepts of quality and diversity, as initially proposed by Djolonga et al. (2020), and detailed in Appendix A.1.
The authors showcased the intriguing characteristics of their metrics across various GPT-2 models and decoding algorithms. This metric is highly effective in evaluating the degree to which the generated distribution aligns with the reference. The methodology behind it was subsequently validated and simplified by Pimentel et al. (2023) and further evaluated with alternative frameworks by Pillutla et al. (2023).
Despite the progress made, these approaches present a challenge: interpreting them as a measure of quality and diversity is not straightforward, as they condense these two concepts into a single divergence measure. Characterizing Precision and Recall independently has only been mentioned by Pillutla et al. (2023) who conducted some toys experiments on small models or decoding algorithms.
Therefore, we believe that Precision and Recall for text generation are in its early stages. We argue that a deeper understanding of these metrics and their practical implications is needed.

heading: 4 Precision and Recall to assess text generation
text: In this paper, we introduce a framework for calculating Precision and Recall between distributions on texts. These distributions characterize a dataset, or a model, and its generative capability. The extended version of Precision and Recall builds upon the notion of support of distributions, which is in our case challenging to define and estimate.

heading: 4.1 Pipeline to compute Precision and Recall
text: We propose the following pipeline to compute the Precision and Recall metrics, as shown in Figure 5:
1. Sampling. Select N sequences randomly in the dataset to build the set of references: X ref ={ xref1 , . . . ,x ref N } . Sample N sequences from
the distribution Q to assess and build the set of outputs: X out = { xout1 , . . . ,x out N } .

heading: 2. Samples pre-processing.
text: (a) Compute the latent representations of each set using the embedding function ϕ:
X refϕ = { ϕ(xref1 ), . . . , ϕ(x ref N ) } X outϕ = { ϕ(xout1 ), . . . , ϕ(x out N ) }
(b) Perform Principal Component Analysis (PCA) on the union of sets X refϕ and X outϕ to reduce the dimensionality of the data by retaining 90% of the variance. This process yields the sets Zrefϕ and Zoutϕ , which consist of points, respectively, distributed on P̂N and Q̂N , the empirical distributions in the latent space.

heading: 3. Support estimation.
text: (a) For each set Zϕ ∈ { Zrefϕ ,Zoutϕ } , for each
point zi ∈ Zϕ computes the pairwise distances. Define Bk(zi,Zϕ) as the ball centered on zi with radius being the distance to the k-th nearest neighbor in Zϕ.
(b) Each support is defined as the union of balls Bk(zi,Zϕ) for all zi in Zϕ:
Suppk(P̂N ) = ⋃
zi∈Zrefϕ Bk(zi,Zrefϕ )
Suppk(Q̂N ) = ⋃
zi∈Zoutϕ Bk(zi,Zoutϕ )

heading: 4. Precision and Recall computation.
text: Precision = 1
N
∑
zi∈Zoutϕ
1 zi∈Suppk(P̂N ) (2a)
Recall = 1
N
∑
zi∈Zrefϕ
1 zi∈Suppk(Q̂N ) (2b)
The parameters are detailed in the next section.

heading: 4.2 Precision and Recall for text generation.
text: The embeddings used for the MAUVE metric have proven to be effective at capturing the properties of the text at the word level (Pimentel et al., 2023) and content level (Pillutla et al., 2021). Building on this success, we adopt the same embedding function, ϕ, which leverages the output of the last layer of GPT-2LARGE, coupled with a PCA. The PCA serves a dual purpose: it reduces data dimensionality and also filters out noise, which is particularly beneficial for k-NN based manifold estimation. As presented in Appendix B.1, the use of other models for the embeddings gives consistent results.
To choose the other parameters – the number of samples N and the neighbor parameter k – we investigate how Precision and Recall are affected by their variations. We use the WebText generation task with LLAMA-2 7B Chat (comprehensively described in Section 5.3) as a test case, prompting the model to generate between 100 and 10,000 samples. The findings are depicted in Figure 6. Additional details can be found in Appendix B.4.
Our observations are as follows: (1) Precision and Recall values stabilize with an increasing N , plateauing at approximately N = 3000. (2) As
k increases, Precision and Recall values gradually approach 1, aligning with the expectation that a larger k leads to a wider estimated support. In the rest of the paper, we set N to 4000 samples and k to 4.

heading: 5 Practical use cases for LLMs
text: Equiped with our freshly introduced Precision and Recall we now present an analysis of the quality of generation of a broad range of LLMs. We focus on three different tasks to illustrate the versatility of our metrics and the new insights they can provide.

heading: 5.1 Experimental settings
text: For the models, we stress the distinction between pre-trained models and instruction-tuned (or chat) models. The former are models that have been trained on a large corpus of text, and the latter have been fine-tuned with instructions and optionally aligned with human feedback. More details on the experimental settings are provided in Appendix B.
Models. We considered the following recent LLMs: • LLAMA-2 {7, 13, 70}B and their counterparts
fined-tuned and aligned with human-feedback (Touvron et al., 2023).
• VICUNA (Zheng et al., 2023), derived from LLAMA-2 7B model trained on a large opensource instruction dataset.
• MISTRAL 7B and its instruction-tuned version MISTRAL 7B Instruct (Jiang et al., 2023).
• PYTHIA {6.9,12}B (Biderman et al., 2023).
Text generation. For all tasks, we use the model to generate 4000 samples. For WebText and Creative writings, we additionally use five different seeds for the random generator to account for the stochasticity of the generation process. We then compute the average Precision and Recall over the the different seeds.
WebText generation. We run generations of a wide suits of LLMs, of sizes ranging from 7B parameters to 70B parameters, with both pre-trained and instruction-tuned models. In this setting, we make models generate texts close to their pretraining data distribution. Models are prompted with a minimal instruction of 10 words extracted from the WebText train set, and we evaluate their ability to continue the prompt. To account for the different tokenizations, input and generation
lengths are constrained in terms of words, rather than in terms of tokens.
Biographies generation. We evaluate the abilities of LLMs to generate Wikipedia-like biographies, given some in-context examples of biographies. The benchmark for comparison consists of summary sections from Wikipedia pages of individuals who have been distinguished with either "Good Article" or "Featured Article" accolades. The models receive a prompt that includes an instruction followed by a varying number of in-context biography examples. Our evaluation focuses on analyzing how the quality and diversity of the generated biographies evolves as we increase the number of in-context examples provided.
Creative texts generation. We task the models with generating creative texts based on a set of 50 manually-crafted creative instructions such as Write about a dream you had or Write a script for a short film. We leverage the stochastic decoding procedure of models to generate a large number of outputs based on these 50 prompts. The specificity of this task is that there is no reference dataset available.

heading: 5.2 Preliminary results
text: First, we present some preliminary results to illustrate the potential of our metrics.
AG news topics modeling. We consider a reference distribution P made up of journal articles on two topics (World, Sport), extracted from the AG news dataset (Zhang et al., 2015). We then simulate three models. Q1 corresponds to a model that generates texts only from the World topic. Q2 generates texts from the two reference topics. Q3 generates text on divergent topics compared to P .
When considering P as a reference, Q1 therefore suffers from a lack of diversity and Q3 from a lack of quality. The experiment is illustrated in Figure 2. Precision and Recall metrics allow us to clearly distinguish between these cases, while, for instance, MAUVE would give the same score for both, demonstrating that our metrics can capture the trade-off between quality and diversity on a content-based level.
Correlation with other metrics. Figure 7a shows the correlation between Precision and Recall and other distribution-based metrics on the Wikipedia Biography dataset in our experiments. We observe that: 1. Recall expectedly reflects word-level diversity,
as indicated by its correlation with Self-BLEU and Distinct-4.
2. Precision shows no correlation with these metrics, but does correlate with MAUVE.
This suggests that Precision captures the adequacy of distributions, but not diversity, essentially indicating the quality of the generated texts. A complementary analysis of MAUVE on the WebText dataset is available in Appendix B.4.

heading: 5.3 Evaluation of open-ended generation
text: Instruction-tuned models are more precise and less diverse than pre-trained models. Figure 1 shows the Precision / Recall graph of various models. Instruction tuned models have a higher Precision and a lower Recall than their pre-trained counterparts. This is consistent with the expected effect of human preference alignment and instruction tuning: models are encouraged to generate more "human-like" texts, which do not encompass any notion of diversity.
Larger models are more diverse. The Recall is consistently better for larger models. Larger models have a better expressive power, and this diversity is reflected by their higher Recall.

heading: 5.4 Generating Biographies.
text: The results are presented in Figures 8a and 8b. We plot the Precision and the Recall as a function of the number of in-context examples.
The number of in-context examples makes the model more precise. For both pre-trained and instruction-tuned models, the Precision increases with the number of in-context examples. This is consistent with the intuition that several in-context examples help the model generate texts that are closer to the expected distribution.
Increasing the number of in-context examples make "chat" models more diverse. Figure 8b confirms our previous findings that instructiontuned models are less diverse than their pre-trained counterparts. However, by increasing the number of in-context examples, we observe a strong regain in diversity: chat models leverage the diversity in the prompt to generate more diverse texts.
Chat models plateau. After a few numbers of in-context examples, we observe that both the Pre-
cision and the Recall plateau. This suggests that the model has captured the distribution of the reference datasets after a few examples and that adding more examples does not bring additional information.

heading: 5.5 Creative text generation
text: For this task, no reference dataset is available. Still, our novel Precision and Recall metrics offer a means to compare models, where conventional methods fall short.
Estimation of overlaps of the distribution of models. Since our metrics Precision and Recall corresponds to different support overlap, they can be used to analyze models comparatively.
Models of the same family share close distributions but size induces some differences. Table 1a shows that compared to the reference LLAMA-2 70B Chat, Precision and Recall of LLAMA-2 7B Chat and LLAMA-2 13B Chat increases with their size.
Intepreting distributions overlap. Compared to LLAMA-2 70B Chat, MISTRAL 7B Instruct has an almost perfect Precision but suffers from a lower Recall. Geometrically, this means that its distribution fits almost perfectly in the support of LLAMA2 70B Chat, but does not cover it totally. Table 1b
illustrates even further this phenomena: compared to MISTRAL 7B Instruct, every model has a quite low precision but a high recall. Geometrically, the LLAMA-2-based models cover a wide portion of the support of MISTRAL 7B Instruct, but also assigns mass to points outside this distribution.

heading: 5.6 Correlation with human-like preferences
text: Finally, we evaluated how our proposed metrics relate to human evaluation. Since one of our evaluation involve significantly long inputs, we conducted a mock-human evaluation using GPT-3.5-turbo. To evaluate the quality, we ask GPT-3.5-turbo to rate our generated biographies on three dimensions: coherence, insightfulness and Wikipedia-likeness. We also experimented with a proxy of diversity, by asking GPT-3.5-turbo to rate the similarity of batches of 10 texts (i.e., low score if there are many similar texts). More details regarding the evaluation setup are given in Appendix B.2.
Precision Correlates with GPT-3.5 quality evaluation. Models with high precision generate texts that align closely with the reference distribution, which consists of high-quality Wikipedia articles. Consequently, higher precision typically corresponds to better GPT-3.5 scores, as illustrated in Figure 8. Note that instruction-tuned models, which are tuned similarly to GPT-3.5, were excluded from the correlation analysis with GPT-3.5 quality scores due to their uniformly high scores, relatively to pre-trained models.
Recall correlates with human-like diversity evaluation. As depicted on Figure 8, we found a positive correlation between our introduced Recall and human-like evaluation of diversity. This confirms our findings regarding the generated contents of LLMs.

heading: 6 Conclusion
text: We introduced Precision and Recall as automated metrics for comparing generative model outputs against a reference distribution. These metrics serve as independent indicators of the quality and diversity of the generated text. In challenging scenarios where evaluation protocols are scarce, such as tasks with high generation complexity, we demonstrate the efficacy of our metrics in assessing the quality and diversity of model output. Furthermore, our metrics revealed nuanced model behaviors that lie beyond the scope of traditional evaluation benchmarks. We believe that our work contributes to advancing the evaluation of generative models, offering valuable insights to better understand their capabilities, and aiding in the development of more refined evaluation protocols within the community.

heading: 7 Limitations
text: Our study proposes to adapt the framework of Kynkäänniemi et al. (2019) from image to text generation. However, limitations of the original image-based framework, as highlighted by Naeem et al. (2020) and Kim et al. (2023), may also apply to our adaptation to NLP. In particular, the sensitivity to outliers and the generalizability of these metrics across different domains warrant further investigation.
The metrics proposed in our study are based on an auxiliary model, specifically GPT-2LARGE embeddings, to rate the quality of text generations. Although the metrics tend to be correlated across models as shown in Appendix B.1, the choice of embeddings may still significantly impact the performance and applicability of these metrics. Future research could explore the effects of alternative
embedding models on the Precision and Recall measurements in text generation tasks.
We present these metrics as valuable tools for evaluating a model’s open-ended generation capabilities. However, to comprehensively assess LLMs across specific dimensions and tasks, these metrics should be supplemented by other approaches. In particular, the relationship with traditional human evaluation needs to be investigated more thoroughly in future research. Although we performed a mock human evaluation using GPT-3.5-turbo, this is not a complete assessment of how these metrics compare to the human evaluation.
We restricted our analysis to English language on most popular LLMs.

heading: 8 Ethical considerations
text: Precision and Recall are metrics that rely heavily on the distribution of the data they are tested against. If the data contain biases, these metrics may favor models that perpetuate those biases. This issue is crucial because it can lead to unfair outcomes, particularly for underrepresented groups, emphasizing the need for a well-designed reference dataset.
To address this problem, we propose a practical solution: calculate Precision and Recall for specific groups within the data, such as different genders or ethnic minorities. This approach helps to ensure that the model is equally accurate and fair between all groups. Our objective is to prevent models from overlooking or misrepresenting any group. Further investigations should focus on this consideration.

heading: 9 Acknowledgements
text: This work has been partly funded through project ACDC ANR-21-CE23-0007. This project was provided with computing AI and storage resources by GENCI at IDRIS thanks to the grants 20XX-AD011014022R1, 20XX-AD011014053 and 20XX-A0151014627 on the supercomputer Jean Zay’s V100/A100 partition.

heading: A Definition of Precision-Recall Curves
text: In this section, we introduce a mathematical tool called the PR-Curve, which is used to evaluate generative models. PR-Curves are closely related to the MAUVE score. We will then list the main differences between PR-Curves and the MAUVE score to highlight the limitations of the latter. Following this, we will demonstrate with a straightforward example that the MAUVE score struggles to distinguish between a lack of diversity and a lack of quality. This difficulty can be explained by some of the distinguishing points between PR-Curves and the MAUVE score.
A.1 Definition
Recognizing the need to distinguish between quality and diversity, and drawing a parallel with the concepts of precision and recall from classification, Sajjadi et al. (2018) introduced a new definition of Precision and Recall for discrete distributions.
Definition 2 (Precision-Recall trade-off (Sajjadi et al., 2018)). A distribution Q has a precision α and a recall β with respect to a distribution P if there exists distributions µ, νP and νQ such that:
P = αµ+ (1− α)νP (3a) Q = βµ+ (1− β)νQ. (3b)
This definition is illustrated in Figure A.1. The distribution νP represents the part of P that cannot be generated by µ and thus Q, and νQ represents the part of Q that should not be generated by µ. The set of all attainable couples (α, β) such that there is a distribution µ from Equation (3) defines various trade-offs. The Precision Recall Curve can then be defined as the set of all Pareto-optimal trade-offs. More recently, Simon et al. (2019) has extended the definition for any continuous distribution P and Q:
Theorem 1 (PR-Curve (Simon et al., 2019)). The set of Pareto-optimal trade-offs, that is, the PR-Curve, can be represented as the set of points (αλ, βλ){λ∈[0,∞]} ∈ [0, 1]2 such that:
   αλ = EQ [ min ( 1, λp(x)q(x) )] ,
βλ = EP [ min ( 1, 1λ q(x) p(x) )] .
Intuitively, for any given trade-off parameter λ ∈ [0,+∞], the precision αλ diminishes when q(x) is smaller than λp(x) at certain points x, indicating overestimation of regions by the distribution Q. On the contrary, the recall βλ declines when p(x) is smaller than λq(x) for specific points x, suggesting underestimation of regions by Q. More concretely, an example of PR-Curves is depicted in Figure A.2. Figure A.2a shows samples drawn from the reference distribution P . Here, Q1 represents a distribution capable of generating high-quality images, albeit only for the first 5 digits, while Q2 represents a distribution that generates noisy images on all labels. The PR-Curve of Q1 surpasses that of Q2 for high values of λ (top left), indicating the superior quality of Q1. In contrast, for low values of λ (lower right), the PR-Curve of Q2 exceeds that of Q1, indicating greater diversity in Q2.
For a two-number summary of these curves, Sajjadi et al. (2018) has proposed the Fγ-scores: Fγ = maxλ(1 + γ) 2αλβλ/(γ 2αλ + βλ). The F1/8 is employed to assess the quality of the model, and F8 score is used to assess diversity. Abusing the notation, the scores F1/8 and F8 are often referred as Precision and Recall. In practice, the PR-Curves are estimated by sampling from the distributions P and Q, and by performing density estimation on the embeddings on the samples. There are several ways to estimate the PR-Curves, including a k-means estimator (Sajjadi et al., 2018) or a discriminator method (Simon et al., 2019).
Connection with PR-Curves and Precision/Recall: In this paper, we have used another two-number summary of the PR-Curve introduced by Kynkäänniemi et al. (2019): the maximum precision α∞ and the maximum recall β0. As a matter of fact, one can show that:
α∞ = max λ αλ = Q(Supp(P )) and β0 = max λ βλ = P (Supp(Q)).
These two values encapsulate the definition of support provided by Kynkäänniemi et al. (2019). The first value, β0 = P (Supp(Q)), measures the proportion of P covered by the support of Q. The second value, α∞ = Q(Supp(P )), measures the proportion of Q covered by the support of P . The primary practical advantage is that these two values can be computed without performing a density estimation. As detailed in Section 3, we use a k-nearest neighbors estimator to estimate these values.
A.2 MAUVE score MAUVE score definition. The MAUVE score (Pillutla et al., 2021) is a measure of discrepancy between two distributions P and Q in the embedding space. The MAUVE score is defined as the area under the divergence frontier between P and Q. The divergence frontier is defined as the set of points (απ, βπ) for π ∈ [0, 1]:
{ απ = exp (−DKL(Q∥πP + (1− π)Q)) βπ = exp (−DKL(P∥πP + (1− π)Q)) .
In other words, the MAUVE score is a one-number summary of another quality-diversity curve.
Connection between MAUVE and PR-Curves. The MAUVE score is based on a curve different from PR-Curves, and there is no straightforward connection between the two. But, from a practical point of view, both curves are estimated using density estimation on the embeddings of samples from P and Q. Pillutla et al. (2021) used a k-means estimator to estimate the divergence frontier and then computed the area under the curve to obtain the MAUVE score.
Since both curves are based on density estimation, we can compare them using the same dataset, embeddings, and density estimation method. This allows us to highlight the limitations of the divergence frontier used in the MAUVE score. In our AG News experiment, fully described in Section 5.2, we exemplify these limitations. Specifically, Figure A.3 shows that the underlying curves used for computing the MAUVE score struggle to differentiate the discrepancies highlighted in Figure 2, while the original PR-Curves from Sajjadi et al. (2018) can.
When adapting the F1/8 and F8 scores to the MAUVE curves and plotting the results in Figure A.4, we find that these adapted scores are substantially less meaningful than those from Sajjadi et al. (2018). Furthermore, the definitions of Precision and Recall we used, based on support-based metrics (Kynkäänniemi et al., 2019), appear to be more informative than the F1/8 and F8 scores.
Difference between Precision/Recall and MAUVE. As detailed in the previous paragraph, the MAUVE score is a single-number summary of a divergence frontier, whereas the PR-Curves provide a two-number summary of the precision-recall curve. Consequently, in addition to the fact that the divergence frontier is less capable of distinguishing between quality and diversity, MAUVE, being a single metric, is inherently less informative than Precision and Recall. In Figure A.5, we examine the correlations between Precision, Recall, the MAUVE score, and the F1/8 and F8 scores on Wikipedia biographies, as detailed in Section 5.3. We observe that the F1/8 and F8 scores correlate more with Precision than with Recall and strongly correlate with each other. This strongly indicates that the MAUVE score cannot effectively differentiate between the quality and diversity of models.
To conclude, MAUVE and our metrics are all distribution-based metrics that rely on discretized approximations of both the target distribution P and the model’s distribution Q, computed using embedding functions. However, we can identify three reasons why MAUVE differs from the metrics introduced in this paper:
• MAUVE is a one-number summary of a divergence frontier, whereas Precision and Recall provide a two-number summary of the PR-curve.
• MAUVE is based on the divergence frontier, which has been experimentally shown to be less informative than the PR-Curves.
• MAUVE uses the divergence frontier estimated with k-means, while Precision and Recall are estimated with a k-nearest neighbors estimator.

heading: B Experiments
text: B.1 Other embedding functions
We evaluated the biographies generated using different embedding models. We computed the correlation between the embeddings GPT-2LARGE and the other embeddings and reported the results in Table B.1. All Precision and Recall scores are positively correlated, demonstrating the robustness of our method. Notably, stronger correlations are observed between the decoder-only models, likely due to their pretraining objectives, which make them well-suited for evaluating open-ended text generation.
B.2 GPT-3.5-turbo evaluation
Here we describe the complete evaluation setup we used for GP-T3.5-turbo evaluation.
Evaluation dimensions and instructions. Below is a description of the evaluation dimensions we considered for both the three quality dimensions and diversity.
• Coherence: GPT-3.5-turbo was asked to rate the ease of reading and understanding the text on a scale from 1 (difficult to read) to 5 (very easy to read).
• Insightfulness: The insightful or interesting nature of the biography was rated on a scale from 1 (not interesting) to 5 (highly interesting).
• Plausibility: This dimension assessed how likely the text is to be a biography from Wikipedia, with ratings from 1 (not likely) to 5 (very likely).
• Diversity: This dimension evaluated the variation among biographies, assessing if they differ in terms of names, professions, etc., with a score of 0 indicating similarity and 1 indicating diversity. This is evaluated on batches of 10 texts.
Evaluation process. Below are the instruction and prompt given to GPT-3.5-turbo.
• For assessing quality:
– Instruction: "You will be asked to rate a Wikipedia biography based on coherence, insightfulness, and likelihood on a scale from 1 to 5. Please provide three numerical answers using the template in JSON ("score_coherence": score, "score_insightful": score, "score_likely": score)."
– Prompt: "On a scale of 1 to 5, rate the following biography on three criteria: * Coherence: How easy is it to read and understand the text? (1 for difficult to read, 5 for
very easy to read)
* Insightfulness: How insightful or interesting is the biography? (1 for not interesting, 5 for highly interesting) * Probability: How likely is this text a biography from Wikipedia? (1 for not likely, 5 for very likely)
Biography Text: {text} Scores:"
• For assessing diversity:
– Instruction: "You will be asked to rate the similarity between several biographies. Give a score of 0 if some biographies are similar in terms of names, profession, etc., and give a score of 1 if they are all different. Provide the answer in JSON ("similarity_score": score)."
– Prompt: "Rate the similarity of the following texts: {concatenated_texts}
Similarity Score:"
B.3 Implementation details This section provides full details on the experimental protocol adopted in this paper.
Texts input length. All tested models do not share the same tokenization. To avoid a potential bias due to different generations’ lengths, input and output length constraints are expressed in terms of words rather than tokens. The average number of tokens per word has been calculated for each model on the train set of the WebText dataset3 and reported in Table B.2.
Hardware and software setup. Generations were conducted on either A100 GPUs with 80Gb memory or on V100 with 32Gb memory. Models were run with a bfloat16 precision on A100 and with float32 precisions on V100, except for the PYTHIA models, which have been ran with float16 mixed precision, following the original setup (Biderman et al., 2023). For models with ≈ 7B parameters, generating 4000 samples takes approximately 5 hours on 2 A100s, it takes about 10 hours for models with ≈ 13B parameters on 2 A100, and about 40 hours for models with ≈ 70B parameters on 4 A100. Generating the samples required approximately 4200 GPUs hours. Code is built on PyTorch (Paszke et al., 2019), with the HuggingFace library (Wolf et al., 2020).
Computational cost of Precision and Recall. Since Precision and Recall are based on a k-NN computations, computing these metrics is especially fast, even across large datasets. Once the features are computed, computing Precision and Recall takes less than 5 seconds on a standard laptop for 4000 samples, instead computing MAUVE takes approximately 40s.
3Available at https://github.com/openai/gpt-2-output-dataset
Other metrics implementations. For computing SelfBLEU, Disctinct-N and MAUVE we relied on the official implementations found in https://github.com/koadman/mauve, or https://github.com/ krishnap25/mauve-experiments.
Implementations. The code used to run and evaluate the generations, as well as the exact datasets are available at https://github.com/AlexVerine/pr-4-llm.
B.4 WebText Generation Data. Reference dataset is extracted from the official OpenAI WebText dataset at https://github. com/openai/gpt-2-output-dataset, under a MIT license and is composed of articles extracted from WEB urls. We extracted 15k samples from this datasets for our experiments.
Input formatting. Non-chat models are simply prompted with the first 10 words of random WebText articles. For chat models, they are explicitly asked to continue the prompt. We input the last 10th word out of the instruction tokens to avoid idiomatic expressions such as Sure! Here is . . . and ensure fair comparison between models. The different prompts are illustrated in Table B.3.
Generation setup. For this task, we used the default recommended generation setup, i.e. nucleus sampling for all models, with parameters displayed in Table B.4. LLAMA-2 default parameters are from the official repository https://github.com/facebookresearch/llama/blob/main/example_text_ completion.py and MISTRAL’s ones from their official API https://docs.mistral.ai/api/. For PYTHIA’s models, we did not find any mention of recommenced generation parameters and hence used a temperature and nucleus p of 1.0.
2000 4000 6000 8000 10000
Number of samples
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
M et
ri cs
Precision Llama-2 7B Llama-2 7B Chat
Recall Llama-2 7B Llama-2 7B chat
(a) Evolution of the metrics Precision and Recall as the number of samples N increases. The shaded area represents the standard deviation computed over 5 random seeds for the set ouputs. The averages reach a plateau around N = 3000.
5 10 15 20 25 30
Number of k nearest neighbors
0.0
0.2
0.4
0.6
0.8
1.0
M et
ri cs
Precision Llama-2-7b Llama-2-7b-chat
Recall Llama-2-7b Llama-2-7b-chat
(b) Evolution of the metrics Precision and Recall as the parameter k increases. The shaded area represents the standard deviation computed over 5 random seeds for the set ouputs.
Figure B.1: Evolution of the metrics Precision and Recall as the number of samples N and the parameter k increases.
Evaluation setup. To determine the optimal number of samples and the parameter k for estimating Precision and Recall, we analyze the behavior of these metrics with varying N and k. We employ the WebText generation task with LLAMA-2 7B Chat as a reference. Specifically, we generate 10, 000 samples per seed and compute the average Precision and Recall for N ranging from 100 to 10, 000, and k ranging from 1 to 30. The trends of Precision and Recall with respect to the number of samples N , while k is fixed at 4, are depicted in Figure B.1a. Conversely, the trends of Precision and Recall with respect to the parameter k, while N is fixed at 4, 000, are shown in Figure B.1b. We observe that Precision and Recall stabilize as N increases, reaching a plateau around N = 3, 000. Thus, we set the number of samples at N = 4, 000 for subsequent experiments. Additionally, Precision and Recall increase towards 1 as k increases, aligning with the intuition that the estimated support is more covering as k increases. Therefore, we set k = 4 as it facilitates a clear differentiation between the two models. We also include the standard deviation for all models described in Table B.4. Standard deviations are computed by varying the output set, the target set, and both. These results are summarized in Table B.5. We recommend consistent references for all models to ensure a fair comparison, as the standard deviation can significantly impact the results.
Additional results. We report MAUVE scores for WebText on Figure B.2. Contrary to what is reflected by Precision and Recall, pre-trained and instruction-tuned models exhibit similar MAUVE scores.
B.5 Wikipedia Biographies Generation
Data. We took inspiration from the WikiBio dataset (Lebret et al., 2016) and extracted Wikipedia articles corresponding to humans. We kept only the articles with a "Good" or "Featured" badge, which are considered as the best quality articles. We then extracted the summaries of these pages to use as reference biographies. Finally, we kept only articles that have at least 80 words and at most 350 words, for a total of 6637 texts. Our processed dataset and the code use for its construction is available in the supplementary materials.
Input formatting. Pre-trained and chat models are prompted with a small instruction and a variable number of in-context examples. The different prompts are illustrated in Table B.6.
Generation setup. We use the default model generation parameters, as described in Appendix B.4. However, we set the maximum number of tokens generated for all models to 448 for LLAMA-2-based models (including VICUNA), 432 for MISTRAL and 390 for PYTHIA, respecting the average number of tokens per word reported in Table B.2.
B.6 Creative texts generation
Data. The full list of creative instructions is presented in Listing 1.
Listing 1: List of Creative Prompts
Wri te a b o u t a dream you had . C r e a t e and w r i t e a b o u t a new c h a r a c t e r . Wr i t e a b o u t a p l a c e you ’ d l o v e t o v i s i t . Wr i t e a b o u t an i m p o r t a n t l i f e e v e n t . Wr i t e a b o u t l i f e 100 y e a r s from now . Wr i t e a s t o r y where magic e x i s t s i n e v e r y d a y l i f e . Wr i t e a poem a b o u t a p e r s o n a l e x p e r i e n c e . Wr i t e a s pe ec h f o r a c a u s e you b e l i e v e i n . Wr i t e a s h o r t mys t e ry s t o r y . Wr i t e a modern day f a i r y t a l e . Wr i t e a s t o r y s e t i n a h i s t o r i c a l p e r i o d . Wr i t e a s t o r y a b o u t a t e c h n o l o g i c a l advancement . Wr i t e a l e t t e r t o your f u t u r e s e l f . Wr i t e a s t o r y from an animal ’ s p e r s p e c t i v e . Wr i t e a week ’ s wor th o f d i a r y e n t r i e s f o r a c h a r a c t e r . Wr i t e a s h o r t s t o r y u s i n g m y t h o l o g i c a l c h a r a c t e r s . Wr i t e a c o n v e r s a t i o n between two c h a r a c t e r s . Wr i t e a b o u t a day i n your l i f e . Wr i t e a s e r i e s o f Haikus a b o u t s e a s o n s . D e s c r i b e a p l a c e w i t h o u t naming i t . Wr i t e a news a r t i c l e a b o u t an e v e n t i n your town . Wr i t e a r e c i p e wi th a s t o r y . Wr i t e a one − a c t p l a y . Wr i t e a s t o r y s e t i n a d y s t o p i a n f u t u r e . Wr i t e a s a t i r i c a l e s s a y on a t r e n d i n g t o p i c . Wr i t e a humorous comic s t r i p . Wr i t e a p i e c e i n s p i r e d by n a t u r e . Wr i t e a s h o r t s t o r y a b o u t your f a v o r i t e f i c t i o n a l c h a r a c t e r . Wr i t e i n s t r u c t i o n s f o r an i n v e n t e d machine . Wr i t e a s t o r y u s i n g on ly me tapho r s . Wr i t e a s c r i p t f o r a s h o r t f i l m . Wr i t e a c o n v e r s a t i o n i n t e x t message f o r m a t . Wr i t e a l i m e r i c k a b o u t a funny e v e n t . Wr i t e a s t o r y a b o u t a j o u r n e y t o s p a c e . Wr i t e a s c r i p t f o r a documenta ry on a s u b j e c t o f your c h o i c e . Wr i t e a eu lo gy f o r a c h a r a c t e r from your f a v o r i t e book . Wr i t e a s c r i p t f o r a r a d i o show . Wr i t e a song a b o u t a memorable e v e n t . Wr i t e a s t o r y wi th a h i dd en meaning . Wr i t e a f a b l e wi th a mora l . Wr i t e a comedic monologue . Wr i t e an o p i n i o n p i e c e on a c u r r e n t e v e n t . Wr i t e a parody of a p o p u l a r book or movie . Wr i t e a s t o r y where a c h a r a c t e r has a m a g i c a l power . Wr i t e a s t o r y a b o u t t ime t r a v e l . Wr i t e a s t o r y based on an o b j e c t i n a mys te ry box . Wr i t e a g h o s t s t o r y . D e s c r i b e a h a u n t e d house w i t h o u t u s i n g c l i c h e d e s c r i p t i o n s . Wr i t e a s t o r y a b o u t a c o n s p i r a c y t h e o r y . Wr i t e a s t o r y a b o u t a non − t r a d i t i o n a l s u p e r h e r o .

heading: C License
text: Models weights. Llama-2 weights are released under the licence available at https://ai.meta.com/ llama/license/. Mistral models and weights are released an Apache 2.0 licence. Vicuna models and weights are released under an Apache 2.0 licence. Pythia models and weights are released under an Apache 2.0 licence.
Data. The WebText dataset we used is released under a MIT License. Wikipedia’s articles are under the reative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA) and the GNU Free Documentation License (GFDL). AG news is available at https://huggingface.co/datasets/ag_ news but no licence are disclosed.
D Wikipedia Biographies samples

heading: Future Work
text: Future research could explore the effects of alternative
embedding models on the Precision and Recall measurements in text generation tasks. We present these metrics as valuable tools for evaluating a model’s open-ended generation capabilities. However, to comprehensively assess LLMs across specific dimensions and tasks, these metrics should be supplemented by other approaches. In particular, the relationship with traditional human evaluation needs to be investigated more thoroughly in future research. Although we performed a mock human evaluation using GPT-3. 5-turbo, this is not a complete assessment of how these metrics compare to the human evaluation. We restricted our analysis to English language on most popular LLMs