Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models
Authors: Yeming Wen
Swarat Chaudhuri
Venue: NeurIPS 2024 poster (2024)
Keywords: diverse generation
instruction fine-tuning
synthetic dataset
Primary area: generative_models
TL;DR: we proposed a framework that leverages synthetic data, data partitioning, and model adaptation to elicit diverse responses from foundation models.

Abstract: Presenting users with diverse responses from foundation models is crucial for enhancing user experience and accommodating varying preferences. 
However, generating multiple high-quality and diverse responses without sacrificing accuracy remains a challenge, especially when using greedy sampling. 
In this work, we propose a novel framework, Synthesize-Partition-Adapt (SPA), that leverages the abundant synthetic data available in many domains to elicit diverse responses from foundation models.
By leveraging signal provided by data attribution methods such as influence functions, SPA partitions data into subsets, each targeting unique aspects of the data, and trains multiple model adaptations optimized for these subsets.
Experimental results demonstrate the effectiveness of our approach in diversifying foundation model responses while maintaining high quality, showcased through the HumanEval and MBPP tasks in the code generation domain and several tasks in the natural language understanding domain, highlighting its potential to enrich user experience across various applications.

Synthesize, Partition, then Adapt:
Eliciting Diverse Samples from Foundation Models
Yeming Wen'& Swarat Chaudhuri Department of Computer Science The University of Texas Austin
Abstract
Presenting users with diverse responses from foundation models crucial for enhancing user experience and accommodating varying preferences. However, generating multiple high-quality and diverse responses without sacrificing accu- racy remains challenge, especially when using greedy sampling. In this work, we propose novel framework. Synthesize-Partiion Partition-Adapt (SPA), that leverages the abundant synthetic data available in many domains to elicit diverse responses from foundation models. By leveraging signal provided by data attribution meth- ods such as influence function, SPA partitions data into subsets, each targeting unique aspects of the data, and trains multiple model adaptations optimized for these subsets. Experimental results demonstrate the effectiveness of our approach in diversifying foundation model responses while maintaining high quality, show- cased through the HumanEval and MBPP tasks in the code generation domain and several tasks in the natural language understanding domain, highlighting its potential to enrich user experience across various applications.
Introduction
maasformer-based foundation models have revolu- tionized the fields of natural language processing (NLP) and code generation with their remarkable abil- ities a wide range of understanding and generation tasks (Vaswani et al.. 2017; Devlin al., 2019; Brown et al., 2020; Chen et al., 2021). These models are typically pre-trained on vast amounts of text data and then undergo instruction fine-tuning - post-training process - improve alignment with user expecta- tions and enhance the overall user experience (Ouyang al., 2022). Due the high cost of -anntttad data, synthetically generated datasets (Wang al., 2022b) such as OSS-Instruct (Wei al., 2023) and Al- paca (Taori et al., 2023) have become an important component of instruction tuning, demonstrating strong effectiveness in improving foundation model performance.
Prompt
Give me personal wehsite template
Response
Response
Figure 1: A user is expecting two diverse templates from the foundation model.
To date, these synthetic datasets have been primarily used to align foundation models with instruc- tions or to induce certain preferable behaviors. In this paper, we focus on different of synthetic data: in improving the diversity of foundation models' outputs. Diversifying the generated responses crucial for accommodating diverse user preferences and enhancing user satisfaction. Consider the scenario illustrated in Fig. 1, where user prompts foundation model with "Give me personal website template". In this case, we would prefer the model to generate two diverse templates while
38th Conference on Neural Information Processing Systems (NcurIPS 2024).
"ysen@utexasedd

---

maintaining good quality, providing users with a variety of styles and layouts. Conventional meth- ods for improving diversity, such as temperature sampling (Ackley al., 1985; Hinton et al., 2015; Wang et al., 2019, 2023), rely on sampling techniques that anneal the probabilistic distribution of outputs. These methods often trade off diversity for quality, as the generated responses may deviate from the learned distribution and produce hallucination or less coherent outputs (Lee, 2023). More- over, these techniques are not applicable when using greedy sampling, which is often preferred for its simplicity and precision. This highlights the need for approaches that not only align foundation model outputs with user expectations but also elicit diverse responses without sacrificing quality.
In this paper, we present framework, Syurhesize-Partitioon Partition-Adaue Adapt (SPA), that achieves these objec- tives. The framework partitions the synthetic data and adapts foundation models these partitions in the post- post-training stage. By leveraging the inherent diversity in the training data, this approach can generate diverse responses without compromising accuracy. The potential of partition-and-an approach is further amplified by the increasing availability of large-scale synthetic datasets because the utility of instruction-tuning a single model on the entire dataset diminishes. In particular, we show that influence function (Koh & Liang, 2017) can be an effective signal to partition synthetic datasets into subsets, each targeting unique aspects that elicit distinct model behaviors. However, SPA is not limited to influence function and can be extended to other partitioning strategies. By training multiple adaptations on these subsets using parameter-efficien fine-tuning techniques, such as LoRA (Hu et al., 2021), enable the generation of diverse and accurate responses.
To demonstrate the effectiveness of our approach, we conduct experiments on a range of tasks in both the code generation and natural language understanding domains. We evaluate our method on the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets for code generation, as well as several natural language understanding tasks. The results showcase the ability of our approach to diversify model responses while maintaining high accuracy, highlighting its potential to enrich user experience across various applications.
To summarize, the main contributions of this paper are as follows:
We propose SPA, a novel framework that leverages synthetic data, data partitioning, and model adaptation to elicit diverse responses from foundation models.
We demonstrate the effectiveness of SPA in diversifying foundation model responses while maintaining sampling quality through extensive experiments on code generation and natural language understanding tasks.
We highlight the potential of SPA to leverage the increasing availability of large-scale synthetic datasets for improving the diversity of foundation model responses.
2 Background
2.1 Instruction Fine-tuning
By fine-tuning foundation models on human-annotated data that demonstrates desired behaviors, instruction tuning aims to improve the alignment between the model's outputs and the user's inten- tions (Ouyang et al., 2022; Wei et al., 2021; Sanh al., 2022). Let D (æ,, wi) denote a dataset of input-output pairs, where I; represents the input instruction and y/4 represents the correspond- ing desired output. The objective of instruction tuning is to minimize the following loss function: (()) = - 1 EN loga (z4)24) where / represents the parameters of the foundation model, and (z.) is the probability of generating the target response y! given the input 24.
i=
Classical approaches for instruction tuning typically require a substantial amount of parallel labeled data of NL intents and gold model responses. Collecting large-scale, high-quality annotated datasets often conseminn and expensive. To mitigate this issue, researchers have explored the use of synthetic data for instruction tuning. By leveraging techniques such as data augmentation (Wei & Zou, 2019; Sennrich et al., 2016) and Pack-translation (Edunov et al., 2018), synthetic data can be generated at scale, providing a cost-effective alternative to human-annotated datasets. Furthermore, synthetic instruction-follovin data can also be generated from the foundation model itself (Wang et al., 2022a; Honovich et al., 2022; Taori et al., 2023; Peng et al., 2023; Wen et al., 2024, inter alia).

---

2.2 Data Attribution and influence function
Data attribution methods aim to quantify the importance or influence of individual training points on a model's 's predictions. One such method is the influence function (Koh & Liang, 2017). Formally, let [()) denote the loss function of the model, where 0 represents the model parameters. The influence of training point 2 on the model's parameters 0 given by I(z) -H 'ee((z, 0). where Hg is the Hessian matrix of the loss function with respect to the model parameters, and VoL(2, 0) is the gradient of the loss function with respect to the model parameters, evaluated at the training point 2. Next, the influence of elevating the weight of 2 on the loss associated with test point Stest is:
[(z, Ztest) = - VaL(erest H) VoL(z,
(1)
It is impossible to calculate the full Hessian H// matrix in deep neural networks. Koh & Liang (2017) developed simple and efficient implementation that requires only oracle access to gradients and fiessian-vector products. This implementation makes feasible to apply influence function to large-scale models. However, the vast parameter space of foundation models presents an even greater challenge, rendering the direct application of influence function impractical. response to this, recent advancements in Grosse et al. (2023) have further refined the methodology, enabling the application of influence function large language models.
3 Problem Formulation
Given a user input x, our goal is to generate a di- verse set of high-quality responses y1. Y2..... YK from a foundation model M. One approach to generating diverse responses is sample from the model multi- ple times using techniques like temperature sampling: y& MMxx;6 8, r), where & 1., K and 9 rep- resents the model parameters and r is the temperature hyperparameter. However, this approach often trades off diversity for quality as studied in Chung et al. (2023). An alternative approach is to train multiple model adaptations M1,,M Mr and sample one response from each adaptation:
Passg] on HumanBual
44
Parcesaage Ditaset Usee
Figure 2: pass@1 1 on HumanEval after fine-tuning on some percentage of OSS- Instruct dataset (Wei et al., 2023) using LORA. The plot demonstrates the dimin- ishing returns observed with increasing amounts of data used for parameter effi- cient fine-tuning.
(2) where 0₂ represents the parameters of the k-th model adaptation. By training each adaptation on a differ- ent subset of the data that captures unique aspects and yields distinct model behaviors, we can generate di- verse responses while maintaining their quality. Moreover, this approach allows us to elicit diverse samples even with greedy sampling, which often preferred for maximum precision.
Yk = (xxx;;x), k == 1,,....
Traditionally, training multiple model adaptations has been considered unfavorable due to the re- peated training process, which can be computationally expensive and time-consuming. However, with the increasing popularity of instruction tuning, it has become common practice to go through a post-training stage using instruction data before deploying the model to users. This post-training stage presents an opportunity to train multiple model adaptations without incurring significant addi- tional costs, making the approach more feasible and practical in real-world scenarios.
As the volume of synthetic data grows, the utility of fine-tuning a single model on the entire dataset diminishes due to the diminishing returns in the post-training stage, as demonstrated in Fig. 2. The pass@1 I accuracy after fine-tuning on the entire synthetic dataset using LoRA is roughly the same as only consuming 15% of the data?. This creates an opportunity to leverage the abundant synthetic data to train multiple model adaptations, each specializing in a specific subset of the data. In this work, we propose the Synthesize, Partition, then Adapt (SPA) framework to address the diverse re- sponse generation problem. SPA leverages existing synthetic datasets, data partitioning techniques, and marameter-efficies the collection of these adaptations, SPA generates diverse and high-quality responses, enhancing the overall user experience.
fine-tuning methods to train multiple model adaptations. By sampling from
"This does not suggest full parameter fine-tuning shares the same diminishing return.
3

---

Language model
Synthetic data
Partition by attribution score
Generate
Data attribution
Instruction tuning
Instruction tuning
Multiple Adaptations
def strlen(string: str): return nue 2 ==
def strien(string: str): return (num & 1)
Sampling
Prompt
def strlen(string: str): return nue 2 == num 2
Generate Python function to check if number is even.
Figure 3: An illustration of the Synthesize, Partition, then Adapt (SPA) framework. SPA partitions synthetic dataset according data attribution scores, which can be obtained using various methods such as influence function or lexical overlap. Multiple foundation model adaptations are then trained on each subset. Sampling from the collection of these model adaptations can present users with diverse responses. SPA is not limited to a specific attribution method.
4 Partitioning Synthetic Data and Training Adaptations
We present the technical details of our proposed SPA framework for training multiple adaptations. We leverage an existing synthetic dataset D {x,, y J.v. for the purpose of this study. The use of an existing synthetic dataset allows us to focus on the effectiveness of the Partition then Adapt steps in eliciting diverse samples, while demonstrating the flexibility of our framework work with various synthetic datasets. Fig. 3 provides an overview of the framework. After obtaining the synthetic data, our approach consists three main steps: (1) computing data attribution scores for synthetic data points, (2) partitioning the synthetic dataset based on these scores, and (3) training multiple foundation model adaptations using parameter-effie efficiien fine tne-tuning techniques like LORA.
4.1 Computing Data Attribution Scores
Consider pre-trained foundation model M with parameters 0. Our goal leverage the synthetic dataset DD to train a set of K foundation model adaptations {M Each adaptation focuses on specific subset of the data that yields similar model behaviors. To partition the synthetic dataset, we employ data attribution methods that measure the importance of each training point to the model's predictions. Although we use influence function as an example to label the data, the SPA framework is not limited to influence function and can be extended to other data attribution methods, such as lexical overlap or TRAK (Park al., 2023). To calculate the influence function, we first fine-tune the pre-trained foundation model M the synthetic dataset D. The fine-tuning process optimizes the model parameters
minimize the loss function L(0) on the synthetic dataset using LoRA (Hu et al., 2021): [()) : En
0)), where E(.,-) is a suitable loss function, such as cross-entropy loss for language modeling tasks. This fine-tuning process yields the optimized model parameters
Next, we select a set of A test queries {(x] ("x .Y. (m) ((m))
>M which can be collection of questions
requiring various expertise knowledge to solve. For each test query (x("m) (m) (m))
y((m), ), we compute the
influence score of each synthetic data point (x..y:) E D using Eq. (1):
I((x;, Vi). (x- (m) .y: (m)
-ol(y) (m7) ((x("))))) Volly, M(x;;))
(m))
To efficiently compute the influence scores, we employ the stochastic estimation method proposed by Koh & Liang (2017), which approximates the inverse Hessian-vector product using conjugate

---

gradients. Although even this method is generally infeasible in foundation models due to their vast parameter space, the use LoRA (Hu al., 2021) makes it feasible by significantly reducing the number of trainable parameters. The computational cost of estimating the influence of a test query between the entire dataset 'D is the same as calculating the gradient of D. Another option to address this issue is to use the K-FAC approximation of the Hessian, as proposed by Grosse et al. (2023). We focus on the LORA approach and leave the exploration of K-FAC and other approximations for future work.
4.2 Partitioning Synthetic Dataset
After computing the data attribution scores for each synthetic data point with respect to the M test points, we obtain an influence matrix I R Nx M, where I,m represents the attribution score of the i-th synthetic data point for the m-th test point. To partition the synthetic dataset 'D into K subsets (Dxx 1K a clustering algorithm can be applied to solve the following objective:
Σ Σ
L.::
1
(4)
(D k= (X:Y))ED (x)-))))D,
where Ii denotes the i-th row of the influence matrix I, subject to U D2 D and D₂ nDw 0 for all k + k' In this work, we assume partitions are disjoint for the simplicity of the study.
The clustering algorithm assigns each synthetic data point yx) to one the K subsets based on the similarity of its influence scores across the M test points. This partitioning ensures that data points within each subset have similar impacts on the model's predictions. The choice of the clustering algorithm may depend ease of implementation, in this study, we use a ranking heuristic to partition the synthetic dataset. The details of this heuristic will be explained in the experiment section 1..1. However, it is important to note that our SPA framework is not limited to any specific clustering algorithm.
the specific characteristics of the dataset. For simplicity and
4.3 Training Multiple Adaptations with LORA
Once the synthetic dataset is partitioned into K subsets, we train a foundation model M₁. for each subset Dj using parameter--fffcient fine-tuning techniques like LORA (Hu et al., 2021). LoRA adapts the pre-trained foundation model parameters 0 by learning low-rank matrices A E Rxxd and B₃ E d for each weight matrix W E Rdd in the pre-trained foundation model, where < d is the rank of the adaptation matrices.
The adapted weight matrix W. for the foundation model adaptation M& is computed as: W W + A5 During the fine-tuning process, only the adaptation matrices A₃ and B are learned, while the pre-trained weights W remain frozen. This significantly reduces the number of trainable parameters, making it feasible to train multiple foundation model adaptations with limited com- putational resources. The training objective for each foundation model adaptation Mr is given by ming, Paal DEE,, Ma (x;; 0x)) where 0g represents the parameters of M₁, which include the pre-trained weights 0 and the LoRA adaptation matrices AR, B₁. By training multi- ple foundation model adaptations using LORA, we can efficiently adapt the pre-trained foundation model to different subsets of the synthetic data, each focusing on specific aspect of the data that yields similar model behaviors. This approach enables the creation of a diverse set of specialized models that capture different knowledge expertise present in the synthetic data, while leveraging the knowledge acquired during the pre-traine pre-tranning phase.
Inference with Multiple Adaptations During inference, given a user input x, our goal is to gen- erate diverse set of responses by leveraging the multiple foundation model adaptations trained on different subsets of the synthetic data. To achieve this, we randomly sample foundation model adaptation M₁ from the set of K adaptations {M& and generate the output y using the se- lected adaptation. By randomly sampling from the set of adaptations, we can generate diverse set of responses for the user input x. This approach ensures that the generated responses are not only diverse but also maintain reasonable quality. It is worth noting that this approach is compatible with various sampling techniques, such as temperature scaling, top-k and top-p sampling, which can further enhance the diversity of the generated responses.
5

---

To generate multiple diverse responses for the user input x, we can repeat the random sampling process multiple times, each time selecting different adaptation and generating response. This allows us to present the user with set of alternative responses that capture different perspectives or styles, enhancing the overall user experience. Unlike temperature sampling, which can degrade the quality of the generated responses, our approach maintains the quality of each response by leveraging the specialized knowledge captured by each adaptation. Moreover, our approach can generate diverse samples even when greedy sampling is used.
Experiments
In this section, we present the experimental setup and results for evaluating the effectiveness of our proposed SPA framework in improving the diversity of foundation model outputs. We conduct ex- periments both code generation tasks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) and several natural language understanding tasks.
5.1 Experimental Setup
Base Model and Synthetic Dataset For the code generation experiments, we use CodeLLaMA 7B (Rozière et al., 2023) as the base foundation model. CodeLLaMA is a state-of-the-arl language model specifically designed for code-related tasks, pre-trained on a large corpus of code and natu- ral language data. For the synthetic dataset, we utilize the OSS-Instruv dataset (Wei et al., 2023), which consists of 75,000 code-related question-answeriin answering pairs generated by GPT-3.5 Turbo (Ope- nAL, 2023). In the natural language understanding domain, we employ Llama-2 13B (Touvron et al., 2023) as the base foundation model. Llama-2 is a powerful language model trained on a di- verse range of web-scale data, demonstrating strong performance across various natural language understanding tasks. For the synthetic dataset, we use Platypus (Lee et al., 2023), which focuses on improving LLMs' STEM and logic knowledge. Platypus consists of curated sub-selection of public text datasets, comprising approximately 25,000 question-and answer pairs.
Data Attribution Scores We compare two methods for computing data attribution scores: influ- ence function and lexical overlap.
For the influence-based method, we hand-write 12 examples that cover wide range of knowledge for each domain. For each of these examples, we calculate the influence score with respect to each training example in the corresponding synthetic dataset using Equation 3. We then select the top test queries whose distribution influence scores over the dataset has the highest variance. This ensures that the selected test queries have diverse impacts the synthetic dataset, capturing different aspects of the domain knowledge. The resulting influence matrices Leode E R 75,000 and In/u E 800 are used for partitioning the OSS- and Platypus datasets, respectively,
For the lexical overlap method, we compute the BM25 score (Robertson et al., 1994) between each training example and the hand-written test queries. The BM25 score is calculated as follows:
N W+1
( (1 - ( f(x) Lavp
(ki + ()
I(2, Zguery)
Σ BESquery
log
+ [(2,1) +1
(5)
N₂
where f(z, t) is the overlap count, N is the number of training examples, L(z) is the length of the example, and Long is the average example length. We adopted the framework and the hyperpa- rameters in Ly & Zhai (2011). While we focus on influence function in this work, exploring the effectiveness of alternative data attribution methods like BM25 could be interesting direction for future research. More details are provided in Appendix A.
Partitioning the Synthetic Datasets To train multiple foundation model adaptations, we first set the hyperparameter K, which represents the total number of adaptations. We use K = 8 for both code generation and natural langauge understanding domain. For each data point the synthetic dataset, we aim to find the test queries that provides the most influence. Formally, for each synthetic data point (x,yy yr), we assign it to the subset Dj corresponding to the test point with the highest arg grggmaee Ix,i- where I represents either the influence matrix the BM25 score matrix. This process partitions the OSS-Innttruc dataset
influence score or the BM25 score: k*

---

HUMANEVAL
MBPP
Methods
pass @
pass@5
diversity
avg. KL
pass
pass 5
diversity avg. KL
Single 0.1)
50.02
56.42
0.58
NA
60.15
64.16
0.53
NA
Random (r 0)
50.15
63.10
0.69
0.008
60.65
70.42
0.64
0.014
Lexical (T = 0)
50.30
66.74
0.78
0.011
60.33
71.17
0.71
0.018
Influence (r = 0)
50.15
69.05
0.85
0.017
60.46
73.68
0.78
0.020
Table I: Results on the HumanEval and MBPP. 7 denotes the temperature used for sampling. SPA with influence function achieves the best performance in terms of diversity score and avg. KL divergence) while maintaining comparable pass @ performance the single adaptation bascline. p measures sample quality but also has positive correlation with diversity.
into K groups for code generation and the Platypus dataset into K groups for natural language understanding, Each group is associated with specific test example that has the highest influence on the data points within the group.
With the partitioned synthetic dataset, we train K model adaptations using the LORA technique, as described in $4.3. Each adaptation Mr is trained on the corresponding subset D& of the synthetic dataset, focusing on the specific coding knowledge captured by the associated test point.
Evaluation Metries We use the following
metrics assess the diversity:
1. Average KL. Divergence: Let P, and Pj be the probability distributions of the generated responses from two model adaptations i and j. respectively. The KL divergence between P, and Pj is defined P.(z)
as (K Pý) = Σ, (a(x) log P2(x)) The average KL. divergence calculated by averaging the pairwise KL divergence between all possible pairs of model adaptations. A higher average KL divergence indicates greater diversity among the model adaptations,
N-1
Average KL Divergence =
Σ Σ (PI(P.11 j=1+1
(6)
K
2. Sample Diversity: The average KI. divergence evaluates the diversity at the distributional level. We also consider the sample diversity which measures the uniqueness individual responses. We calculate the diversity score among K randomly generated samples for each problem. The diversity score defined as the proportion unique samples within the generated set. Specifically, it is calculated by taking one minus the ratio of the number of duplicate pairs to the total number of generated pairs.
Baselines We consider two baselines in the evaluation: (1) Single Adaptation, where a single model adaptation is trained on the entire synthetic dataset using LORA, and (2) Multiple Adapta- tions (random), where multiple adaptations are trained on randomly partitioned subsets of the syn- thetic dataset using LORA A. Hyperparameters used to train adaptations are provided Appendix A.
5.2 Code Generation Results
In the code generation domain, we evaluate the performance of our proposed methodology on two popular code generation benchmarks: HumanEyal (Chen al., 2021) and MBPP (Austin et al., 2021). HumanEval consists of 164 hand-written programming problems with corresponding test cases, while MBPP contains 399 held-out programming problems collected from online resources?. These benchmarks assess the ability to generate functionally correct code.
pass @ l metric In addition the diversity metrics, also evaluate the sample quality by pass @ and pass @ 5, measuring the percentage of problems for which at least one of the k generated samples passes all the test cases. Note that the pass@5 metric has strong correlation to the diversity of the samples. More diverse samples generally lead to higher pass@5 for k > 1.
Tabb. presents the evaluation results of SPA framework and the baselines the HumanEval and MBPP benchmarks. For the multiple adaptation methods, including random partitioning, lexical
We used the evalplus (Liu et al., 2023) framework to evaluate samples.
7

---

Pasogg
Pasag3
Peroentoge ariqus udens
300
4.00
Incel
Figure 4: How sampling temperature affects pass@1 pass@55, and Diversity Score for different methods on the HumanEval benchmark. The results are averaged over 4 checkpoints.
overlap, and influence function, greedy decoding (T 0) to generate samples. For the single adaptation baseline, we use temperature r 0.1 to induce some diversity in the generated samples, greedy decoding would not produce any diversity in this case.
Our primary focus on comparing the diversity metrics, namely the Diversity Score and the Average KL Divergence (avg. KL), across the different methods. SPA with influence function achieves the highest Diversity Scores 85% and 78% on HumanEval and MBPP, respectively. indicating that the generated samples are more unique and diverse compared to the other methods. Similarly, SPA with influence function yields the highest Average KI. Divergence 0.017 and 0.020 on HumanEval and MBPP, demonstrating greater diversity at the distributional level.
The random partitioning and lexical overlap approaches also improve upon the single adaptation baseline in terms of diversity metrics, but to a lesser extent than influence function. In particular, the lexical overlap induces more diversity than the random adaptations baseline. This suggests that even simpler data attribution methods can be beneficial for enhancing diversity when training mul- tiple specialized adaptations. It is worth noting that the pass scores, while primarily measuring sample quality, also have a positive correlation with diversity. SPA with influence function achieves the highest pass@5 scores of 69.05% and 73.68% on HumanEval and MBPP, indicating that the generated samples not only exhibit greater diversity but also maintain high quality.
In summary, these results underscore the effectiveness of our SPA framework in generating diverse code samples without compromising quality. By leveraging influence function for data partitioning and training multiple adaptations using LORA, SPA enables the generation of diverse and accurate code solutions, even when using greedy decoding. We also showed that training more adaptations than did not lead more diversity in Appendix B.
Impact of Temperature Fig. 4 presents the impact of temperature on pass 1, pass@5, and Di- versity Score for different methods on the HumanEval benchmark. The first plot shows that all methods, including Single, Random, Lexical, and Influence, exhibit similar patterns in terms of pass@1 1 performance. They achieve maximum accuracy (around 50.2%) when 7 = 0 and gradually decrease approximately 46.5% the temperature increases 0.5.
However, both pass@5 and Diversity Score improve for all methods as the temperature increases, which is expected higher temperatures encourage the model to generate more diverse samples. Notably, SPA with influence function (Influence) maintains its advantage over other methods across all temperature values, outperforming Single, Random, and Lexical methods. Although the perfor- mance gap between Influence and other methods narrows as the temperature increases due to the inherent diversity promotion of higher temperatures, Influence still maintains a lead at = 0.5.
5.3 Natural Language Understanding Results
To demonstrate the effectiveness of SPA in the natural language understanding domain, we evaluate its performance on several diverse tasks, including Big- Big-Bennc Hard (BBH) (Suzgun et al., 2022), GPQA (Rein et al., 2023), MMLU (Hendrycks et al., 2020), and WinoGrande (Sakaguchi et al., 2019). For tasks that involve multiple-choice questions, we asked the model to continue generating text even after producing answer choice for the purpose of measuring sample diversity. As shown in Fig. 5, SPA with influence function consistently achieves higher diversity scores and average KI. divergence compared to the lexical overlap and random adaptation across all tasks. Interestingly,
8

---

Avg. divergence across
Diversity tasks
0.16
influence
0.8
0.14
Texical cal
0.7
0.55
KLDwergonce
0.5
2.47
Dicersty
0.4
0.06
0.3
0..3
0.04
0.02
0.1
3.60
0.0
KMIL
RBH
Figure 5: Average KL divergence and diversity score various natural language understanding tasks. SPA with influence function consistently outperforms the lexical overlap and random adapta- tions, demonstrating its effectiveness in generating diverse samples across different NLU tasks.
random adaptations achieve better diversity than lexical overlap the GPQA task, suggesting that the effectiveness of partitioning methods may change depending on the task.
The diversity scores and average KL divergence values vary across tasks, reflecting the inherent differences in the nature and complexity of each task. Tasks like MMLU, which cover a wide range of subjects, tend to yield higher average KL divergence. We also notice that larger gap in average KL divergence does not necessarily translate to a proportionally greater difference in diversity scores. This suggests that while average KL divergence captures the dissimilarity between the generated sample distributions, may not always directly correlate with the actual diversity of the samples. Nonetheless, the consistent improvement achieved by SPA with influence function highlights its robustness and adaptability to various natural language understanding challenges.
6 Related Work
Sampling-based methods have been widely explored to generate diverse text from language models. One of the most common approaches is temperature sampling (Ackley al., 1985; Hinton al., 2015). Several studies have investigated the impact of temperature on model sampling and its effect on the diversity-quality trade-off (Caccia et al., 2018; Renze & Guven, 2024; Wang et al., 2023). Higher temperatures lead to more diverse but potentially less coherent samples, while lower temper- atures produce more conservative and deterministic outputs. When using high temperatures, human interventions can help to correct eтгоrs during the sampling process (Chung et al., 2023). Dynamic temperature strategies have also been explored during the model training and inference stages (Lin et al., 2018; Zhang et al., 2018; Wang et al., 2019; Chang et al., 2023).
Besides adjusting temperature, top-k, top-p (nucleus) sampling (Holtzman al., 2019) and their variants are common sampling methods (Fan et al., 2018; Meister et al., 2022; Hewitt et al., 2022; Ravfogel et al., 2023), which restrict the sampling space dynamically adjust the number of tokens considered at each step. Another line of works studied how to formulate quality-diversity trade-off search or RL problem (Naik al., 2023; Lim al., 2024; Mudgal et al., 2023; Bradley al., 2023; Ji et al., 2023).
7 Conclusion
In summary, we proposed SPA, which that leverages synthetic data, data partitioning, and model adaptation elicit diverse responses from foundation models. By partitioning synthetic datasets into subsets that capture unique aspects of the data and training multiple model adaptations optimized for these subsets, SPA enables the generation of diverse and high-quality responses.
Limitation One main challenges is the computational cost associated with influence function, which require several extra epochs of backward passes estimate, Future work could explore more efficient data attribution methods, such as TRAK (Park et al., 2023) and K-FAC (Grosse et al., 2023).

---

Additionally, the ranking heuristics used to approximate Eq. (4) can be replaced by more advanced clustering algorithms. Additionally, serving multiple LoRA adaptations poses significant computa- tional challenge in real-time serving framework. Recent works such as S-L.RA and FL.RR (Sheng al., 2023; Wen Chaudhuri, 2024) can be considered to accommodate this overhead.
References
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. A learning algorithm for boltz- mann machines. Cogn. Sci., 9:147-169, 1985. URL pastss:/aape semanti wwwpaaniicccolar.oooooomo
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. J. Mach. Learn. Res., 18: 18:116:1111::40 2016. URL https://apil.
semanticscholar. prg/CorpusID: 10569090.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07755 2021.
Herbie Bradley, Andrew Dai, Hannah Benita Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bel- lagente, Jeff Clune, Kenneth O. Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through ai feedback. ArXiv, abs/2310.13032 2023. URL https: //api. semanticacholar. org/CorpusID: /ts//orppuIDD:264055666 264405960.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vose Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: /tsttps:///rreee httpt::::///rooeedingss.s http://oooeeiinn...rrrirpppppppp neurips cc/paper_fteeeeeceeeeeeee wwwts//pppee///////// ffiee/ 14577766666100004444100000022020
Massimo Caccia, Lucas Caccia, William Fedus, H. Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. ArXiv, abs/1811.02549, 2018. URL https://api.
semanticscholar. Prr/CoorpusID: 53208122.
Chung-Chine Crigg-Chinggo Chang, D. Reitter, Renat Aksitov, and Yun-Hsuan Sung. Kl-divergence guided tem- perature sampling. ArXiv, abs/2306.01286, 2023. URL https: // seg.p.ssmmaticcoolleeee org/ CorpusID:259063771
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri- son Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Voss,rt-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr. Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa. Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. ArXiv, abs/2107.03374 2021. URL https://apl semanti comanticscholas. org/ Cttp/CCrpusI:::: 23575557777 :235p5554:
John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while main- taining accuracy: Text data generation with large language models and human interventions. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https:
//api. wwwanticscholartttttteeeacaaaaaa org/ org/CorpusID : 259096160.
10

---

Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the As- sociation for Computational Linguistics, 2019. URL httpss//app https::/api..emanticchhhe semanticscholar org/ :rrpuuII::2967330
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-tramslation at scale. Eilen Riloff, David Chiang, Julia Hockenmaier, and unnichii Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500, 2018. Association for Computational Linguistics. doi: http///llaatooooooooo//////a
Brussels, Belgium, 118866///1/D1111040 URL
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In An- mual Meeting of the Association for Computational Linguistics, 2018. URL https://api.
semanticscholar owoo/orr/ppeeeeooooooooo Chtp/CrrpusDDD.. 44134226.
Roger Baker Grosse, Juhan Bac, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukovsiut.e, Ka- rina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Sam Bowman. Studying large language model generalization with influence functions. ArXiv, aos/2308.03296, 2023. URL httpps//aap cemanticscholane
wttg/CorpuuID::26662887
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, hass22099033000 2020. URL httpss///ap..sennnncccooooo
ceemanticsccolla.. wwwp:ooogg//aaeeeee 221516475.
John Hewitt, Christopher D. Manning, and Percy Liang. Truncation sampling as language model desmoothing. Conference on Empirical Methods in Natural Language Processing, 2022. URL
https: //api. www.nticscho org/ Corpus ID: 253157390.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural net- work. ArXiv, abs/1503.02531, 2015. URL https: / /api...p.../ semanticschollar arg/CorpusID; 7200347.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ArXiv, abs/1904.09751, 2019. URL https://ppi. www.nticsccoolar....... C. CorpusID: 127986954.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. ArXiv, abs/2212.09689, 2022.
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685, 2021. URL httpss///pp..ssmmmiicccc...aaaa org/ waw.a.or//oooooeeooeeeeeeeoeeoo :IterpusI::235480000
Haozhe Ji, Pei Ke, Hongning Wang, and Minlie Huang. Language model decoding as direct met- rics optimization. ArXiv, abs/2310.01041, 2023. URL https: http::://api.ssemaiiccccoooo semanticscholes org/
CorpusID:263605888
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017. URL htttt::///ppp http:///ppp.smmmmcccccchheeee sewantccscholarr..
org/CorpusID: 13193974.
Ariel N. Lee, Cole Hunter, and Nataniel Ruiz, Platypus: Quick, cheap, and powerful refinement of 1lms. ArXiv, abs/2308.07317, 2023. URL https: //api. semanticscholar org/CorpusID: 260886870.
Minhyeok Lee. A mathematical investigation of hallucination and creativity in gpt models. Mathe- matics, 2023. URL httt::://ppp..emmmiicccoo..ee wwtp/oorps//o/iieeeeee ID: 258768397.
Bryan Lim, Manon Flageat, and Antoine Cully. Large language models as in-context ai generators for quality-diversity. ArXiv, abs/2404. 15794, 2024. URL. https: //api. http://ppp..aatiiccceeeee semanticscholar. org 269362558..

---

Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and ann' Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2985-2990, Brussels, Belgium, Octoher-November 2018. Association for Computational Linguistics. doi: 10.18653/ 1331.
v1/DI8- 1331. URL htt.ttts::/laaaaallllloooooooo
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// openreview wwwpeeeetemoddiiueeeeeeeee
Yuanhua Lv and ChengXiang Zhai. Lower-boundigg term frequency normalization. In Interna- lional Conference on Information and Knowledge Management, 2011. URL httpps///ap
semanticscholar. org/ Corpus :t 14029221.
James Martens. Deep learning via hessian-free optimization. In International Conference on Ma- chine Learning. 2010. URL https://api. www.stiiscoooseee.m......e.
org/CorpusII] :tte:::::111151111 11154521.
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. Typical decoding for natural lan- guage generation. ArXiv, abs/2202.00666, 2022. URL https: wwt..eeeeaanncolll..... org CorpusID:246442062
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins. Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models, ArXiv, abs/2310.17022,, 2023. URL https:
seaaaiissemnnccccolllle. www.aachhoaa....oeeeeeeoeeeeeeee het/CorppsII:::00 264491118
Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. Diversity of thought improves reasoning abilities of llms. 2023. URL https://api.
semanticscholar wwt.:/ccchhaaaoooooooouoooee orgg/orppsID:28777 267938465.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https://api.
semanticscholar. Irrg/CorpusID::: 2575328815
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.021555 2022.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. International Conference on Machine Learning, 2023. URL https://apl www.seeeinnnnooleeee www.:oolr.rrrrrrrrorre/e org/CorpusID :57757267..
Baolin Peng, Chunyuan Li. Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv, abs/2304.033777, 2023.
Shauli Ravfogel, Yoav Goldberg, and Jacob Goldberger. Conformal nucleus sampling. In An- nual Meeting of the Association for Computational Linguistics, 2023. URL https://api. 258479879.
semanticscholar, wwwaaro.rraeeeeeee.e
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level prof q&a bench- mark. ArXiv, (tb//2311.12022 2023. URL https: httppp///app.ssemattcccccooo.o. 265295009.
Matthew Renze and Erhan Guven. The effect of sampling temperature on problem solving in large language models. ArXiv, abs/2402.052011, 2024. URL www.nticcccoolr...rogo..... wtt.://ccccaaaaoooorrrrrruaaaa CorpusID::255447768.
org/
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi at trec-3. In Text Retrieval Conference, 1994. URL. http:: //api. http://app..mammiiieeeeeeme semaantc org/CorpusID 41563977.
12

---

Baptiste Rozière. Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Evtimov, Joanna Bitton, Man- ish P Bhatt, Cristian Cantón Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D D'efosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Seialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. ArXiv, abs/2308. 102//208.11250, 2023. URL https://api. semanticscholar org/ wttg/oorpusD::21100999
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande. Com- munications of the ACM, 64:99 106, 2019. URL https: //apii////paaa stw.pii.semmaiiccoooooa... org/
1erpusID:188893658
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Araud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022. URL https: http///pppreveiieeee/ http:/wwwww.eeeeeeeeeeeeeeeeeeee net www//fruumm? 9VbbbDODN,
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 86-96, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v// P16-1009. URL httpp:s://aaaaathllllooo hrtpa/annnoooooooo/e1----111 org/P16-1009.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and lon Stoica. S- lora: Serving thousands of concurrent lora adapters. ArXiv, abs/2311.03285, 2023. URL
https://api. semanticscholar wttp://ooooosss1::00000377
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovin llama model. https://github. http:///ggghaab...ommm/tstttluas httm///ataa-lbb/ttafoodddd alpaca, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Andrew Trotman, Antti Puurula, and Blake Burgess. Improvements to bm25 and language models examined. Proceedings of the 19th Australasian Document Computing Symposium, 2014. URL https://api. www.nticschoiiii..
wwwpcchhlarrrrg/oooooo org/CorpusID: 222222227.222
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Neural Information Processing Systems, 2017. URL https: wwt.p.sssmaticchollaaa. arg/CorpusID: 13mmssII:376689, 13756489.
13

---

Chi Wang, Susan Liu, and Ahmed Hassan Awadallah. Cost-effective hyperparameter optimization for large language model generation inference. ArXiv, abs/2303.04673, 2023. URL https: sempppsisantcccccoooooooo
Pei-Hsin Wang, Sheng- lou Hsieh, Shih-Chieh Chang. Yu-Tin Tu-Ting Chen, Jia- Yu Pan, Wei Wei, and Da- Chang Juan. Contextual temperature for language modeling. ArXiv, abs/2012.13555, 2019. URL
https:/ /ttps://api... semanticscholls org//oo//ooo Corpus 214250287.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv, wttp/211...05t. 2022a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022b.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP)), pp. 6382-6388 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. /0.18663//1/DD1-1677 URL http::::/aaclaatholllg/oo//
org/D19-1670
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. URL https://api.
http::///ap..smmancccccoooo www.sti
2374111688
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023.
Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In
The Twelfth International Conference on Learming Representations, 2024. URL https:// wwwpi//wweee/ooommddddeeeeeeeeee
Yeming Wen, Pengcheng Yin, Kensen Shi, Henryk Michalewski, Swarat Chaudhuri, and Alex Polozov. Grounding data science code generation with input-output specifications. ArXiv, abs/2402.08073, 2024. URL htt..ppp:e/gg....mmmnccccoooo..
semanticscholar org.CorpusID:227737722
Xu Zhang, Felix X. Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up soft- max embedding. ArXiv, abs/1809.04157,, 2018. URL https: //api. httppp////pp.eeemaiccccc///////e semanticscholar. org/ CorpusID:5219355
A Experimental Setup Details
This section provides additional details on the experimental setup that were not included in the main content due to space constraints.
Computing Data Attribution Scores For the lexical overlap method, we a publicly available BM25 (Lvv & Zhai, 1; Trotman et al., 2014) implementation written in Python and released under https://pypi..ot org www..////p..o.eeeaaaannbobbaaeae
We used the default hyperparameters.
When calculating the influence function, we employ the conjugate gradient method with LiSSA approximation (Martens, 2010; Agarwal et al., 2016). We leverage a publicly available implementa- For the Ons-rntt dataset, we use a damping factor of 0.001, a depth of 120, and 500 repeats, following the guideline that the product of depth and repeats should be roughly equal to the dataset size. For the Platypus dataset, we use a depth of 120 and 200 repeats. It is worth noting that computing the influence function is also intensive with LOR A. Each column of the matrix Eq. (4) requires approximately one epoch of backward passes over the entire synthetic dataset. On the OSS-Instruct dataset, this takes roughly hours using a single 100 80GB GPU. However, this is offline computation which consumed before deploying the model to users.
tion from sctps://githut com/ www.oooonoooooooooooooieeeee
14

---

After obtaining the data attribution matrix, we observe that using the ranking heuristic presented in $5.1 leads to imbalanced partitions. To achieve more balanced partitions, we normalize the data attribution matrix before applying the heuristics. We leave the exploration of more advanced clus- tering algorithms, such as k-means, for future work.
Details for Model Adaptations In this section, provide details the computing resources and hyperparameters used for training the model adaptations in both the code generation and nat- ural language understanding domains. For the code generation experiments, we use machine with A100 40GB GPUs and train each partition for 400 steps, which takes approximately 80 minutes (each partition). The hyperparameters bigcode- proj hett/et///ae pectstaarodder peroddr//rreee main. The base model CodeLLaMA-7B-Pythheee use b precision to accelerate training. The per-device train batch size is set I, with gradient accumulation step of 20. We use a learning rate 2e-4 with a cosine learning rate scheduler and 20 warmup steps. For the LORA hyperparameters,
mostly adopted from https: htt//githut com/
and we
rank (r) of 16, alpha of 16.
In the natural language understanding domain, we train each partition for 400 steps, which takes approximately 40 minutes, using the Llama-2 13B model as the base model. The training time is shorter compared to the code generation domain because the Platypus dataset much smaller than ISS-Instruc.. The hyperparame- ters are mostly adopted from https: / /tss://githne /gtb. com/ Seieennee/Plaaa
Diversity v.s. # adaptations
0,850
0.825
0.775
We per-device batch size of and a gradient accumulation step of 4. The learning rate is set to le-4, with total of 20 warmup steps.
0.750
influence
0.725
random
0.700
the computational costs mentioned this section, including the time and resources required for comput- ing data attribution scores and training model adapta- tions, are offline. These costs are incurred before de- ploying the models to users, and they do not affect the inference time.
Figure 6: Diversity score as function of the number of adaptations on the Hu- manEval benchmark.
B Impact of Number of Adaptations
In this section, we investigate the impact of the number of model adaptations on the diversity of the generated responses. We focus on the HumanEval benchmark in the code generation domain and vary the number of adaptations from 8 to 12. The results are presented in Fig. 6.
As shown in Fig. 6, the diversity creases from to 12, regardless of the partitioning method used. These results suggest that increas- ing the number of adaptations beyond certain point may not necessarily lead to an improvement in the diversity of the generated responses.
remains relatively stable as the number of adaptations in-
C Test Queries
In this appendix, we provide the hand-written test queries used in our experiments for both the code generation and text generation domains. These examples were utilized to compute data attribution scores. Most of the examples are generated by GPT-4 (OpenAI, 2023).
C.1 Code Generation Domain
1. """Title: Longest Pal inddomicc Subsequence
2 Query: Write a function to find the longest palindromic subsequence in a given string. Solution: иm*
5 def longest batgest_ppllnddromccsubbeeeeeeee
len(s)
dp = [[o] for in range (n)]
15

---

for rn):
dp[i][i]
for length in range (2, n+1):
for in range(n-length-1))
length
15
=[i] a[j] length 2:
16
dp[i] [j] =
elif s[i] == s[j]:
dp[i][j] dp[i+11 [j-1]
else:
20
dp[i] [j] max (p-il(j-1))
return dp[o] [n-1]
2. """Title: Fibonacci Number
Query: Implement a function to calculate the nth Fibonacci number using dymamic programming. Solution:
fibonacci (n):
if n <= 0:
return 0
elif 1:
return 1
33
fib [0] (n + 1)
fib[1] =
for in range n + 1): fib[i] = fibli 1] + fibli - 2]
38
return fib(n]
42 3. """Title: Sun of Two Largest Elements
Query: Create function that takes a list of integers and returns the sum of the two largest elements in the list. Solution:
46
def sum_of wwwoof_wo____oooooooooooooo Swo_largesttoee (nums):
if len (nums) 2:
return sum (nums)
50
largest second_largest float ('-inf')
for num in nums:
if largest:
second second_largest. largest largest num
55
elif num > largest second proondlarrges--
return largeat second largest
4. "="Title: Maximum Subarray Sum
Query: Implement function find the maximum aubarray in given array ~ integers.
63
Solution:
def nax_subarray.. sum (nums) max_sum float(' (t-ifff current_aum
68
for num in nums:
16

---

current_sum max (num, current_sum num)
max_ max (max_ aum, current_aume
72
73
return _um
5.
"""Title: First Non-Repeating Character
Query: Create function that takes string and the first non-repeating
+ character the string.
Solution:
def first_non_ (ww..epppiin..caaater(()))) char_count 0
for char in 8: char count [char]
gentttgeeetchaa (char, 0)
for char 8: 86
if char_count [char] == 1: return char
return None
90
6. """Title: Merge Two Sorted Lists
Query: Write function merge two sorted lists into single sorted list. Solution:
95
def merge_ seree__ooted_iisee (listi, list2);
merged_liat list
[]
=
while (1nisi))) and j len (1niatt));
100
if listi[i] liat2[j]:
101
berged_list append (1isti11))
+=
103
else:
104
merged larged_istt.appendee append
105
+=
while len (listi):
108
merged_liee appen((1ist1
109
+=
while len (1ist2);
112
merged_ lisedd___stt___eeeeeeeeeeeeeeess append
113
1
114
return merged list list
117
7. """Title: Remove Prime Numbers from List
118
Query: Create function that takes a list of integers and returns a new list with all the prime numbers removed. Solution:
121
def is (s_prime(nnn)) (num):
122
if num < 2: return False for in (2, (num 0.5) 1): if num % i 0: return False
125
return True 126
127
def remove remove_prime numbers (nums)
130
return [num for num nums not is_prime (num)]
131
8. """Title: Longest Common Substring

---

133 strings.
Query: Write function to find the longest common substring between two given
134
Solution:
135
136
def longest ceceess_commoo_suute sireronn_uubstri(ssrrr str2):
len(s))i), len(str2)
dp [[o] (n 1) for range( 1)]
139
max_ max_length 0
140
end _ideideex 0
141
for in range(1, 1):
143
for in range if strili 1] str2 1]:
1):
144
145
dp[i][]] = dpli 1][ 1] + if dp[i][]] [j] max_length:
max_length dpli] [j]
148
end_index
149
else:
150
dp [i] [j]
152
start_index end_inder max benghhngth
153
return stri (erartttt__ndes end_ index]
154
9. """Title: Kth Largest Element in Unsorted Array
Query: Implement function to find the kth largest element in unsorted array. Solution:
157
138
159
def kth_largee lah_largest__m _eeesstteeemeeeteeeeeee (nums, k):
k len (nums)
161
162
def partition right):
163
pivot numa [right]
164
i left -
166
for range range (left, right):
167
if nums [j] <= pivot: += nums [i], numa [j] numa [j], nums [i]
168
171
nums 1], nums [right] nums [right], nums return i +
172
def quick_select select (left, right): if left right: return nums [left]
176
177
Divot_inder partition(left,, right)
180
if k lpioo__iidde return nums [k]
181
182
elif k < pivot_index:
return (lett,,, pivot_index 1)
else:
185
return quick_ seletsseletttee (pivotttpiivt index 1, right)
186
return (uick_select(O, len (nums) 1)
189
=""Title: Product Array Elements
190
Query: Create function that takes a list of integers and returns the product of all the elements.
Solution:
193
product_of. Sfftuctt_f__eemmntee elemente (nums):
194
product = 1
195
for num in nums:
18

---

196
product ==
return product
198
199
11.
"""Title: Binary Search
200
Query: Implement a function to perform binary search sorted list of integers. Solution:
203
binary_search (nums, target):
204
left = 0
205
right len (nums)
207
while left right:
208
mid (left right) // 2
209
if nums [mid] == target: return mid
212
elif numa [mid] target:
213
left = mid +
214
else:
right mid
216
217
return
218
12. """Title: Find Missing Number
Query: Create function that takes list of integers from (inclusive) with one number missing and returns the missing number. Solution:
-
221
222
def find_miin hind_missing.r number (nums):
224
len (nums)
225
expected_sur sum (n 1)) // 2 actual_sum = sum (nums)
226
return expected_sum sum actual_sun
C.2 Text Generation Domain
1. Title:
Economic Impacts of the Black Death
2 Query: Explain the economic impacts of the Great Mortality in medieval Europe. Response:
The the Great Mortality drastically reduced the population of Europe, leading to severe labor shortages, higher wages, lower prices for land, and shift economic power from the feudal lords to the working class and merchants.
2. Title: Photosynthesis Process
Query: Describe the process photosynthesis and its importance to the Earth's 's ecosystem.
+
7 Response: Photosyathesis is the process by which green plants and some other
4 organisms use aunlight to syathesize nutrients from carbon dioxide and water. It generates oxygen as byproduct, which is vital for most life forms Earth.
Title: Calculating Travel Distance
10 Query: If car travels at 60 miles per hour for 3 hours, how far has it gone?
Explain your calculation.
Response: The has traveled 180 miles, calculated 60 miles/hour hours.
13 4. Title: Utilitarianism vs Deontological Ethics
14 Query: Discuss the main differences betreen utilitarianian and deontological ethics. Response: Utilitarianism focuses the outcomes consequences of actions to determine morality, while deontological ethica considers the actions themselves and the adherence duties rules as basis for morality.
15
5. Title: Advancements in Quantum Computing
Query: What are the key advancements in quantum computing over the last decade?
19

---

Response:
Key advancements include the development of quantum supremacy, error correction, and the creation of stable qubita, enhancing computing power b reliability.
20
Wedding Traditions in India Query: Compare the wedding traditions of Northern and Southern India. Reaponse:
6. Title:
Northern Indian weddings often feature elaborate rituals like Sangeet and Mehendi, while Southern Indian weddinge are marked by rituals like Kashi Yatra
-
+ and Donjal. Both have vibrant traditions but differ in cultural practices and attire.
Title: Deforestation the Amazon
26 Query: What are the primary causes of the Amazon rainforest's deforestation and what measures are being taken to address it?
Response: Primary causes include agriculture, logging, and infrastructure development. Measures address this include enforcement of laws, satellite monitoring, and international cooperation on sustainable practices.
28
8. Title: Theme of Ambition in Macbeth
Query: Analyze the theme of ambition in Shakespeare's 'Nacbeth'.
Response: Ambition in 'Macbeth' serves as both driving force a tragic flaw for the characters, particularly Macbeth, leading to his rise and eventual domfall as he succumbs to the ambition spurred by the prophecy and his wife's encouragement.
Title: Global Impact Renewable Energy
34
Query: Discuss the global impact of renewable energy sources on climate change. Response: Renewable energy sources like solar and wind have significant impact on mitigating climate change reducing dependence fossil fuels, decreasing greenhouse gas emissions, and promoting sustainability. Countries adopting renewable energy contribute to global reduction in carbon footprints, which can help meet the goals set by international climate agreements like the Paris Agreement.
Title: Role Artificial Intelligence Healthcare Query: Explain how artificial intelligence is tramsforming healthcare. Response: Artificial intelligence in healthcare is transforming the industry by enhancing diagnostic accuracy, improving treatment personalization, and optimizing operational efficiencies. AI applications include predictive analytics for patient management, automated imaging and diagnostics, and robot-assisted surgeries, leading to faster, more accurate patient care and reduced healthcare costs.
38
Title: Cultural Sigmificance Food Japan
42 Query: Describe the cultural significance of food in Japan and how it reflects Japanese society.
Response: Food in Japan is deeply intertwined with the nation's culture, reflecting aspects of beauty, seasonality, and regional diversity. Traditional dishes like sushi and bento embody aesthetic principles and social customs, such as respect for nature and meticulous attention to detail. Food rituals, such as tea ceremonies, also highlight the importance of mindfulness and harmony in Japanese society.
45
12. Title: Economic Effects Globalization Query: Analyze the economic effects of globalization on developing countries. Response: Clobalization has both positive and negative economic effects developing countries. the positive side, it allows markets, increases capital inflow, and promotes technology transfer, leading to job creation and econonic growth. However, it can also lead to economic dependency, cultural homogenization, and the potential exploitation of local resources and labor, which might exacerbate inequalities and social tensions.
international
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We propose the SPA framework which is the main contribution of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation stated in the Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
No theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Hyper-paramene parameters given in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
In the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
ooooooooolill oliccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.c.. cc/ wttpccccpppcccccceee wttpc/uuudee//oeeeeeeeeeeeeeeeee
ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
The results are averaged over 4 checkpoints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is given in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tte::////uui../.eee
Answer: [Yes]
Justification: No violation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethies.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research on how to generate diverse samples. It is not directly applied in any product.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We use existing dataset and models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the used datasets, models and the repos.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswi wecohhhodee com/ /atattts has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
The code is included in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "qp5xXbKipq", "reviewer": "Reviewer_BWst", "summary": "This paper aims to improve the diversity of generations of LLMs. Current methods are not great because when they offer more diversity, they lose in quality (for instance worse performance). This is for instance the case of temperature sampling. In this paper, they propose synthetise partition and adapt which consists in taking a synthetic dataset, partitioning it using influence function for instance and a few high quality test examplars, train a multiple models on each of these subsets and then eventually, use this collection of models to do sampling. They show that thanks to their approach, they are able to get better diversity while maintaining good performance on code and MMLU.", "questions": "I mentioned my questions in the weaknesses seciton.", "limitations": "The authors mentioned the limitations of their work and they look sound to me.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, I like this paper. I think that the question addressed by the authors is central and to me one of the biggest issues with current LLMs is their lack of their generations diversity. To my knowledge, this paper is one of the first ones that proposes a method to alleviate this issue and I hope many other works will follow. The paper is well presented, easy to read and the results are clear.", "weaknesses": "I want to raise a few points regarding the approach:\n\n- **In-context approach?**: I think that one criticism could be that he method may seem a bit computationally heavy (because of the data clustering step + training many models), I know that one of the most commonly used methods in practice to ensure diversity is to select different text chunks and use them to seed the prompt and get a different generation. This was used for instance in the Phi and cosmopedia approaches to generate synthetic textbooks [1]. In other words, do you think that the finetuning step is necessary? Wouldn't putting one of the clusters of data (that you create with data attribution) **In-context** be enough?\n\n\n- **Role of the rank of LoRA**: I imagine that the rank of LoRA may play a role here? Intuitively, if the rank is very small, all the models will have similar representations even after fine-tuning. As you increase the rank of LoRa, you may increase the diversity of your generations. Is it correct?\n\n- **Diversity measures**: The diversity measures proposed by the authors are legit. However, maybe another interesting one would be to train a new model on datasets that they have generated with their approach. If the newly trained model gets a better performance, I think this would be the best proof for diversity.\n\n- **Increasing the number of clusters leads to better diversity + performance?**: the current paper just considers the case where there are 8 clusters in the data attribution phase. One experiment that would have been nice is also to understand in the case of code, how pass@5 improves as the number of clusters increase ? \n\n- **Some improvements that may be done for the introduction**: I think it is a shame that the method is not fully presented in the introduction. Maybe it would be good to summarize it in a few words? Besides, I believe that it would be nicer if Figure 3 is in page 2 as a main figure.\n\n\n\n[1] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., ... & Li, Y. (2023). Textbooks are all you need. arXiv preprint arXiv:2306.11644.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We'd like to address your main concerns.\n> In-context v.s. Fine-tuning\n\nWe can formulate the difference as sampling from $P_{\\phi_1} (y|x)$ and $P_{\\phi_1}(y|x')$ for in-context versus $P_{\\phi_1}(y|x)$ and $P_{\\phi_2}(y|x)$ for our approach ($\\phi$ denotes model parameters). We argue that sampling from multiple distinct distributions (SPA) produces more diversity than modifying the prompt. In the general response, we conducted additional experiments comparing SPA with Diversity of Thought, an in-context approach. The results show SPA's significant advantage. Moreover, one could potentially combine them by sampling from $P_{\\phi_1}(y|x)$ and $P_{\\phi_2}(y|x')$, further enhancing diversity. \nFinally, we assume the LLM must go through the synthetic data tuning before deployment, whether for domain adaptation, value alignment, or distilling knowledge from a stronger model. Given the diminishing returns observed with increasing synthetic data, the motivation for SPA remains strong.\n> Role of LoRA rank\n\nWe agree that your intuition about LoRA rank is correct. A smaller rank would lead to more similar representations, potentially reducing diversity, while a larger rank might enhance it. We used a fixed rank for consistency in our experiments, but exploring different ranks is an interesting direction for future work. Yet, the benefit of sampling from multiple distinct distributions remains valid regardless of the rank.\n> Number of adapters\n\nThe optimal number of adapters highly depends on the size of the synthetic dataset, with larger datasets typically inducing a higher optimal number of adapters. We observed improvements in diversity when increasing the number of adapters from 4 to 8, though this data was not included in Fig. 6, suggesting that 8 adapters is roughly the sweet spot for the synthetic dataset we used (75k data points).\n> Further improvements\n\nThank you for your value suggestions for new diversity measures and how to improve the intro section. Regarding the new diversity measure, we'd appreciate clarification on training a new model on generated datasets, as our single model adaptation baseline is already trained on the entire synthetic data. We’ll summarize the method more concisely in the intro and move Fig. 3 ahead in our revision.\n\nWe appreciate your positive feedback and hope this addresses your concerns. We kindly ask for your support of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIBfZRWdj2", "reviewer": "Reviewer_j8SA", "summary": "The paper introduces Synthesize-Partition-Adapt (SPA), a framework designed to generate diverse and high-quality responses from foundation models. SPA uses synthetic data and data attribution methods to partition data into subsets, training multiple model adaptations for these subsets.", "questions": "- The motivation of three phases: synthesize, partition, then adapt is not clearly discussed. How to ensure that the existing dataset is diverse?\n    \n- The framework heavily relies on the existing synthetic dataset where its diversity is already a concern. As the authors mentioned, the problem of generating diverse synthetic dataset is still a problem with current generative models. Would finetuning on these synthetic datasets destroy the generalizability of the models?\n    \n- The method relies on $M$ questions requiring various expertise knowledge to compute influence matrix for partitioning. Does those M questions need to be manually chosen?\n    \n- Why partitioning the datasets using the importance scores and then finetune different LoRAs with K partitions would lead to K diverse LoRAs components? The connection is vague to me.\n    \n- The method relies on a predefined number of partitions K (or finetuned LoRA components) when training. What if during the inference time, we would generate a lot more than K diverse responses? Does each finetuned model still suffer from diversity problem?\n    \n- During inference, LoRA components are sampled randomly, does it affect the quality of generated responses? What if some components being biased or learn some skills while losing other skills? It is not clear that, after partition and finetuning, which LoRA learns which expertise. Can each component has a weight depending on the question?\n    \n- The paper is missing most of the important baselines and citations, I listed some here for reference:\n    \n    - KL-Divergence Guided Temperature Sampling\n        \n    - Diversity of Thought Improves Reasoning Abilities of LLMs\n        \n    - Large Language Models as In-context AI Generators for Quality-Diversity\n        \n    - Controlled Decoding from Language Models\n        \n    - Quality-Diversity through AI Feedback\n        \n    - Language Model Decoding as Direct Metrics Optimization", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "I believe this paper is studying an important problem. The diversity problem in LLM has received attention recently.", "weaknesses": "- The motivation of three phases: synthesize, partition, then adapt is not clearly discussed (See my questions). The authors spend more time to describe what they are doing but not why they are doing so or why these specific designs would lead to desired outcome, which is diverse sampling technique.\n    \n- The experiments only compare with two simple baselines ablated from the proposed methods, omitting existing baselines in the literature.\n    \n- The paper omits most of the existing works in the literature regarding diverse sampling for LLMs and only considers simple temperature sampling techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We'd like to address your main concerns.\n> Motivation of why doing SPA and why doing so leads to diverse samples. Role of synthetic data and the generalizability of it.\n\nWe would first like to clarify that LLMs must typically go through synthetic data tuning before deployment, whether for domain adaptation, value alignment, or distilling knowledge from a stronger model. SPA is designed to **utilize** this existing synthetic data used in LLM post-training stages (or mid-training stage). Operating within this existing paradigm, SPA addresses the inefficiency of tuning a single model on **increasingly larger** synthetic datasets due to diminishing returns. \n\n\nThe first motivation, as stated in Section 3, is to turn this potential inefficiency into an advantage for diversity from the multi-model perspective. The second motivation is sampling from multiple distinct distributions (i.e., different model adaptations) naturally yields more diverse outputs compared to sampling multiple times from a single distribution (i.e., a single model). \nThis is why SPA leads to diverse samples by creating multiple model adaptations, each trained on a different partition of the synthetic data. Importantly, due to diminishing returns, training on data partitions doesn't significantly affect accuracy, as shown in Sec. 3.\n\nFurthermore, our partitioning strategy, especially when using influence functions, creates subsets of data that induce distinct model skills. This further enhances the diversity of the resulting adaptations (compared to the random and lexical partitions).\n\nRegarding generalizability, since SPA uses data already part of the post-training process, it doesn’t introduce new generalization risks. While our method benefits from more diverse data, it's designed to improve diversity even when working with less diverse synthetic datasets, by being able to sample from multiple distinct distributions.\n> More comparisons and related works\n\nThank you for this feedback.  For the additional comparisons, please see the general response. We didn't compare to QD search methods because SPA and QD methods operate on fundamentally different principles. SPA operates at the model level, creating multiple models during an offline process. The main computational cost occurs during the offline partitioning and adaptation phases. This allows for quick, parallel diverse sampling at inference time. In contrast, QD search operates at the generation level, iteratively going through mutation, evaluation, and refinement cycles. It introduces overhead during sampling, making it less suitable for real-time applications. We will also expand our literature review to include all the suggested works and other relevant studies.\n> Connection between partitioning and diverse LoRA components\n\nThe influence function captures how training examples affect model predictions on specific test queries. Our partitioning strategy groups examples with similar influence patterns, resulting in K clusters. Each cluster consists of synthetic data that most strongly influences a particular test query (detailed at line 257). When fine-tuning LoRA on these individual clusters, each component focuses on the skills emphasized by its associated test query.\n\nImportantly, we don't require extremely diverse LoRA components because the diversity primarily stems from sampling across multiple distinct distributions (motivation #2). Even with random partitioning, this strategy ensures that these distributions are sufficiently different to generate diverse outputs when sampled collectively.\n> Generating more than K diverse responses\n\nWhile we train K adapters, SPA is not limited to generating only K diverse responses. Each adapter can produce multiple outputs, especially when combined with other sampling techniques (e.g., temperature sampling). In our experiments, for temperatures > 0, we generated a total of 120 samples to calculate pass@1, pass@k, and diversity scores. For SPA, this meant sampling 15 samples from each of the 8 adapters. In contrast, for the single model baseline, all 120 samples were drawn from the same distribution. This sampling strategy highlights a key advantage of SPA: even when generating many more than K samples, we're still drawing from multiple distinct distributions. This inherently promotes diversity compared to repeatedly sampling from a single distribution.\nMoreover, SPA is complementary to other diversity-enhancing methods. \n> Quality and bias in randomly sampled LoRA components\n\nOur experiments show that random sampling of LoRA components during inference doesn't significantly affect response quality, as evidenced by our pass@1 results (also because of motivation # 1). Our partitioning strategy aims to create balanced subsets that cover different aspects. The additional comparisons in the general response showed that SPA is less biased than diversity of thought. However, we acknowledge that some components might specialize more in certain areas. In practice, a held-out dataset can be used to identify and remove overly biased components before deployment. Future research could explore weighted component selection, using a similar routing strategy in MOE.\n> Selection of M queries\n\nThe M questions don't necessarily need to be manually chosen. In our experiments, we used a semi-automated approach: we few-shot prompted ChatGPT to generate ~20 diverse queries and then manually selected 8 of them to cover a wide range of topics (Appendix C).\n\nWe appreciate your feedback and hope our clarifications have addressed your concerns about motivation and the empirical results.  If you have any remaining concerns, please let us know. We're grateful for the opportunity to further explain our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k2ir2Ue6Hp", "reviewer": "Reviewer_9mQS", "summary": "This paper presents a framework for eliciting diverse outputs from language models while maintaining quality/accuracy. The framework consists of: first partitioning a (synthetic) dataset of supervised instruction tuning data, then training parameter efficient model adapters on each partition, and finally at inference time sampling from each of the the models trained in the prior step.\n\nWithin this framework, the paper contributes empirical evaluations of 2 methods (influence functions and token overlap) for partitioning the instruction tuning datasets and measures the effects on both code completion and natural language generation benchmarks. They find that both methods generate more diverse output over random partitioning (i.e. an ensemble of models trained on random subsets of the data), suggesting that the method used to partition is important.\n\nThe authors contrast their method with temperature based sampling approaches to generating diverse data from an LLM.", "questions": "Experiment Design\n\n*   How were the 12 queries for each task generated? Could the authors shed any light on what they think important properties of this query set are?\n*   What are the generation params for Section 5.3? (is it all greedy sampling)\n*   Also in Section 5.3, why not compare with a single model at high temperature (Similar to the \"Single\" condition used for the code generation tasks)?\n\n*   Results\n*   How do the charts in figure 4 look like as temperature increases even further? Why did you stop at 0.5? The CodeLLama paper reports their main results at temperate=0.8 so it would be useful for readers to see if and where the methods converge in terms of diversity.\n*   Did you try using fewer than 8 model adapters? While there doesn't seem to be any much improvement observed in using more than 8 adapters in this setting, do you have a sense of how few adapters one could use?\n\n\n*   Metrics\n\n*   For the average KL divergence metric could you explain how you go from token level probability distributions for sequence level probability distribution for Pi and Pj? Or do you mean something else, I'm mostly trying to understand how you handle sequences of different lengths in the computation.\n*   For the sample diversity/Diversity score metric what value of K (line 273) is used in the experiments? And is the total number of samples generated in each condition the same?\n*   In figure 4 how many samples are generated for each method to calculate pass\\@k? And in the case of the SPA methods how is that distributed across the various adapters? My understanding is that there are 8 adapters in this experiment.\n*   Do the authors have any thoughts on why pass@1 isn't higher for SPA compared to single model?", "limitations": "Yes. Though the paper could also better acknowledge the extra computational cost at inference time (compared to just sampling multiple times at high temperature).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*   This paper recognizes certain diminishing returns of scaling up dataset size and instead finds alternative uses of large datasets to generate diverse outputs. This is neat approach to thinking about other ways to use \"scaled-up\" data.\n*   Demonstrates that the data partitioning approach does result in significant differences wrt diversity at lower temperatures. In particular the authors demonstrate that model ensembles of randomly partitioned data do not generate as diverse outputs as ones trained on partitions generated more principled approaches.\n*   The paper is overall well written and easy to read.", "weaknesses": "*    Evaluation on natural language (non-code) tasks lacks a quality/accuracy metric for the proposed method (Fig 5). While it is great to see that diversity is increased, the authors do state that maintaining quality is important but it is unclear if that is the case for these tasks. It would be helpful to see \"standard\" eval metrics for some or all of MMLU, BBH, GPQA and WinoGrande using the methods proposed.\n*   More clarity in the tradeoff the proposed methods make vs high temperature sampling would be helpful. At least for the code models, it appears that as temperature increases, the diversity of the single model might converge to that exibited by SPA. If that is the case some discussion of the tradeoffs involved, (e.g. benefits of being able to sample at low temperature) would strengthen the paper.\n*   It is difficult to know how sensitive the results are to the diversity of queries used to generate the partitioning matrix? It would be helpful to understand the performance of a partitioning method that is based on the data but doesn't use human selected test queries (e.g. randomly select 12 prompts from HumanEval and then partition based on that using influence and token overlap approaches). This is not however a fatal flaw by any means.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We'd like to address your main concerns.\n> Accuracy on NL tasks\n\nThe accuracy of SPA methods (influence-based, lexical-based, and random) all achieve similar accuracy to the single model, reaching the same conclusion as the pass@1 metric in Fig. 4a. We didn't observe improved pass@k with increased diversity on these tasks, likely because most are multiple-choice questions. For diversity measurement, we asked models to continue generating text even after producing an answer choice. We omitted accuracy results as they didn't provide new insights beyond the code generation tasks. However, we acknowledge that including this data would have provided a more complete picture and will add it to the revised paper.\n> Trade-off in high temperature sampling, temperature plot stops at 0.5\n\nYou're correct that at higher temperatures, the diversity of single models converges to SPA models. In practice, low temperature or even greedy sampling is often preferred when average sample quality is crucial. As shown in Fig. 4, high temperature leads to a decrease in the pass@1 metric, indicating a drop in sample quality. High temperature sampling may deviate from the learned distribution and produce hallucination or less coherent outputs [1]. We briefly discussed the disadvantages of high temperature in the introduction and will expand it in the revision. \n\nWe limited our plots to temperatures up to 0.5 as SPA's primary goal is to maintain average sample quality (measured by pass@1) while achieving diversity. Fig. 4a shows that temperature 0.5 already leads to a significant drop in pass@1. However, we acknowledge the importance of understanding where methods converge in terms of diversity. Our preliminary experiments indicate convergence around temperature 0.8. We will include these higher temperature results in our revision.\n> Number of adapters\n\nThe optimal number of adapters highly depends on the size of the synthetic dataset, with larger datasets typically inducing a higher optimal number of adapters. We observed improvements in diversity when increasing the number of adapters from 4 to 8, though this data was not included in Fig. 6,  suggesting that 8 adapters is roughly the sweet spot for the synthetic dataset we used (75k data points).\n> KL divergence, hyper-parameters, distribution over LoRA adapters\n\nWe measured the KL divergence at the first decoding step, which is at the token level. This measures the difference in states across different adaptations after seeing the same prompts. This is not applicable to the single model approach.\n\nFor each method (when temperature > 0), we sampled a total of 120 samples to calculate pass@1, pass@k, and diversity score (K=5). This larger sample size helps reduce metric variance. For SPA methods, we distributed samples evenly among adapters (15 samples per adapter). Future work could explore weighted sampling in SPA. The exception is greedy sampling (temp=0 in Fig. 4 and Sec. 5.3), where we only get one sample per adapter, leading to a total of 8 samples.\n> Why pass@1 is not higher, how to select queries.\n\nSPA is primarily designed to improve diversity, which doesn't necessarily improve average sample quality. Hence, it shows similar pass@1 performance to the single model.\n\nOur principle was to create a diverse query set. We used few-shot prompting with ChatGPT to generate ~20 queries, then manually selected 8 covering a wide range of topics.\n> Single model at higher temp in Sec. 5.3\n\nWe didn't include comparisons with single models at higher temperature because, as shown in Fig. 4a, higher temperatures lead to sample quality drops. However, we acknowledge that including this comparison could provide valuable insights. In our revision, we will add this comparison and discuss the trade-offs.\n\nWe appreciate your feedback and hope this clarifies your concern. Please let us know if there is any remaining concerns. \n\n\n[1]: Lee, M. (2023). A Mathematical Investigation of Hallucination and Creativity in GPT Models. Mathematics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QpAdtpO8b1", "reviewer": "Reviewer_4Daa", "summary": "This paper proposes a method to generated diverse responses for the language model. Given a synthetic finetuning dataset and test dataset, they calculate the importance of the finetuning data on each test data using methods such as influence function, and then partition the dataset into several smaller ones. Finally, they finetune several lora weights based on different partition datasets.", "questions": "How to find the optimal number of partition in practice? I suppose there should be a sweet point.\n\nCan authors report the actual time on calculating influence function and training the lora models for the experiments in this paper?\n\nIf you need multiple lora weights to generate diverse responses, are you able to still generate them in parallel without demanding memory requirement?", "limitations": ".", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors conducted experiments to verify that the proposed SPA based on influence function are validated to improve the diversity among baselines.", "weaknesses": "It involves finetuning, and need to finetune several lora weights. This would cause difficulty on parallel generation because we need to store multiple checkpoints into memory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We'd like to address your main concerns.\n> Parallel Generation and Memory Efficiency\n\nRecent advances in serving multiple LoRA adaptations in parallel significantly mitigate the need to store multiple full checkpoints. Notably, the S-LoRA system [1] demonstrates that it's possible to efficiently serve thousands of LoRA adapters with minimal overhead, using techniques like Unified Paging to manage memory efficiently. Furthermore, FLoRA [2] allows each input example in a minibatch to be associated with unique low-rank adaptation weights, enabling efficient batching of multiple LoRA adapters. In these systems, we only need to store the LoRA weights in memory, which are often low-rank, significantly reducing memory requirements. These advancements suggest that SPA can be implemented without significant memory constraints or parallel generation difficulties. As the field continues to progress, we anticipate even more efficient solutions for managing multiple LoRA adapters.\n\n[1]: Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J.E., & Stoica, I. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024\n\n[2]: Wen, Y., & Chaudhuri, S. (2023). Batched Low-Rank Adaptation of Foundation Models. ICLR 2024\n> Best number of partitions\n\nDetermining the optimal number of partitions is indeed an important consideration. This highly depends on the size of the synthetic dataset, with larger datasets typically inducing a higher optimal number of partitions. We conducted an ablation study (Fig. 6 in our paper) examining the effect of the number of partitions on diversity. We found that the diversity score remains relatively stable as the number of adaptations increases from 8 to 12, suggesting that 8 partitions is roughly the sweet spot for the synthetic dataset we used (75k data points). We also observed improvements in diversity when increasing the number of partitions from 4 to 8, though this data was not included in Fig. 6. \n> Computational time\n\nWe provided a rough complexity in the limitation section. To offer more details: One epoch of fine-tuning on the 75k synthetic data points takes approximately 8 hours using 2 A40 GPUs. We trained the LoRA weights for 4 epochs. Influence function calculation costs roughly 1 epoch of the training time, which is approximately 8 hours. We want to highlight that the main computational cost occurs during the offline partitioning and adaptation phases.\nFuture work could explore more efficient data attribution methods, such as TRAK [1] and K-FAC [2], which have the potential to substantially reduce this overhead while maintaining the benefits of our approach.\n\nWe appreciate your feedback and hope our clarifications have addressed your concerns about parallel generation and memory requirement. If you have any remaining concerns, please let us know. We're grateful for the opportunity to further explain our work.\n\n\n[1]: Park, S.M., Georgiev, K., Ilyas, A., Leclerc, G., & Madry, A. (2023). TRAK: Attributing Model Behavior at Scale. International Conference on Machine Learning. ICML 2023\n\n[2]: Grosse, R.B., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukovsiut.e, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., & Bowman, S. (2023). Studying Large Language Model Generalization with Influence Functions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable feedback. Here, we address the concern raised by reviewer j8SA regarding limited comparisons.

Our initial comparisons focused on ablations as SPA represents a novel multi-model adaptation method addressing diminishing returns of abundant synthetic data. Unlike most methods that modify the sampling process, SPA operates at the model level, making it complementary to existing methods.
To provide more context, we've conducted additional experiments comparing SPA with KL-divergence guided temperature sampling [1] and Diversity of Thought [2]. 

For KL-guided temperature sampling, following [1], we maintained two parallel decoding sequences: one with full information and another identical but without the docstring (mainly test cases). At each decoding step, we calculated the KL divergence between the logits of these two sequences' forward passes, using it to rescale the temperature. The newly sampled token was then added to both sequences.

We iterated hyperparameters $T_0 = \\{0.1, 0.2, …, 0.5, 0.8, 1.0\\}$ and $\sigma = \\{1e-3, 1e-2, 0.1, 1, 10\\}$, reporting the best runs for pass@1 and pass@5 separately.  Notably, no combination achieved both high pass@1 and pass@5 simultaneously like SPA did. The results show that KL-guided temperature scaling generates diverse samples but at the cost of average sample quality. 

For the DIV-SE, following Fig.5 in the paper, we generated 8 reasoning approach templates for HumanEval problems. We only used 3 of them (Iterative, BruteForce, Greedy) in the final evaluation due to poor pass@1 performance of the others. This is because the reasoning templates don't generalize well to the diverse coding challenges in HumanEval though they work well in the math domain. The result shows that SPA is better in terms of all metrics when sampled at different temperatures.

These comparisons further show SPA's advantage in sampling from multiple distinct distributions.
| HumanEval   | pass@1 | pass@5 | Diversity | 
|----------------|----------|------------|---------------|
|  DIV-SE temp 0.1  | 46.78   |  58.70     |   0.72     |
| Influence-SPA temp 0.1 | 49.80  |  69.59     |  0.85    |
|
|  DIV-SE temp 0.2  | 45.61   |  61.70     |   0.79     |
| Influence-SPA temp 0.2 | 48.74  |  69.82     |  0.86    |
|
|  DIV-SE temp 0.3  | 44.57   |  62.01     |   0.84     |
| Influence-SPA temp 0.3 | 47.64  |  70.89     |  0.88    |
|
|  DIV-SE temp 0.4  | 43.63   |  63.34     |   0.88     |
| Influence-SPA temp 0.4 | 46.97  |  71.91     |  0.91    |
|
|  DIV-SE temp 0.5  | 42.01   |  64.36     |   0.91     |
| Influence-SPA temp 0.5 | 46.82  |  72.49     |  0.92    |
|
| KL-guided temp 0.1 | 49.24 | 60.84 |  0.54 |
| KL-guided temp 0.8 | 42.17 | 66.63 |  0.93 |



[1] Chang, C., Reitter, D., Aksitov, R., & Sung, Y. (2023). KL-Divergence Guided Temperature Sampling. ArXiv, abs/2306.01286.


[2] Naik, R., Chandrasekaran, V., Yuksekgonul, M., Palangi, H., & Nushi, B. (2023). Diversity of Thought Improves Reasoning Abilities of LLMs. ArXiv, abs/2310.07088

Meta-review: This paper addresses the critical challenge of generating diverse responses from foundation models while maintaining output quality. The proposed Synthesize-Partition-Adapt (SPA) framework offers a novel approach by leveraging synthetic data and model adaptations, distinguishing itself from methods that solely modify the sampling process. The experimental results, particularly in code generation tasks, demonstrate the effectiveness of SPA in improving diversity metrics without sacrificing performance. While concerns were raised about comparisons to existing baselines and the practicality of serving multiple LoRA weights, the authors have adequately addressed most of these issues in their rebuttal (including the benchmarks for practical serving of LoRA weights based on recent papers during the rebuttal phase). The importance of the problem and the innovative nature of the solution outweigh the remaining concerns, making this paper a valuable contribution to the field. Therefore, the paper is recommended for acceptance to NeurIPS 2024. 
The authors are encouraged to consider the feedback of the reviewers in finalizing their paper, including empirical or analytical comparisons with state-of-the-art diverse sampling methods to provide a more comprehensive evaluation of SPA's performance, providing more insight into the scalability and real-world implementation of serving multiple LoRA weights, as well as considering writing improvements starting with the introduction, summarizing the method more concisely and moving key figures earlier in the paper for better reader engagement. In the future, authors might also want to explore the combination of SPA with in-context learning approaches to potentially further enhance diversity.
Decision: Accept (poster)