Synthesize, Partition, then Adapt: Eliciting Diverse Samples from Foundation Models

Authors: yeming  wen
swarat  chaudhuri
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:15:32

Abstract: Presenting users with diverse responses from foundation models is crucial for enhancing user experience and accommodating varying preferences. 
However, generating multiple high-quality and diverse responses without sacrificing accuracy remains a challenge, especially when using greedy sampling. 
In this work, we propose a novel framework, Synthesize-Partition-Adapt (SPA), that leverages the abundant synthetic data available in many domains to elicit diverse responses from foundation models.
By leveraging signal provided by data attribution methods such as influence functions, SPA partitions data into subsets, each targeting unique aspects of the data, and trains multiple model adaptations optimized for these subsets.
Experimental results demonstrate the effectiveness of our approach in diversifying foundation model responses while maintaining high quality, showcased through the HumanEval and MBPP tasks in the code generation domain and several tasks in the natural language understanding domain, highlighting its potential to enrich user experience across various applications.

Main paper: 
 instruction fine-tuning: 
 By fine-tuning foundation models on human-annotated data that demonstrates desired behaviors, instruction tuning aims to improve the alignment between the model's outputs and the user's intentions (Ouyang et al., 2022; Wei et al., 2021; Sanh et al., 2022) . Let D = (x i , y i ) N i=1 denote a dataset of input-output pairs, where x i represents the input instruction and y i represents the corresponding desired output. The objective of instruction tuning is to minimize the following loss function:L(θ) = -1 N N i=1 log θ (y i |x i )where θ represents the parameters of the foundation model, and p θ (y i |x i ) is the probability of generating the target response y i given the input x i .Classical approaches for instruction tuning typically require a substantial amount of parallel labeled data of NL intents and gold model responses. Collecting large-scale, high-quality annotated datasets is often time-consuming and expensive. To mitigate this issue, researchers have explored the use of synthetic data for instruction tuning. By leveraging techniques such as data augmentation (Wei & Zou, 2019; Sennrich et al., 2016) and back-translation (Edunov et al., 2018) , synthetic data can be generated at scale, providing a cost-effective alternative to human-annotated datasets. Furthermore, synthetic instruction-following data can also be generated from the foundation model itself (Wang et al., 2022a; Honovich et al., 2022; Taori et al., 2023; Peng et al., 2023; Wen et al., 2024, inter alia) .Data attribution methods aim to quantify the importance or influence of individual training points on a model's predictions. One such method is the influence function (Koh & Liang, 2017) . Formally, let L(θ) denote the loss function of the model, where θ represents the model parameters. The influence of a training point z on the model's parameters θ is given by I(z) = -H -1 θ ∇ θ L(z, θ). where H θ is the Hessian matrix of the loss function with respect to the model parameters, and ∇ θ L(z, θ) is the gradient of the loss function with respect to the model parameters, evaluated at the training point z. Next, the influence of elevating the weight of z on the loss associated with a test point z test is:EQUATIONIt is impossible to calculate the full Hessian H -1 θ matrix in deep neural networks. Koh & Liang (2017) developed a simple and efficient implementation that requires only oracle access to gradients and Hessian-vector products. This implementation makes it feasible to apply influence function to large-scale models. However, the vast parameter space of foundation models presents an even greater challenge, rendering the direct application of influence function impractical. In response to this, recent advancements in Grosse et al. ( 2023) have further refined the methodology, enabling the application of influence function to large language models. Pass@1 on HumanEval 
problem formulation: 
 Figure 2 : pass@1 on HumanEval after fine-tuning on some percentage of OSS-Instruct dataset (Wei et al., 2023) using LORA. The plot demonstrates the diminishing returns observed with increasing amounts of data used for parameter efficient fine-tuning.Given a user input x, our goal is to generate a diverse set of high-quality responses y 1 , y 2 , ..., y K from a foundation model M. One approach to generating diverse responses is to sample from the model multiple times using techniques like temperature sampling:y k = M(x; θ, τ ),where k = 1, 2, ..., K and θ represents the model parameters and τ is the temperature hyperparameter. However, this approach often trades off diversity for quality as studied in Chung et al. (2023) . An alternative approach is to train multiple model adaptations M 1 , M 2 , ..., M K and sample one response from each adaptation:EQUATIONwhere θ k represents the parameters of the k-th model adaptation. By training each adaptation on a different subset of the data that captures unique aspects and yields distinct model behaviors, we can generate diverse responses while maintaining their quality. Moreover, this approach allows us to elicit diverse samples even with greedy sampling, which is often preferred for maximum precision.Traditionally, training multiple model adaptations has been considered unfavorable due to the repeated training process, which can be computationally expensive and time-consuming. However, with the increasing popularity of instruction tuning, it has become common practice to go through a post-training stage using instruction data before deploying the model to users. This post-training stage presents an opportunity to train multiple model adaptations without incurring significant additional costs, making the approach more feasible and practical in real-world scenarios.As the volume of synthetic data grows, the utility of fine-tuning a single model on the entire dataset diminishes due to the diminishing returns in the post-training stage, as demonstrated in Fig. 2 . The pass@1 accuracy after fine-tuning on the entire synthetic dataset using LORA is roughly the same as only consuming 15% of the data2 . This creates an opportunity to leverage the abundant synthetic data to train multiple model adaptations, each specializing in a specific subset of the data. In this work, we propose the Synthesize, Partition, then Adapt (SPA) framework to address the diverse response generation problem. SPA leverages existing synthetic datasets, data partitioning techniques, and parameter-efficient fine-tuning methods to train multiple model adaptations. By sampling from the collection of these adaptations, SPA generates diverse and high-quality responses, enhancing the overall user experience.Generate a Python function to check if a number is even.  
partitioning synthetic data and training adaptations: 
 We present the technical details of our proposed SPA framework for training multiple adaptations. We leverage an existing synthetic dataset D = {x i , y i } N i=1 for the purpose of this study. The use of an existing synthetic dataset allows us to focus on the effectiveness of the Partition then Adapt steps in eliciting diverse samples, while demonstrating the flexibility of our framework to work with various synthetic datasets. Fig. 3 provides an overview of the framework. After obtaining the synthetic data, our approach consists of three main steps: (1) computing data attribution scores for synthetic data points, (2) partitioning the synthetic dataset based on these scores, and (3) training multiple foundation model adaptations using parameter-efficient fine-tuning techniques like LORA.Consider a pre-trained foundation model M with parameters θ. Our goal is to leverage the synthetic dataset D to train a set of K foundation model adaptations {M k } K k=1 . Each adaptation focuses on a specific subset of the data that yields similar model behaviors. To partition the synthetic dataset, we employ data attribution methods that measure the importance of each training point to the model's predictions. Although we use influence function as an example to label the data, the SPA framework is not limited to influence function and can be extended to other data attribution methods, such as lexical overlap or TRAK (Park et al., 2023) . To calculate the influence function, we first fine-tune the pre-trained foundation model M on the synthetic dataset D. The fine-tuning process optimizes the model parameters θ to minimize the loss function L(θ) on the synthetic dataset using LORA (Hu et al., 2021) :L(θ) = 1 N N i=1 ℓ(y i , M(x i ; θ)), where ℓ(•, •) is a suitable loss function, such as cross-entropy loss for language modeling tasks. This fine-tuning process yields the optimized model parameters θ.Next, we select a set of M test queries {(x(m) t , y (m) t )} Mm=1 , which can be a collection of questions requiring various expertise knowledge to solve. For each test query (x(m) t , y (m) t), we compute the influence score of each synthetic data point (x i , y i ) ∈ D using Eq. (1):EQUATIONTo efficiently compute the influence scores, we employ the stochastic estimation method proposed by Koh & Liang (2017) , which approximates the inverse Hessian-vector product using conjugate gradients. Although even this method is generally infeasible in foundation models due to their vast parameter space, the use of LORA (Hu et al., 2021) makes it feasible by significantly reducing the number of trainable parameters. The computational cost of estimating the influence of a test query between the entire dataset D is the same as calculating the gradient of D. Another option to address this issue is to use the K-FAC approximation of the Hessian, as proposed by Grosse et al. (2023) . We focus on the LORA approach and leave the exploration of K-FAC and other approximations for future work.After computing the data attribution scores for each synthetic data point with respect to the M test points, we obtain an influence matrix I ∈ R N ×M , where I i,m represents the attribution score of the i-th synthetic data point for the m-th test point. To partition the synthetic dataset D into K subsets {D k } K k=1 , a clustering algorithm can be applied to solve the following objective:EQUATIONwhere I i,: denotes the i-th row of the influence matrix I, subject toK k=1 D k = D and D k ∩ D k ′ = ∅ for all k ̸ = k ′ .In this work, we assume partitions are disjoint for the simplicity of the study.The clustering algorithm assigns each synthetic data point (x i , y i ) to one of the K subsets based on the similarity of its influence scores across the M test points. This partitioning ensures that data points within each subset have similar impacts on the model's predictions. The choice of the clustering algorithm may depend on the specific characteristics of the dataset. For simplicity and ease of implementation, in this study, we use a ranking heuristic to partition the synthetic dataset. The details of this heuristic will be explained in the experiment section §5.1. However, it is important to note that our SPA framework is not limited to any specific clustering algorithm.Once the synthetic dataset is partitioned into K subsets, we train a foundation model M k for each subset D k using parameter-efficient fine-tuning techniques like LORA (Hu et al., 2021) . LORA adapts the pre-trained foundation model parameters θ by learning low-rank matrices A k ∈ R r×d and B k ∈ R d×r for each weight matrix W ∈ R d×d in the pre-trained foundation model, where r ≪ d is the rank of the adaptation matrices.The adapted weight matrix W k for the foundation model adaptation M k is computed as:W k = W + B k A k .During the fine-tuning process, only the adaptation matrices A k and B k are learned, while the pre-trained weights W remain frozen. This significantly reduces the number of trainable parameters, making it feasible to train multiple foundation model adaptations with limited computational resources. The training objective for each foundation model adaptation M k is given by minθ k 1 |D k | (xi,yi)∈D k ℓ(y i , M k (x i ; θ k ))where θ k represents the parameters of M k , which include the pre-trained weights θ and the LoRA adaptation matrices A k , B k . By training multiple foundation model adaptations using LORA, we can efficiently adapt the pre-trained foundation model to different subsets of the synthetic data, each focusing on a specific aspect of the data that yields similar model behaviors. This approach enables the creation of a diverse set of specialized models that capture different knowledge or expertise present in the synthetic data, while leveraging the knowledge acquired during the pre-training phase.Inference with Multiple Adaptations During inference, given a user input x, our goal is to generate a diverse set of responses by leveraging the multiple foundation model adaptations trained on different subsets of the synthetic data. To achieve this, we randomly sample a foundation model adaptation M k from the set of K adaptations {M k } K k=1 and generate the output y using the selected adaptation. By randomly sampling from the set of adaptations, we can generate a diverse set of responses for the user input x. This approach ensures that the generated responses are not only diverse but also maintain reasonable quality. It is worth noting that this approach is compatible with various sampling techniques, such as temperature scaling, top-k and top-p sampling, which can further enhance the diversity of the generated responses.To generate multiple diverse responses for the user input x, we can repeat the random sampling process multiple times, each time selecting a different adaptation and generating a response. This allows us to present the user with a set of alternative responses that capture different perspectives or styles, enhancing the overall user experience. Unlike temperature sampling, which can degrade the quality of the generated responses, our approach maintains the quality of each response by leveraging the specialized knowledge captured by each adaptation. Moreover, our approach can generate diverse samples even when greedy sampling is used. 
experiments: 
 In this section, we present the experimental setup and results for evaluating the effectiveness of our proposed SPA framework in improving the diversity of foundation model outputs. We conduct experiments on both code generation tasks such as HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) and several natural language understanding tasks.Base Model and Synthetic Dataset For the code generation experiments, we use CodeLLaMA 7B (Rozière et al., 2023) as the base foundation model. CodeLLaMA is a state-of-the-art language model specifically designed for code-related tasks, pre-trained on a large corpus of code and natural language data. For the synthetic dataset, we utilize the OSS-Instruct dataset (Wei et al., 2023) , which consists of 75,000 code-related question-answering pairs generated by GPT-3.5 Turbo (Ope-nAI, 2023) . In the natural language understanding domain, we employ Llama-2 13B (Touvron et al., 2023) as the base foundation model. Llama-2 is a powerful language model trained on a diverse range of web-scale data, demonstrating strong performance across various natural language understanding tasks. For the synthetic dataset, we use Platypus (Lee et al., 2023) , which focuses on improving LLMs' STEM and logic knowledge. Platypus consists of a curated sub-selection of public text datasets, comprising approximately 25,000 question-answer pairs. Data Attribution Scores We compare two methods for computing data attribution scores: influence function and lexical overlap.For the influence-based method, we hand-write 12 examples that cover a wide range of knowledge for each domain. For each of these examples, we calculate the influence score with respect to each training example in the corresponding synthetic dataset using Equation 3. We then select the top 8 test queries whose distribution of influence scores over the dataset has the highest variance. This ensures that the selected test queries have diverse impacts on the synthetic dataset, capturing different aspects of the domain knowledge. The resulting influence matrices I code ∈ R 8×75,000 and I nlu ∈ R 8×25,000 are used for partitioning the OSS-Instruct and Platypus datasets, respectively.For the lexical overlap method, we compute the BM25 score (Robertson et al., 1994) between each training example and the hand-written test queries. The BM25 score is calculated as follows:EQUATIONwhere f (z, t) is the overlap count, N is the number of training examples, L(z) is the length of the example, and L avg is the average example length. We adopted the framework and the hyperparameters in Lv & Zhai (2011) . While we focus on influence function in this work, exploring the effectiveness of alternative data attribution methods like BM25 could be an interesting direction for future research. More details are provided in Appendix A.Partitioning the Synthetic Datasets To train multiple foundation model adaptations, we first set the hyperparameter K, which represents the total number of adaptations. We use K = 8 for both code generation and natural langauge understanding domain. For each data point in the synthetic dataset, we aim to find the test queries that provides the most influence. Formally, for each synthetic data point (x i , y i ), we assign it to the subset D * k corresponding to the test point with the highest influence score or the BM25 score: k * = arg max k∈{1,...,K} I k,i . where I k,i represents either the influence matrix or the BM25 score matrix. This process partitions the OSS-Instruct dataset Methods HUMANEVAL MBPP pass@1 pass@5 diversity avg. KL pass@1 pass@5 diversity avg. 1 : Results on the HumanEval and MBPP. τ denotes the temperature used for sampling. SPA with influence function achieves the best performance in terms of diversity score and avg. KL divergence) while maintaining comparable pass@1 performance to the single adaptation baseline. pass@5 measures sample quality but also has a positive correlation with diversity.into K groups for code generation and the Platypus dataset into K groups for natural language understanding. Each group is associated with a specific test example that has the highest influence on the data points within the group.With the partitioned synthetic dataset, we train K model adaptations using the LORA technique, as described in §4.3. Each adaptation M k is trained on the corresponding subset D k of the synthetic dataset, focusing on the specific coding knowledge captured by the associated test point.Evaluation Metrics We use the following two metrics to assess the diversity:1. Average KL Divergence: Let P i and P j be the probability distributions of the generated responses from two model adaptations i and j, respectively. The KL divergence between P i and P j is defined as D KL (P i ∥ P j ) = x P i (x) log Pi(x) Pj (x) . The average KL divergence is calculated by averaging the pairwise KL divergence between all possible pairs of model adaptations. A higher average KL divergence indicates greater diversity among the model adaptations,EQUATION2. Sample Diversity: The average KL divergence evaluates the diversity at the distributional level.We also consider the sample diversity which measures the uniqueness of individual responses. We calculate the diversity score among K randomly generated samples for each problem. The diversity score is defined as the proportion of unique samples within the generated set. Specifically, it is calculated by taking one minus the ratio of the number of duplicate pairs to the total number of generated pairs.Baselines We consider two baselines in the evaluation: (1) Single Adaptation, where a single model adaptation is trained on the entire synthetic dataset using LORA, and (2) Multiple Adaptations (random), where multiple adaptations are trained on randomly partitioned subsets of the synthetic dataset using LORA. Hyperparameters used to train adaptations are provided in Appendix A.In the code generation domain, we evaluate the performance of our proposed methodology on two popular code generation benchmarks: HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) . HumanEval consists of 164 hand-written programming problems with corresponding test cases, while MBPP contains 399 held-out programming problems collected from online resources3 . These benchmarks assess the ability to generate functionally correct code.pass@k metric In addition to the diversity metrics, we also evaluate the sample quality by pass@1 and pass@5, measuring the percentage of problems for which at least one of the k generated samples passes all the test cases. Note that the pass@5 metric has a strong correlation to the diversity of the samples. More diverse samples generally lead to higher pass@5 for k > 1.Tab. 1 presents the evaluation results of our SPA framework and the baselines on the HumanEval and MBPP benchmarks. For the multiple adaptation methods, including random partitioning, lexical overlap, and influence function, we use greedy decoding (τ = 0) to generate samples. For the single adaptation baseline, we use a temperature of τ = 0.1 to induce some diversity in the generated samples, as greedy decoding would not produce any diversity in this case.Our primary focus is on comparing the diversity metrics, namely the Diversity Score and the Average KL Divergence (avg. KL), across the different methods. SPA with influence function achieves the highest Diversity Scores of 85% and 78% on HumanEval and MBPP, respectively, indicating that the generated samples are more unique and diverse compared to the other methods. Similarly, SPA with influence function yields the highest Average KL Divergence of 0.017 and 0.020 on HumanEval and MBPP, demonstrating greater diversity at the distributional level.The random partitioning and lexical overlap approaches also improve upon the single adaptation baseline in terms of diversity metrics, but to a lesser extent than influence function. In particular, the lexical overlap induces more diversity than the random adaptations baseline. This suggests that even simpler data attribution methods can be beneficial for enhancing diversity when training multiple specialized adaptations. It is worth noting that the pass@5 scores, while primarily measuring sample quality, also have a positive correlation with diversity. SPA with influence function achieves the highest pass@5 scores of 69.05% and 73.68% on HumanEval and MBPP, indicating that the generated samples not only exhibit greater diversity but also maintain high quality.In summary, these results underscore the effectiveness of our SPA framework in generating diverse code samples without compromising quality. By leveraging influence function for data partitioning and training multiple adaptations using LORA, SPA enables the generation of diverse and accurate code solutions, even when using greedy decoding. We also showed that training more adaptations than 8 did not lead to more diversity in Appendix B.Impact of Temperature Fig. 4 presents the impact of temperature on pass@1, pass@5, and Diversity Score for different methods on the HumanEval benchmark. The first plot shows that all methods, including Single, Random, Lexical, and Influence, exhibit similar patterns in terms of pass@1 performance. They achieve maximum accuracy (around 50.2%) when τ = 0 and gradually decrease to approximately 46.5% as the temperature increases to 0.5.However, both pass@5 and Diversity Score improve for all methods as the temperature increases, which is expected as higher temperatures encourage the model to generate more diverse samples. Notably, SPA with influence function (Influence) maintains its advantage over other methods across all temperature values, outperforming Single, Random, and Lexical methods. Although the performance gap between Influence and other methods narrows as the temperature increases due to the inherent diversity promotion of higher temperatures, Influence still maintains a lead at τ = 0.5.To demonstrate the effectiveness of SPA in the natural language understanding domain, we evaluate its performance on several diverse tasks, including Big-Bench Hard (BBH) (Suzgun et al., 2022) , GPQA (Rein et al., 2023) , MMLU (Hendrycks et al., 2020) , and WinoGrande (Sakaguchi et al., 2019) . For tasks that involve multiple-choice questions, we asked the model to continue generating text even after producing an answer choice for the purpose of measuring sample diversity. As shown in Fig. 5 , SPA with influence function consistently achieves higher diversity scores and average KL divergence compared to the lexical overlap and random adaptation across all tasks. Interestingly, random adaptations achieve better diversity than lexical overlap on the GPQA task, suggesting that the effectiveness of partitioning methods may change depending on the task.B B H G P Q A M M L U W in o GThe diversity scores and average KL divergence values vary across tasks, reflecting the inherent differences in the nature and complexity of each task. Tasks like MMLU, which cover a wide range of subjects, tend to yield higher average KL divergence. We also notice that a larger gap in average KL divergence does not necessarily translate to a proportionally greater difference in diversity scores. This suggests that while average KL divergence captures the dissimilarity between the generated sample distributions, it may not always directly correlate with the actual diversity of the samples. Nonetheless, the consistent improvement achieved by SPA with influence function highlights its robustness and adaptability to various natural language understanding challenges. 
related work: 
 Sampling-based methods have been widely explored to generate diverse text from language models. One of the most common approaches is temperature sampling (Ackley et al., 1985; Hinton et al., 2015) . Several studies have investigated the impact of temperature on model sampling and its effect on the diversity-quality trade-off (Caccia et al., 2018; Renze & Guven, 2024; Wang et al., 2023) .Higher temperatures lead to more diverse but potentially less coherent samples, while lower temperatures produce more conservative and deterministic outputs. When using high temperatures, human interventions can help to correct errors during the sampling process (Chung et al., 2023) . Dynamic temperature strategies have also been explored during the model training and inference stages (Lin et al., 2018; Zhang et al., 2018; Wang et al., 2019; Chang et al., 2023) .Besides adjusting temperature, top-k, top-p (nucleus) sampling (Holtzman et al., 2019) and their variants are common sampling methods (Fan et al., 2018; Meister et al., 2022; Hewitt et al., 2022; Ravfogel et al., 2023) , which restrict the sampling space or dynamically adjust the number of tokens considered at each step. Another line of works studied how to formulate quality-diversity trade-off as a search or RL problem (Naik et al., 2023; Lim et al., 2024; Mudgal et al., 2023; Bradley et al., 2023; Ji et al., 2023) . 
conclusion: 
 In summary, we proposed SPA, which that leverages synthetic data, data partitioning, and model adaptation to elicit diverse responses from foundation models. By partitioning synthetic datasets into subsets that capture unique aspects of the data and training multiple model adaptations optimized for these subsets, SPA enables the generation of diverse and high-quality responses.Limitation One main challenges is the computational cost associated with influence function, which require several extra epochs of backward passes to estimate. Future work could explore more efficient data attribution methods, such as TRAK (Park et al., 2023) and K-FAC (Grosse et al., 2023) .Additionally, the ranking heuristics used to approximate Eq. ( 4) can be replaced by more advanced clustering algorithms. Additionally, serving multiple LoRA adaptations poses significant computational challenge in real-time serving framework. Recent works such as S-LoRA and FLoRA (Sheng et al., 2023; Wen & Chaudhuri, 2024 ) can be considered to accommodate this overhead.After obtaining the data attribution matrix, we observe that using the ranking heuristic presented in §5.1 leads to imbalanced partitions. To achieve more balanced partitions, we normalize the data attribution matrix before applying the heuristics. We leave the exploration of more advanced clustering algorithms, such as k-means, for future work. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: No violation.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work is foundational research on how to generate diverse samples. It is not directly applied in any product.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We use existing dataset and models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the used datasets, models and the repos.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
give me a personal website template: 
 Response #1Response #2 Transformer-based foundation models have revolutionized the fields of natural language processing (NLP) and code generation with their remarkable abilities a wide range of understanding and generation tasks (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020; Chen et al., 2021) . These models are typically pre-trained on vast amounts of text data and then undergo instruction fine-tuning -a post-training process -to improve alignment with user expectations and enhance the overall user experience (Ouyang et al., 2022) . Due to the high cost of human-annotated data, synthetically generated datasets (Wang et al., 2022b) such as OSS-Instruct (Wei et al., 2023) and Alpaca (Taori et al., 2023) have become an important component of instruction tuning, demonstrating strong effectiveness in improving foundation model performance.PromptTo date, these synthetic datasets have been primarily used to align foundation models with instructions or to induce certain preferable behaviors. In this paper, we focus on a different use of synthetic data: in improving the diversity of foundation models' outputs. Diversifying the generated responses is crucial for accommodating diverse user preferences and enhancing user satisfaction. Consider the scenario illustrated in Fig. 1 , where a user prompts a foundation model with "Give me a personal website template". In this case, we would prefer the model to generate two diverse templates while 38th Conference on Neural Information Processing Systems (NeurIPS 2024).maintaining good quality, providing users with a variety of styles and layouts. Conventional methods for improving diversity, such as temperature sampling (Ackley et al., 1985; Hinton et al., 2015; Wang et al., 2019 Wang et al., , 2023)) , rely on sampling techniques that anneal the probabilistic distribution of outputs. These methods often trade off diversity for quality, as the generated responses may deviate from the learned distribution and produce hallucination or less coherent outputs (Lee, 2023) . Moreover, these techniques are not applicable when using greedy sampling, which is often preferred for its simplicity and precision. This highlights the need for approaches that not only align foundation model outputs with user expectations but also elicit diverse responses without sacrificing quality.In this paper, we present a framework, Synthesize-Partition-Adapt (SPA), that achieves these objectives. The framework partitions the synthetic data and adapts foundation models to these partitions in the post-training stage. By leveraging the inherent diversity in the training data, this approach can generate diverse responses without compromising accuracy. The potential of partition-and-adapt approach is further amplified by the increasing availability of large-scale synthetic datasets because the utility of instruction-tuning a single model on the entire dataset diminishes. In particular, we show that influence function (Koh & Liang, 2017) can be an effective signal to partition synthetic datasets into subsets, each targeting unique aspects that elicit distinct model behaviors. However, SPA is not limited to influence function and can be extended to other partitioning strategies. By training multiple adaptations on these subsets using parameter-efficient fine-tuning techniques, such as LoRA (Hu et al., 2021) , we enable the generation of diverse and accurate responses.To demonstrate the effectiveness of our approach, we conduct experiments on a range of tasks in both the code generation and natural language understanding domains. We evaluate our method on the HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) datasets for code generation, as well as several natural language understanding tasks. The results showcase the ability of our approach to diversify model responses while maintaining high accuracy, highlighting its potential to enrich user experience across various applications.To summarize, the main contributions of this paper are as follows:• We propose SPA, a novel framework that leverages synthetic data, data partitioning, and model adaptation to elicit diverse responses from foundation models.• We demonstrate the effectiveness of SPA in diversifying foundation model responses while maintaining sampling quality through extensive experiments on code generation and natural language understanding tasks.• We highlight the potential of SPA to leverage the increasing availability of large-scale synthetic datasets for improving the diversity of foundation model responses. 
 
 Appendix: In this section, we provide details on the computing resources and hyperparameters used for training the model adaptations in both the code generation and natural language understanding domains. For the code generation experiments, we use a machine with 3 A100 40GB GPUs and train each partition for 400 steps, which takes approximately 80 minutes (each partition). The hyperparameters are mostly adopted from https://github.com/ bigcode-project/starcoder/tree/main. The base model is CodeLLaMA-7B-Python, and we use bf16 precision to accelerate training. The per-device train batch size is set to 1, with a gradient accumulation step of 20. We use a learning rate of 2e-4 with a cosine learning rate scheduler and 20 warmup steps. For the LORA hyperparameters, we use a rank (r) of 16, an alpha of 16. In the natural language understanding domain, we train each partition for 400 steps, which takes approximately 40 minutes, using the Llama-2 13B model as the base model. The training time is shorter compared to the code generation domain because the Platypus dataset is much smaller than OSS-Instruct. The hyperparameters are mostly adopted from https://github.com/ arielnlee/Platypus. We use a per-device batch size of 1 and a gradient accumulation step of 4. The learning rate is set to 1e-4, with a total of 20 warmup steps.All the computational costs mentioned in this section, including the time and resources required for computing data attribution scores and training model adaptations, are offline. These costs are incurred before deploying the models to users, and they do not affect the inference time.In this section, we investigate the impact of the number of model adaptations on the diversity of the generated responses. We focus on the HumanEval benchmark in the code generation domain and vary the number of adaptations from 8 to 12. The results are presented in Fig. 6 .As shown in Fig. 6 , the diversity score remains relatively stable as the number of adaptations increases from 8 to 12, regardless of the partitioning method used. These results suggest that increasing the number of adaptations beyond a certain point may not necessarily lead to an improvement in the diversity of the generated responses.In this appendix, we provide the hand-written test queries used in our experiments for both the code generation and text generation domains. These examples were utilized to compute data attribution scores. Most of the examples are generated by GPT-4 (OpenAI, 2023) . for length in range(2, n+1):13 for i in range(n-length+1): 38 Response: The the Great Mortality drastically reduced the population of Europe, leading to severe labor shortages, higher wages, lower prices for land, and a shift in economic power from the feudal lords to the working class and merchants. Query: Discuss the main differences between utilitarianism and deontological ethics.14 j = i + length -1 15 if s[i] == s[j] and length == 2: 16 dp[i][j] = 2 17 elif s[i] == s[j]: 18 dp[i][j] = dp[i+1][j-1] + 2 19 else: 20 dp[i][j] = max(dp[i+1][j], dp[i][j-1]) 21 22 return dp[0][n-1]fib[i] = fib[i -1] + fib[i -2]Response: Utilitarianism focuses on the outcomes or consequences of actions to determine morality, while deontological ethics considers the actions themselves and the adherence to duties or rules as the basis for morality. Response: Food in Japan is deeply intertwined with the nation's culture, reflecting aspects of beauty, seasonality, and regional diversity. Traditional dishes like sushi and bento embody aesthetic principles and social customs, such as respect for nature and meticulous attention to detail. Food rituals, such as tea ceremonies, also highlight the importance of mindfulness and harmony in Japanese society. Response: Globalization has both positive and negative economic effects on developing countries. On the positive side, it allows access to international markets, increases capital inflow, and promotes technology transfer, leading to job creation and economic growth. However, it can also lead to economic dependency, cultural homogenization, and the potential exploitation of local resources and labor, which might exacerbate inequalities and social tensions. Justification: The information is given in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).* ywen@utexas.eduThis does not suggest full parameter fine-tuning shares the same diminishing return.We used the evalplus(Liu et al., 2023) framework to evaluate samples.dp = [[0] * n for _ in range(n)] 


Decision: This paper addresses the critical challenge of generating diverse responses from foundation models while maintaining output quality. The proposed Synthesize-Partition-Adapt (SPA) framework offers a novel approach by leveraging synthetic data and model adaptations, distinguishing itself from methods that solely modify the sampling process. The experimental results, particularly in code generation tasks, demonstrate the effectiveness of SPA in improving diversity metrics without sacrificing performance. While concerns were raised about comparisons to existing baselines and the practicality of serving multiple LoRA weights, the authors have adequately addressed most of these issues in their rebuttal (including the benchmarks for practical serving of LoRA weights based on recent papers during the rebuttal phase). The importance of the problem and the innovative nature of the solution outweigh the remaining concerns, making this paper a valuable contribution to the field. Therefore, the paper is recommended for acceptance to NeurIPS 2024. 
The authors are encouraged to consider the feedback of the reviewers in finalizing their paper, including empirical or analytical comparisons with state-of-the-art diverse sampling methods to provide a more comprehensive evaluation of SPA's performance, providing more insight into the scalability and real-world implementation of serving multiple LoRA weights, as well as considering writing improvements starting with the introduction, summarizing the method more concisely and moving key figures earlier in the paper for better reader engagement. In the future, authors might also want to explore the combination of SPA with in-context learning approaches to potentially further enhance diversity.

Reviews: review: paper_summary: paper_summary: This paper aims to improve the diversity of generations of LLMs. Current methods are not great because when they offer more diversity, they lose in quality (for instance worse performance). This is for instance the case of temperature sampling. In this paper, they propose synthetise partition and adapt which consists in taking a synthetic dataset, partitioning it using influence function for instance and a few high quality test examplars, train a multiple models on each of these subsets and then eventually, use this collection of models to do sampling. They show that thanks to their approach, they are able to get better diversity while maintaining good performance on code and MMLU.
strength_weakness: strength_weakness: I want to raise a few points regarding the approach:

- **In-context approach?**: I think that one criticism could be that he method may seem a bit computationally heavy (because of the data clustering step + training many models), I know that one of the most commonly used methods in practice to ensure diversity is to select different text chunks and use them to seed the prompt and get a different generation. This was used for instance in the Phi and cosmopedia approaches to generate synthetic textbooks [1]. In other words, do you think that the finetuning step is necessary? Wouldn't putting one of the clusters of data (that you create with data attribution) **In-context** be enough?


- **Role of the rank of LoRA**: I imagine that the rank of LoRA may play a role here? Intuitively, if the rank is very small, all the models will have similar representations even after fine-tuning. As you increase the rank of LoRa, you may increase the diversity of your generations. Is it correct?

- **Diversity measures**: The diversity measures proposed by the authors are legit. However, maybe another interesting one would be to train a new model on datasets that they have generated with their approach. If the newly trained model gets a better performance, I think this would be the best proof for diversity.

- **Increasing the number of clusters leads to better diversity + performance?**: the current paper just considers the case where there are 8 clusters in the data attribution phase. One experiment that would have been nice is also to understand in the case of code, how pass@5 improves as the number of clusters increase ? 

- **Some improvements that may be done for the introduction**: I think it is a shame that the method is not fully presented in the introduction. Maybe it would be good to summarize it in a few words? Besides, I believe that it would be nicer if Figure 3 is in page 2 as a main figure.



[1] Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., ... & Li, Y. (2023). Textbooks are all you need. arXiv preprint arXiv:2306.11644.
questions: questions: I mentioned my questions in the weaknesses seciton.
limitations: limitations: The authors mentioned the limitations of their work and they look sound to me.
paper_summary: paper_summary: The paper introduces Synthesize-Partition-Adapt (SPA), a framework designed to generate diverse and high-quality responses from foundation models. SPA uses synthetic data and data attribution methods to partition data into subsets, training multiple model adaptations for these subsets.
strength_weakness: strength_weakness: - The motivation of three phases: synthesize, partition, then adapt is not clearly discussed (See my questions). The authors spend more time to describe what they are doing but not why they are doing so or why these specific designs would lead to desired outcome, which is diverse sampling technique.
    
- The experiments only compare with two simple baselines ablated from the proposed methods, omitting existing baselines in the literature.
    
- The paper omits most of the existing works in the literature regarding diverse sampling for LLMs and only considers simple temperature sampling techniques.
questions: questions: - The motivation of three phases: synthesize, partition, then adapt is not clearly discussed. How to ensure that the existing dataset is diverse?
    
- The framework heavily relies on the existing synthetic dataset where its diversity is already a concern. As the authors mentioned, the problem of generating diverse synthetic dataset is still a problem with current generative models. Would finetuning on these synthetic datasets destroy the generalizability of the models?
    
- The method relies on $M$ questions requiring various expertise knowledge to compute influence matrix for partitioning. Does those M questions need to be manually chosen?
    
- Why partitioning the datasets using the importance scores and then finetune different LoRAs with K partitions would lead to K diverse LoRAs components? The connection is vague to me.
    
- The method relies on a predefined number of partitions K (or finetuned LoRA components) when training. What if during the inference time, we would generate a lot more than K diverse responses? Does each finetuned model still suffer from diversity problem?
    
- During inference, LoRA components are sampled randomly, does it affect the quality of generated responses? What if some components being biased or learn some skills while losing other skills? It is not clear that, after partition and finetuning, which LoRA learns which expertise. Can each component has a weight depending on the question?
    
- The paper is missing most of the important baselines and citations, I listed some here for reference:
    
    - KL-Divergence Guided Temperature Sampling
        
    - Diversity of Thought Improves Reasoning Abilities of LLMs
        
    - Large Language Models as In-context AI Generators for Quality-Diversity
        
    - Controlled Decoding from Language Models
        
    - Quality-Diversity through AI Feedback
        
    - Language Model Decoding as Direct Metrics Optimization
limitations: limitations: yes
paper_summary: paper_summary: This paper presents a framework for eliciting diverse outputs from language models while maintaining quality/accuracy. The framework consists of: first partitioning a (synthetic) dataset of supervised instruction tuning data, then training parameter efficient model adapters on each partition, and finally at inference time sampling from each of the the models trained in the prior step.

Within this framework, the paper contributes empirical evaluations of 2 methods (influence functions and token overlap) for partitioning the instruction tuning datasets and measures the effects on both code completion and natural language generation benchmarks. They find that both methods generate more diverse output over random partitioning (i.e. an ensemble of models trained on random subsets of the data), suggesting that the method used to partition is important.

The authors contrast their method with temperature based sampling approaches to generating diverse data from an LLM.
strength_weakness: strength_weakness: *    Evaluation on natural language (non-code) tasks lacks a quality/accuracy metric for the proposed method (Fig 5). While it is great to see that diversity is increased, the authors do state that maintaining quality is important but it is unclear if that is the case for these tasks. It would be helpful to see "standard" eval metrics for some or all of MMLU, BBH, GPQA and WinoGrande using the methods proposed.
*   More clarity in the tradeoff the proposed methods make vs high temperature sampling would be helpful. At least for the code models, it appears that as temperature increases, the diversity of the single model might converge to that exibited by SPA. If that is the case some discussion of the tradeoffs involved, (e.g. benefits of being able to sample at low temperature) would strengthen the paper.
*   It is difficult to know how sensitive the results are to the diversity of queries used to generate the partitioning matrix? It would be helpful to understand the performance of a partitioning method that is based on the data but doesn't use human selected test queries (e.g. randomly select 12 prompts from HumanEval and then partition based on that using influence and token overlap approaches). This is not however a fatal flaw by any means.
questions: questions: Experiment Design

*   How were the 12 queries for each task generated? Could the authors shed any light on what they think important properties of this query set are?
*   What are the generation params for Section 5.3? (is it all greedy sampling)
*   Also in Section 5.3, why not compare with a single model at high temperature (Similar to the "Single" condition used for the code generation tasks)?

*   Results
*   How do the charts in figure 4 look like as temperature increases even further? Why did you stop at 0.5? The CodeLLama paper reports their main results at temperate=0.8 so it would be useful for readers to see if and where the methods converge in terms of diversity.
*   Did you try using fewer than 8 model adapters? While there doesn't seem to be any much improvement observed in using more than 8 adapters in this setting, do you have a sense of how few adapters one could use?


*   Metrics

*   For the average KL divergence metric could you explain how you go from token level probability distributions for sequence level probability distribution for Pi and Pj? Or do you mean something else, I'm mostly trying to understand how you handle sequences of different lengths in the computation.
*   For the sample diversity/Diversity score metric what value of K (line 273) is used in the experiments? And is the total number of samples generated in each condition the same?
*   In figure 4 how many samples are generated for each method to calculate pass\@k? And in the case of the SPA methods how is that distributed across the various adapters? My understanding is that there are 8 adapters in this experiment.
*   Do the authors have any thoughts on why pass@1 isn't higher for SPA compared to single model?
limitations: limitations: Yes. Though the paper could also better acknowledge the extra computational cost at inference time (compared to just sampling multiple times at high temperature).
paper_summary: paper_summary: This paper proposes a method to generated diverse responses for the language model. Given a synthetic finetuning dataset and test dataset, they calculate the importance of the finetuning data on each test data using methods such as influence function, and then partition the dataset into several smaller ones. Finally, they finetune several lora weights based on different partition datasets.
strength_weakness: strength_weakness: It involves finetuning, and need to finetune several lora weights. This would cause difficulty on parallel generation because we need to store multiple checkpoints into memory.
questions: questions: How to find the optimal number of partition in practice? I suppose there should be a sweet point.

Can authors report the actual time on calculating influence function and training the lora models for the experiments in this paper?

If you need multiple lora weights to generate diverse responses, are you able to still generate them in parallel without demanding memory requirement?
limitations: limitations: .

Comments: title: Thank you!
comment: Thank you for your engagement throughout this process. Our approach assumes LLMs undergo post-training with synthetic data, a common practice (e.g., Llama 3's iterative post-training). Given the constraints of this paradigm and motivated by diminishing returns of increasingly many synthetic data, sampling from multiple distributions using LoRA is our strategy to outperform a single model tuned on the entire synthetic dataset. This strategy aims to generate diverse, high-quality responses while addressing the challenge of diminishing returns in LLM post-training with synthetic data.

Although you may not be fully convinced, we hope that our paper and the additional experiments we've provided will contribute more insights to the later discussion.
Thank you for your positive feedback. We've also added new experiments in our reply to reviewer 4Daa addressing concerns about parallel generation and memory overhead. We're keen on your support for our submission.
Thank you for your positive feedback. Here are the accuracy results:

| Tasks   | BBH | GPQA | MMLU | WinoGrande |
|----------------|----------|------------|------------|------------|
|  Single |  46.66  |    4.14  |   55.40   |   76.5     |
|  SPA    |   46.48 |    3.80  |   55.26  |   76.3    |


SPA maintains comparable accuracy to the single model. The GPQA evaluation has high variance; even small changes in the prompt can lead to big differences in accuracy. Despite this, we still included GPQA because it provides a good source for evaluating diversity due to its broad range of topics. We will include the distribution in the next revision as you suggested.

We've also added new experiments in our reply to reviewer 4Daa addressing concerns about parallel generation and memory overhead. We greatly appreciate your thoughtful feedback and hope we have addressed your concerns. We're keen on your support for our submission.
Thank you! I am afraid that I would still keep my score because in a high level, I am not quite in favor of the idea of having multiple lora weights.
We sincerely appreciate your reconsideration and the increased score. We'd like to take the opportunity to address your remaining concerns:

> The same training procedure for each LoRA component leads to less diversity

Our approach operates under the assumption that LLMs undergo post-training using synthetic data, which is a common practice (e.g., iterative post-training in Llama 3 utilizes synthetic data in each round). Under this assumption, we have limited flexibility to modify the training procedure. 
Given these constraints, and motivated by diminishing returns, sampling from multiple distributions using LoRA components is our strategy to outperform a single model tuned on the entire synthetic dataset. While each LoRA component on its own may face diversity challenges like you mentioned, it won't be worse than the single model scenario.

Moreover, if modifying the training procedure is allowed, we could incorporate the mutual information as one of the objectives when optimizing LoRAs, this will encourage these LoRA components to be as diverse as possible. 


> Quality-diversity trade-off

We agree that calibrating this trade-off may be challenging in practice. On the other hand, SPA offers an improved Pareto frontier, improving diversity even at low temperatures. Users can now achieve good diversity at low temperatures (even when greedy), which was previously difficult. Thus, while SPA doesn't eliminate the trade-off, it provides users with more options.


> Predefined K and component quality, get the most probable sample

We want to clarify that K is not arbitrarily selected but depends on the synthetic dataset size and the strength of diminishing returns. As shown in Fig 4a, $\sum_i^K \text{Accuracy}(\text{model}_{\phi_i}) / K$, roughly equals the accuracy of a single model $\phi$, fine-tuned on the entire synthetic dataset. The average quality of LoRA components is roughly the same as the single model baseline.


For cases requiring the most probable answer (only one sample is required), we can conduct an offline evaluation on a held-out dataset before deployment, then sample from the best-performing LoRA component.

We hope our clarifications can further address your concerns.
Thank you for your follow-up question. We conducted two simplified scenarios using S-LoRA codebase and vLLM to compare sampling from multiple LoRA components versus a single model. 
Our setup uses a synthetic dataset in S-LoRA repo with 600 prompts and responses ranging from 8 to 512 tokens respectively.
For each prompt, the model generates tokens until the output length matched the response's length. We used Llama-13b with LoRA rank 16, leading to a 10% memory overhead for storing 8 LoRA adapters compared to a single model. When running the serving experiment, the requests are coming with a request rate of 2 per second.

In the first experiment, we simulated a scenario where multiple GPUs were used for serving. Here, the goal was to generate 8 diverse samples per request. We used 8 A100-40G GPUs. For the single model setup, incoming requests (say x and y) were expanded into batches of 8 identical prompts, $[x_1, x_2, ... , x_8]$ and $[y_1,y_2,...,y_8]$ and handled by vLLM’s continuous batching and routing. Usually, it will distribute $[x_1, x_2]$ to GPU-1 job queue, $[x_3, x_4]$ to GPU-2 job queue and so on.

For the SPA model, the batch becomes $[x_i, y_i, LoRA_i]$, where i ranges from 1 to 8.  The router then distributed these to different GPU job queues, with each GPU handling a unique prompt-LoRA pair.
This also gives us 8 samples per request. The results showed that this leads to a minor 3% overhead in both throughput (tokens / sec) and latency (secs / output token), primarily due to the additional step of distributing the LoRA adapters. This approach ensures parallel generation but requires multiple GPUs. However, it is a common practice to use multiple GPUs to serve real-world requests.

| 2 req/s   | tokens / sec | secs / output token | 
|----------------|----------|------------|
 |  Single | 473   |  0.035   | 
|  SPA  |  460  |  0.036     |


In the second experiment, we explored a single GPU scenario using S-LoRA. This setup also stores all LoRA weights in memory, leading to a similar 10% memory overhead as in the first scenario. In this experiment, we only generated one sample per request for simplicity. This means that for the single model, the metrics should be roughly the same as those obtained when sampling 8 outputs using 8 GPUs in the above scenario. In the SPA setup, each request was coupled with a random LoRA adapter, which is handled by the S-Lora library. The result below shows that this configuration leads to a ~10% overhead in throughput and latency.

| 2 req/s   | tokens / sec | secs / output token | 
|----------------|----------|------------|
 |  Single | 480   |  0.035     |
 | SPA  |  433 |  0.040     | 


The first result indicates that SPA can be implemented efficiently in a multi-GPU setup with almost no performance lost. Even in a single GPU scenario, leveraging S-LoRA allows us to sample from multiple LoRA adapters without significant trade-off. This is also an active research field, we could anticipate even more efficient solutions for sampling from multiple LoRA adapters with some future works. We hope these results help address your concerns.
I thank the authors for their response and clarifications. I overall still think this is an interesting paper!

It might be helpful to post the accuracy scores for the NL tasks that you mentioned in your rebuttal here. If accepted readers may also be interested in the distribution of accuracy across the various adapters.
Thank you for your rebuttal! I'm still a bit uncertain on "These advancements suggest that SPA can be implemented without significant memory constraints or parallel generation difficulties. As the field continues to progress, we anticipate even more efficient solutions for managing multiple LoRA adapters." Are you able to implement the combination with some of these methods you mentioned and show some results?
Thank the authors for the rebuttal and conducting additional experiments. For this reason, I increased my score, however, I still tend to rejection for this paper since I'm not really being convinced by diverse sampling process only via multiple LoRA components. Each LoRA component still suffers from the diversity sampling problem since  the training procedure for each LoRA component is still the same. 

Furthermore, calibrating the quality-diversity trade-off typically depends on each use case. Predefining K before finetuning makes it's difficult for the users to calibrate this trade-off, thus limiting its use case in practice. In those cases, we then need to come back to calibrate this trade-off by temperature, which, as the authors argued, has drawbacks. This also relates to my concerns regarding random choices of LoRA components during sampling. What if after finetuning, we want to generate the most probable, high-quality answer?
I thank the authors for responding to my questions. After reading the other reviews, I still believe that the paper is meaningful, has an interesting contribution and may benefit the community. I maintain my score.