HYDRA: Model Factorization Framework for Black-Box LLM Personalization

Authors: yuchen  zhuang
haotian  sun
yue  yu
rushi  qiang
qifan  wang
chao  zhang
bo  dai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:01:39

Abstract: Personalization has emerged as a critical research area in modern intelligent systems, focusing on mining users' behavioral history and adapting to their preferences for delivering tailored experiences. Despite the remarkable few-shot capabilities exhibited by black-box large language models (LLMs), the inherent opacity of their model parameters presents significant challenges in aligning the generated output with individual expectations. Existing solutions have primarily focused on prompt design to incorporate user-specific profiles and behaviors; however, such approaches often struggle to generalize effectively due to their inability to capture shared knowledge among all users. To address these challenges, we propose HYDRA, a model factorization framework that captures both user-specific behavior patterns from historical data and shared general knowledge among all users to deliver personalized generation. In order to capture user-specific behavior patterns, we first train a reranker to prioritize the most useful information from top-retrieved relevant historical records.
By combining the prioritized history with the corresponding query, we train an adapter to align the output with individual user-specific preferences, eliminating the reliance on access to inherent model parameters of black-box LLMs. Both the reranker and the adapter can be decomposed into a base model with multiple user-specific heads, resembling a hydra. The base model maintains shared knowledge across users, while the multiple personal heads capture user-specific preferences. Experimental results demonstrate that \method outperforms existing state-of-the-art prompt-based methods by an average relative improvement of 9.01% across five diverse personalization tasks in the LaMP benchmark.

Main paper: 
 introduction: 
 Pre-trained large language models (LLMs) [35, 36, 2, 31, 4, 30] have revolutionized various natural language processing (NLP) tasks, ranging from traditional recommendation systems [62, 12, 8, 6] to modern virtual assistants [9, 23, 63, 66, 44, 65, 47] . Despite their strong capabilities, LLMs require further customization to consistently demonstrate desirable behaviors to each user and achieve optimal performance in specific use cases [48, 43, 61, 10, 13, 14, 24, 57, 58, 45, 59 ]. As a result, LLM personalization has emerged as a rapidly evolving area of research [20] , with the goal of tailoring the emergent abilities of LLMs to meet the unique needs of individual users.Several existing studies have shown effectiveness in personalizing LLMs, including (1) fine-tuning personalized LLMs for each user [50, 54] and (2) aligning LLMs to personalized preferences through Reinforcement Learning from Human Feedback (RLHF) [22, 55, 17] . However, both fine-tuning and RLHF-based methods require access to model parameters, restricting their use to white-box LLMs only (e.g., LLaMA-2 [51] ). These models tend to be less capable than black-box LLMs (e.g., 38th Conference on Neural Information Processing Systems (NeurIPS 2024). indicates the trainable parameters, whereas indicates the inaccessible fixed parameters. [30] ) because they have access to less training data and smaller model scales. Moreover, RLHF-based methods require more explicitly attributed characteristics (e.g., style) [22, 17] from implicit user behavior history and necessitate excessive annotation efforts for capturing human preferences.Without access to modify the model parameters for black-box LLM personalization, an alternative solution is to augment user-specific content and/or context into the prompt template. One straightforward approach is to incorporate the user's complete profile or entire historical behavior into the prompt design [6, 18, 25, 52, 64, 5, 64] . However, integrating the entire profile may exceed the length limitations of LLMs and lead to substantial costs, while randomly selected records cannot effectively capture representative patterns. To address this dilemma, retrieval-augmented generation (RAG) approaches [41, 40, 29, 21] have been explored by extracting the most relevant information from the user's historical data to facilitate personalized generation. One limitation is that a retrieval-augmented framework encodes different users independently in a personalized prompt, making it challenging to capture the shared (global) patterns of the entire user group [50] . Moreover, in comparison to finetuning the entire or partial model parameters to create personalized language models for individual users [50, 22, 17] , simply augmenting the input prompt through a centralized LLM without updating model parameters may diminish the effectiveness of personalization (Figure 1 ).To address these challenges, we propose HYDRA, a learning-based model factorization framework that captures both user-specific and shared behavior patterns to enable effective personalization within black-box LLMs. HYDRA leverages a retrieval-augmented workflow, where a retriever initially extracts relevant user behaviors from historical data for effective user-specific preference identification. To achieve personalized generation, we focus on the training process of two fundamental components:(1) a personalized reranker to prioritize useful user information from the retrieved records, and (2) a personalized adapter to align black-box LLM outputs with user-specific preferences, without requiring access to internal model parameters. Both the reranker and the adapter can be decomposed into a base model with multiple personalized heads, similar to a Hydra. By employing model factorization, we effectively integrate shared (global) knowledge, captured by the centralized base model, with user-specific preferences, harnessed through multiple user-specific heads, to enhance generalization across the entire user group.We conduct extensive experiments on LaMP [41] , a comprehensive language model personalization benchmark, to evaluate the personalization capabilities of HYDRA across multiple dimensions, including three text classification tasks and two text generation tasks. Notably, HYDRA achieves an average improvement of 4.8% over the best-performing baselines across all five diverse tasks. Further in-depth studies reveal the robust capability of HYDRA in scaling up to accommodate larger user groups and extensive behavior history, as well as adapting to user behavior shifts. We also demonstrate the effectiveness of shared knowledge in enhancing user experience through both quantitative and qualitative analyses. To facilitate future research in black-box LLM personalization, we will release the code repository and model checkpoints for transparency and reproducibility.Our main contributions are as follows: (1) We propose HYDRA, a black-box LLM personalization framework that effectively mines user behavior history and adapts to user preferences for enhanced user experience; (2) HYDRA integrates shared (global) knowledge from the base model and individual (local) preference from multiple user-specific heads through model factorization to deliver generalizable personalization; and (3) HYDRA significantly outperforms existing prompt-based methods across five diverse tasks in the LaMP benchmark [41] , introducing a novel learning-based solution that achieves more effective adaptation to individual users in black-box LLMs. 
related works: 
 In-Context Learning. Vanilla personalized prompting approaches leverage the powerful in-context learning capability of LLMs by using the user's randomly sampled behavior history as contextual samples. Existing studies [6, 18, 25, 52, 64] have utilized encoding user histories-whether personal ratings, interaction histories, or exemplary reviews-as few-shot examples to facilitate LLMs in generating personalized content in various down-stream applications. Additionally, research has shown that utilizing a longer user history can potentially lead to better performance [5, 64] .Profile-Augmented Prompting. Improving upon the random sample strategy and leveraging the insights from the enhanced performance with more historical information, profile-augmented generation (PAG) summarizes user preferences and behavior patterns into natural language profiles for query augmentation. For instance, Richardson et al. [38] employ instruction-tuned LLMs to generate abstract summaries of user history data, integrating summarization for enhanced personalization. Similarly, ONCE [26] creates user profiles by summarizing topics and regions of interest from their browsing history, thereby assisting LLMs in capturing user preferences for downstream tasks.Retrieval-Augmented Prompting. Compared to random sampling in in-context learning and the use of entire histories in PAG, retrieval-augmented prompting excels at extracting the most relevant records from user behavior history to enhance LLM personalization, thereby efficiently managing the growing user behavior data within LLMs' limited context length and supporting personalized generation with more relevant evidence. For instance, LaMP [41, 40] introduces a retrieval-augmented method to obtain the most relevant content from the user's behavioral history and incorporate it into the prompt design. Similarly, AuthorPred [21] retrieves relevant past user-written documents for personalized text generation. Pearl [29] proposes a generation-calibrated retriever to select from historic user-authored documents, enhancing personalization with relevant prompt augmentation.Limitations. Similar to in-context learning, personalization based on PAG is prone to be easily distracted by irrelevant information retrieved, especially when there is a shift in user behavior between the current query and the user's historical records. Despite the improvement upon PAG by retrieving relevant information, RAG-based methods may still suffer from the quality of retrieved information, where the "most relevant" information may not necessarily be the "most useful" information to answer a new query. Additionally, prompting-based methods not only lack deeper personalization analysis due to their reliance on a single centralized model but also lack access to global knowledge due to the user-specific prompt design. Black-box LLM personalization refers to tailoring model generations to align with user preferences according to their history [20, 50, 17] , without having access to model parameters. Specifically, given a black-box LLM G and a training dataset D train = {(q u , r u , H u )}, where for each user u, q u indicates the input sequence, r u refers to the target output, and H u represents the user's historical behavior containing preference information. The user history data H u = {h i u } includes all user behaviors h i u , consisting of (q i u , r i u ) pairs, mirroring the task-specific query-answer format (q u , r u ). The goal of personalization is to adapt the LLM's generation r to the target output r, conditioned on both the input and the user history.Traditional fine-tuning methods train universal models for all users, whereas personalized tuning aims to develop a unique model for each user u, denoted as θ (u) , which captures the unique characteristics of its user-specific data distribution D (u) . Ideally, each user model should possess both shared (general) and user-specific knowledge. Thus, the problem can be formulated as a decomposition of θ (u) into a shared part σ and a personalized part τ (u) to learn general and user-specific knowledge, respectively. The training object of LLM personalization can be formulated as: min σ,τ (1) ,τ (2) EQUATIONwhere L u (σ; τ (u) ; D (u) ) denotes the loss function of user u, and U is the total number of users. Specifically, in the black-box LLM personalization, the model parameters in G are not accessible, Inference with 𝝈𝝈 and 𝝉𝝉 (𝒖𝒖 ′ )Figure 2 : Overview of HYDRA. HYDRA follows a retrieval-augmented framework: (1) Firstly, we extend an original RAG to a two-stage retrieve-then-rerank workflow, where we rerank the most useful information from relevant user behavior records to capture user-specific preference (Section 3.2); (2) Secondly, augmented by the selected historical data, we train an adapter to align the output of blackbox LLMs with personalized human preference (Section 3.3). Both the reranker and the adapter can be decomposed into a base model with multiple user-specific heads, resembling a hydra-like structure (Section 3.4). The base model maintains shared knowledge across users, while multiple personal heads capture user-specific preferences. represents the model decomposition for personalization.making it infeasible to directly fine-tune the black-box LLM. Thus, we present HYDRA, a model factorization framework for black-box LLM personalization, as shown in Figure 2 .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have discussed the limitations of HYDRA in Appendix A.1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
retrieve-then-rerank: 
 To capture user-specific preference, we follow a retrieve-then-rerank workflow to (1) retrieve the relevant user behavior records and (2) rerank them based on usefulness. Specifically, given an input query x, we employ a retriever to retrieve top-N user history behaviors that have the highest relevance scores when compared with the input x. The objective of the reranker is to identify user historical records that can serve as useful user preference information for answering the user's incoming queries.Training Data Generation. Within the training data, each user only has a single query, which is insufficient for capturing the relationships between the behaviors of the same user. Scoring all the history pairs of users is quadratic in|D| u=1 |H u | × |H u |, which becomes prohibitively timeconsuming. In order to obtain a high-quality candidate set of training samples for each user query q u , we retrieve the top-M relevant history records, denoted as R(q u , H u ). Additionally, to gain a better understanding of the user's history, we randomly sample another M historical records that can be considered as the user's previous queries. For each of these previous queries, we also retrieve the top-M relevant historical records. The training candidates can be represented as:EQUATIONwhere q i indicates the query, r i indicates the ground-truth answer to the query, and h i indicates the candidate history. We then utilize an LLM as a labeling function to measure the potential improvement (i.e., usefulness) of each history upon the LLM personalized generation. Specifically, for each candidate (q i , r i , h i ), we first sample the generation of LLM ri by using the input context q i and the candidate history h i from p LLM (r i |q i , h i ). Next, we compare the generation ri with the ground-truth answer r i , and create a binary label y i = 1(r i = r i ). For generation task, the condition is soften to y i = 1(r i ≈ r i ), where the Rouge metrics between r i and ri is above a pre-defined threshold. To optimize the reranker, we combine the user query with the candidate history in an entailment learning style, resulting in the training inputs .x i = {[CLS] q i [SEP] h i [SEP]}.Training the reranker for each user allows for a personalized selection of relevant historical information based on the user's query. The training objective involves using the cross-entropy loss between the predictions made by HYDRA-Reranker heads and the ground truth. For a specific user u and the corresponding training sample (x u i , y u i ) ∈ D reranker , we can calculate the cross-entropy loss function for HYDRA-Reranker as follows:EQUATIONwhere p u i indicates the model prediction made by HYDRA-Reranker. HYDRA-Reranker Inference. With the trained reranker, our objective is to select the top-k most relevant candidates from the retrieved user history for a personalized generation. For each query q i from user u in the test data, we retrieve the most relevant history H u i = {h u i,1EQUATIONComplementary to personalizing the few-shot demonstrations for LLMs in Section 3.2, we further align the black-box LLM generation to user-specific preference through the training of a personalized adapter, eliminating the need for directly accessing model parameters.Training Data Generation. To take full advantage of the preference information hidden in the user query and history, we generate the candidates for the adapter training:EQUATIONFor each input query q i from the training dataset, we augment the reranked history candidates C i with the query and sample b candidate responses from the black-box LLM:EQUATIONIn comparison with the ground-truth personalized generation r i , we can evaluate each generated solution ri,j and assign a corresponding binary label y i as y i = 1(r i,j = r i ). Using the model generations, we establish a new dataset for the adapter training, denoted as D adapter = {x i,j , y i } |Dadapter| i=1, where x i,j = {[CLS] q i [SEP] ri,j [SEP]} represents the concatenation of the user query with the entire candidate generation.Training. We utilize the model decomposition training strategy (detailed in Section 3.4) to train personalized adapters for each user, which allows us to tailor the selection of candidate generation that best aligns with the user's preference. To calculate the task-specific loss function, we follow Eq. (3) and employ the same cross-entropy loss between the predictions of user-specific heads and the ground truth to optimize HYDRA-Adapter.HYDRA-Adapter Inference. During the process of model inference, we conceptualize the black box LLM as a proposal generator, while the adapter functions as an evaluator. Specifically, for each test query x i from user u, we adopt the best-of-b inference, also known as rejection sampling. Initially, we sample b candidate generations {r i,j } b j=1 from the LLM. By passing them through the HYDRA-Adapter, we obtain the corresponding score for each candidate{p u i,1 , • • • , p u i,b }.The solution with the highest score is then chosen as the final answer:EQUATIONTo further effectively capture the shared knowledge across all users as well as the individual userspecific behavior patterns, a direct solution is to develop a parameter decomposition method, where the shared parameters store the general knowledge, benefiting from a high model capacity, while the remaining parameters are designed to learn user-specific preferences that complement the overall understanding (typically on a smaller scale). Hence, it is adequate to employ a smaller-sized model (adapter) to represent these personalized parameters, instead of fine-tuning the entire LLM.Model Factorization. Formally, in HYDRA, we assume that each personalized language model is associated with a set of weights Θ = {θ (u) } u∈D , where θ (u) represents the weights of the personalized model for user u. Each θ (u) is decomposed as:EQUATIONwhere σ represents the base model parameter matrix that is shared among all users, τ (u) represents a personalized head model, and ⊗ indicates the operation of appending the user-specific head τ (u) on top of the base model σ. Specifically, we add |D| heads to the final hidden states s generated by the original model. The u-th head measures the usefulness between query and histories, or the level of preference for the generation from the perspective of the u-th user. The prediction of the u-th head is denoted as p (u) . For each head, we employ a single layer of feed-forward network. We define u-th head asEQUATION2 ] and the prediction can be outlined as:p u = softmax(W (u) 2 • Tanh(W (u) 1 • s + b (u) 1 ) + b (u)2 ), where Training Strategy. We then train the base model in conjunction with multiple user-specific heads. For each sample (x u i , y u i ) of user u from the training data and the model prediction p u i , we update the base model and the u-th head accordingly:EQUATIONEQUATIONwhere α indicates the learning rate and L represents the task-specific loss function.Fit Test User History. In order to accommodate the newly incoming users in the test data, we cannot reuse the personalized heads from the training data. Therefore, we need to create and initialize new heads. To fit the test users' history, we freeze the base model and solely focus on the heads. This fitting process is simple and requires minimal computational resources. Similar to the update of τ (u) in Eq. (10) , when given a new user u ′ for testing, we also leverage the task-specific loss function L(•) to update the head τ (u ′ ) :EQUATIONwhere (x u ′ i , y u ′ i ) are obtained from test user history. Personalized Inference for Test Users. Upon fitting the test user history, we obtain the base model σ, which contains general knowledge across users, and the user-specific head τ (u ′ ) , which captures the user-specific knowledge in the test user history. Therefore, we can apply personalized inference directly to the test user u ′ , given its query x u ′ :EQUATIONwheref σ⊗τ (u ′ ) (•) indicates the inference of the model σ ⊗ τ (u ′ ) .4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provide the code in a .zip file during the submission and peer-review stage. We will release the code repository and model checkpoints for transparency and reproducibility on GitHub upon acceptance. Data is publicly available in LaMP benchmark [41] .Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Datasets and Tasks. We adopt a widely used language model personalization benchmark, LaMP [41] , focusing on a diverse set of personalized text classification and generation tasks, including (1) Personalized News Categorization (LaMP-2N), (2) Personalized Movie Tagging (LaMP-2M), (3) Personalized Product Rating (LaMP-3), (4) Personalized News Headline Generation (LaMP-4), and (5) Personalized Scholarly Title Generation (LaMP-5). For data splitting, we follow the user-based separation setting provided by the LaMP benchmark, with 100 randomly selected users for training and an additional 50 randomly selected users for testing. No shared users are displayed across splits for specific measurements in personalization for new users. Additional details of the LaMP benchmark are available in Appendix D. Baselines. We compare our proposed HYDRA with existing state-of-the-art black-box LLM personalization methods, including in-context learning (ICL-Random, with random selected k-item from user behavior history), retrieval-augmented prompting (RAG) [41] , and profile-augmented prompting (PAG) [38] . We also present the experimental results of gpt-3.5-turbo [30] (zero-shot) without profile/history augmentation in order to showcase the baseline performance of the backbone language model. Learning-based personalization, such as fine-tuning and RLHF-based methods, cannot be applied to black-box LLMs (e.g., gpt-3.5-turbo) due to the unavailability of model parameters. Details of baseline implementations are available in Appendix E.Acc. ↑ F-1 ↑ Acc. ↑ F-1 ↑ MAE ↓ RMSE ↓ R-1 ↑ R-L ↑ BLEU ↑ R-1 ↑ R-L ↑ BLEU ↑ gpt-3.5-Dataset (→) LaMP-2N LaMP-2M LaMP-3 LaMP-4 LaMP-5 Method (↓) Acc. ↑ F-1 ↑ Acc. ↑ F-1 ↑ MAE ↓ RMSE ↓ R-1 ↑ R-L ↑ BLEU ↑ R-1 ↑ R-L ↑ BLEU ↑ HYDRA 0.Evaluation Metrics. Following the evaluation metrics specified in LaMP [41] , we utilize accuracy (Acc.) and F-1 score (F-1) for the test classification tasks in LaMP-2N and LaMP-2M. Additionally, we employ mean absolute error (MAE) and root mean squared error (RMSE) for the ordinal multiclass classification task in LaMP-3. To comprehensively evaluate the personalized text generation tasks in LaMP-4 and LaMP-5, we report the ROUGE-1 (R-1), ROUGE-L (R-L), and BLEU metrics.Implementation Details. For both baselines and our proposed HYDRA, we follow the same prompt template in the LaMP benchmark [41] and employ gpt-3.5-turbo(1106) and BM25 [39] as the backbone black-box LLM and default retriever, respectively. Additionally, both HYDRA-Reranker and HYDRA-Adapter leverage the lightweight LongFormer-Base (110M) [1] as the backend language models. We include more implementation details in Appendix F.Table 1 presents the main experimental results of five varied personalization tasks in the LaMP benchmark. Compared to the zero-shot setting in gpt-3.5-turbo [30] , even a random selection of historical records from user behavior or profiles enhances the model performance in most tasks, suggesting that personalization contributes to improved performance with black-box LLMs. HYDRA exhibits substantial relative improvements across all five tasks, with an average of 9.01% over the bestperforming baselines. Specifically, HYDRA outperforms the best-performing alternative by 14.71% in accuracy for the Personalized News Categorization (LaMP-2N) task, 3.85% in accuracy for the Personalized Movie Tagging (LaMP-2M) task, 20.00% in MAE (where lower values indicate better performance) for the Personalized Product Rating (LaMP-3) task, 2.89% in R-1 for the Personalized News Headline Generation (LaMP-4) task, and 3.58% in R-1 for the Personalized Scholarly Title Generation (LaMP-5) task. This further improvement can be attributed to the improved quality (usefulness) of retrieved records in better representing user-specific preferences and the effective integration of global knowledge sourced from the entire user group.From Table 1 , both HYDRA-Reranker and HYDRA-Adapter demonstrate their effectiveness by significantly improving the RAG and PAG baselines separately. Furthermore, we observe that HYDRA outperforms each of them individually, showcasing the complementary role of both components. In Table 2 , we further eliminate the personalized training strategy of HYDRA, deteriorating to training a singular model across the entire user group, which solely incorporates global knowledge and lacks individualized customization for each user. The decline in model performance across all tasks highlights the significance of personalization through user-specific heads. In addition, we conduct additional experiments to comprehensively evaluate different components of HYDRA in black-box LLM personalization, including the effect of retrievers (Appendix G.4), the effect of adapters (Appendix G.5), and the effect of reranker with case studies and error analysis (Appendix H). We study the impact of users in the training data, as depicted in Figure 3 (a)-(c). As the number of users increases from 20 to 100, we observe that HYDRA reaches over 90%Number of History per User. We examine the impact of the number of historical records per user in Figure 3 (d)-(f). We randomly chose 50 users from each range of number of history per user. As the number of historical records increases, we consistently observe improved performance in both classification and generation tasks, as indicated by all metrics. This demonstrates the robustness of HYDRA in effectively capturing user-specific preferences, especially with a larger amount of user historical data.User. In Figure 3 framework in black-box LLM personalization. However, an inconsistency is noted with RAG in LaMP-3, which may be attributed to the presence of noisy or irrelevant retrieved history. This finding emphasizes the importance of retrieval quality and raises the significance of the HYDRA-Reranker in measuring the usefulness of retrieved items. Additional scale-up experimental results are available in Appendix G.2.Dataset (→) LaMP-2N LaMP-2M LaMP-3 LaMP-4 LaMP-5 Method (↓) Acc. ↑ F-1 ↑ Acc. ↑ F-1 ↑ MAE ↓ RMSE ↓ R-1 ↑ R-L ↑ BLEU ↑ R-1 ↑ R-L ↑ BLEU ↑ gpt-3.Set. We also conduct additional scale-up experiments (Table 3 ) to evaluate HYDRA with an increased number of users, increasing from 100 to 1000, across all five tasks. Our findings from the scale-up experiments show that HYDRA maintains its performance advantages over baselines as the number of users increases.Performance w/o Shared Knowledge. We conduct additional experimental analysis on both text classification and generation tasks for a comprehensive evaluation. In Figure 4 , we study how the shared knowledge is leveraged in HYDRA. Specifically, we create three settings with different levels of leveraging shared knowledge: ( Performance under User Behavior Shift. Following the setup in Tan et al. [50] (see Appendix G.3 for additional details), we evaluate HYDRA under user behavior shift when the query is not relevant to all historical records. As illustrated in Figure 5 , our proposed HYDRA continues to outperform the current state-of-the-art baselines in the face of user behavior shift. This serves as evidence of its robustness and generalizability in providing a widely applicable solution for black-box LLM personalization. This improvement can be attributed to the personalized reranker, which reevaluates the candidates based on their utility rather than solely on relevance, together with the personalized adapter further adjusts to align with user preferences. is important to note that potential private information may be contained in the retrieved historical records or the query itself, which could introduce a potential risk of private data leakage [60] .On-Device Inference. In real-world recommendation systems or virtual assistants, on-device inference with small-scale language models may be necessary [56] . While HYDRA-Adapter is a lightweight language model with 110M parameters and is efficient in fine-tuning during black-box LLM adaptation, the extensive model parameters of black-box LLMs themselves may eliminate the possibility of on-device inference.Potential Positive Societal Impacts. The proposed HYDRA addresses an important challenge posed by the inherently opaque nature of state-of-the-art LLMs, like GPTs [30] , enabling learning-based personalization in black-box LLMs. Effective personalization of black-box LLMs can lead to significant improvements in user experience and engagement across a wide range of applications, from virtual assistants to recommendation systems. By tailoring the outputs of LLMs to the unique preferences and behavioral patterns of individual users, HYDRA has the potential to provide more useful, relevant, and satisfying interactions. This could enhance productivity, decision-making, and overall quality of life. Moreover, the ability to personalize language models without requiring access to their internal parameters makes this technology more accessible. This democratization of personalization capabilities can benefit a broader segment of the population, including individuals and organizations that may not have the resources or technical expertise to fine-tune or adapt large language models directly. Increased accessibility to personalized language models could lead to more inclusive and equitable technological solutions.Potential Negative Societal Impacts. However, the personalization of language models also carries some potential risks and negative societal impacts, including concerns around user privacy and data security [20] , as the personalization process requires collecting and storing sensitive user behavioral data. Improper handling or misuse of this data could lead to privacy breaches and unauthorized profiling of individuals. Robust data governance frameworks and stringent privacy safeguards will be crucial to mitigate these risks. Additionally, the personalization of language models could potentially exacerbate issues of algorithmic bias and filter bubbles. If the personalization process is not carefully designed to avoid reinforcing existing biases or limiting users' exposure to diverse perspectives, it could contribute to the creation of echo chambers and the perpetuation of biased decision-making.Future Works. Our research highlights several meaningful avenues for future exploration. It is imperative to prioritize fairness, diversity, and inclusivity in the personalization process in order to address privacy-related concerns. Furthermore, one potential direction is to expand the scope of our applications beyond LLM personalization to encompass multi-modal vision language model personalization. This endeavor will not only enhance user-centric intelligent systems but also contribute to elevating the overall user experience.Overall, the proposed HYDRA framework has the potential to deliver significant positive societal impacts through enhanced user experience and increased accessibility to personalized language models. However, careful consideration of privacy, data security, and algorithmic bias mitigation will be necessary to address the potential negative consequences and ensure the responsible development and deployment of this technology.We strictly follow data usage guidelines during the interaction with Microsoft Azure's Open AI API service. Although we utilize all publicly available datasets in this study, we have withdrawn from the human review process by completing and submitting the Azure OpenAI Additional Use Case Form1 to prevent any potential information leaks.We summarize existing LLM personalization studies in ICL-Random ✗ ✗ ✗ ✗ ✗ ✗ ✓ RAG [40] ✓ ✗ ✓ ✓ ✗ ✗ ✓ PAG [10] ✓ ✗ ✓ ✓ ✗ ✗ ✓ Learning-based Methods Fine-Tuning ✗ ✓ ✗ ✗ ✗ ✓ ✗ OPPU [50] ✓ ✗ ✗ ✗ ✗ ✓ ✗ RLPHF [17] ✓ ✗ ✗ ✗ ✗ ✓ ✗ HYDRA (Ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓Aligning Language Models to Personal Preferences. While RLHF aligns LLMs with general human preferences [33, 67, 46, 37] , it is inadequate for capturing diverse and individual perspectives, as it assumes a reward model based only on average annotator preferences and ignores variations in user-desired outputs even for the same prompt [3, 42] . The personalized-RLHF (P-RLHF) framework [22] addresses diverse user preferences encoded in human feedback by jointly training a user model to capture individual preferences and a reward model to generate personalized language. Similarly, fine-grained RLHF [55] enhances personalization by providing rewards after each generated segment and integrating multiple reward models for various feedback types, including information incorrectness, irrelevance, and incompleteness. Additionally, Reinforcement Learning from Personalized Human Feedback (RLPHF) [17] further considers scenarios of conflicting preferences and achieves personalized alignment by decomposing preferences into multiple dimensions. However, RLHF-based solutions often focus on preference alignment by explicitly decomposing preferences into multiple predefined dimensions specified by the user, which may not always be feasible in real-world scenarios.Personalized Parameter-Efficient Fine-Tuning (PEFT). An alternative approach to personalization is to develop individual language models tailored to each user [50, 54] . One notable example of relevant work, although only available on white-box LLMs, is One PEFT Per User (OPPU) [50] , which trains a PEFT model for each user using Low-Rank Adaptation (LoRA) [14] . OPPU effectively combines parametric user knowledge embedded in personalized PEFT parameters with non-parametric knowledge obtained through retrieval and user profiling, ensuring a more comprehensive and personalized user experience.Limitations. Learning-based methods primarily focus on white-box LLM personalization, involving personalized alignment through parameter merging and personalized reward models, which is not applicable to black-box LLMs due to the opacity of their model parameters. Specifically, RLHF-based solutions require explicit personalization information (e.g., difficulty, style) as preference pairs in addition to behavior history, which is difficult to collect and necessitates additional human effort in annotations. Compared with existing personalization solutions that develop individual user models for user-specific alignment, HYDRA takes into consideration shared (global) knowledge for more effective personalization with potential generalization to the entire user group. In addition, personalization is a strategic priority for many corporate corporations, where most state-of-the-art solutions remain proprietary and are not available as open-source. Fine-Tuning [7] ✗ ✗ ✗ ✓ ✗ Instruction-Tuning [53] ✗ ✗ ✗ ✓ ✗ Continual Pre-Training [10] ✗ ✗ ✗ ✓ ✗ Adapter [13] ✗ ✗ ✗ ✓ ✓ Prefix-Tuning [27] ✗ ✗ ✗ ✓ ✓ LoRA [14] ✗ ✗ ✗ ✓ ✓ Grey-Box LLM Adaptation LMaaS [49] ✓ ✗ ✗ ✓ ✓ kNN-Adapter [15] ✓ ✓ ✗ ✗ ✓ CombLM [32] ✓ ✓ ✗ ✓ ✓ Proxy-Tuning [24] ✓ ✓ ✗ ✓ ✓ Black-Box LLM Adaptation BBox-Adapter [48] ✓ ✓ ✓ ✓ ✓ HYDRA-Adapter (Ours) ✓ ✓ ✓ ✓ ✓Following Sun et al. [48] , we leverage the level of access to LLM internal parameters to determine its classification as white-box, grey-box, or black-box LLMs (Table 5 ). White-box LLMs grant full access to both model parameters and output probabilities, whereas grey-box models provide access only to output probabilities. In contrast, black-box models restrict access to both parameters and output probabilities.Fine-tuning APIs, such as the OpenAI GPT-3.5-turbo API [34] , enable the effective adaptation of black-box LLMs by allowing users to upload training data and download fine-tuned outputs. However, black-box LLM adaptation through APIs presents several significant challenges [48] : (1) Transparency: The fine-tuning process is largely opaque, with limited visibility into key aspects such as the extent of trainable layers and specific model weights. This lack of transparency hinders optimal customization, as users are restricted to adjusting a narrow set of hyperparameters, such as the number of tuning epochs. (2) Privacy: Uploading training data via APIs raises concerns about potential privacy breaches, particularly in sensitive domains. (3) Cost: Fine-tuning APIs incur substantially higher costs compared to inference, making the adaptation process expensive. Moreover, the cost of fine-tuning escalates significantly when hyperparameter tuning is involved, further increasing the financial burden. As a result, we propose HYDRA-Adapter to eliminate the requirement for direct access to model parameters or API services in order to enable black-box LLM adaptation.historical article-title pairs. This task evaluates the language model's ability to replicate the authors' stylistic nuances in headline generation.• Task 5: Personalized Scholarly Title Generation (LaMP-5) is a text generation task that involves generating titles for input articles based on the author's historical article and title pairs. This task extends the exploration of personalized text generation into scholarly domains, explicitly focusing on generating titles for research articles.LaMP-6 has been excluded since the dataset is not publicly available. Furthermore, two additional tasks (LaMP-1 and LaMP-7) have been excluded from our empirical evaluation due to the inconsistent formats between user history and query.• Personalized Citation Identification (LaMP-1) is a binary text classification task that involves predicting citation choices. It requires determining which of two candidate papers will be cited in a new paper based on a user's historical publication data. This process utilizes user profiles that include the titles and abstracts of their previous works. This task aims to evaluate a language model's capability to identify the citation preferences specific to each user.• Personalized Tweet Paraphrasing (LaMP-7) is a text generation task that involves paraphrasing an input tweet into a personalized one, using the user's historical tweet data for stylistic guidance. This task evaluates a model's proficiency in reproducing the unique stylistic patterns in authors' social media posts. We compare our proposed HYDRA with both non-personalized and personalized baselines. The nonpersonalized baselines consist of zero-shot gpt-3.5-turbo [30] and ICL-Random (In-Context Learning with Random items from user behavior history). On the other hand, the personalized baselines include RAG (Retrieval-Augmented Personalization) and PAG (Profile-Augmented Personalization). k denotes the number of retrieved items from the user behavior history (or profiles). For all baselines, we follow the same prompt template in the LaMP benchmark [41] and employ gpt-3.5-turbo(1106) as the backbone black-box LLM. We utilize BM25 [39] as the default retriever for all experiments.• gpt-3.5-turbo [30] processes only the user's query (i.e., the input question) without leveraging the user's profile data.• ICL-Random augments the user's query with random k items from behavior history, enabling in-context learning to improve the capability of the original backbone LLM.• RAG [41] augments the user's query with top k retrieved items from the corresponding user's behavior history, following the retrieval-augmented personalization method in LaMP.• PAG [38] augments the user's query with the user's profile summary generated by LLMs (gpt-3.5-turbo) which concludes the user's behavior patterns. We further combine PAG with k retrieved items from the corresponding user's behavior history.Hardware and Software. We conduct all black-box LLM personalization experiments on CPU: AMD(R) EPYC(R) 7702 64-Core Processor @ 1.50GHz and GPU: NVIDIA A100-SXM4-80GB using Python 3.10.13.Hyperparameter Configurations. We set the maximum sequence length for a generated solution as 512 tokens and the temperature as 1.0 for flexibility in the generations of black-box LLMs, which serve as potential solution candidates. For other baselines, we maintain the temperature to 0 to avoid potential instability in performance. During the black-box LLM adaption stage, we set b = 8 for the generation of intermediate candidates using HYDRA-Adapter. During the training stage, we set the learning rate to 2e -5, the global batch size to 64, and the number of training epochs to 2 as default hyperparameter settings for all experiments. In addition, we employ AdamW [19] as the optimizer with a weight decay of 0.01. We include prompt details in Appendix I.We conduct further analysis of the main experimental results of dominant approaches for black-box LLMs, specifically prompt-based methods, including RAG and PAG. Compared to the zero-shot setting in gpt-3.5-turbo [30] , even a random selection of historical records from user behavior or profiles enhances the model performance in most tasks, suggesting that personalization contributes to improved performance with black-box LLMs. However, we also observe several instances of less optimal performance, particularly in LaMP-3. We hypothesize that this inconsistency may arise from the introduction of noise and irrelevant behavior patterns by the retrieved items, potentially complicating the understanding of user behavior patterns. Additionally, both RAG [41] and PAG [38] demonstrate marginal improvements over random selection, which may benefit from augmenting relevant behavior or profile information. The experimental results also consistently demonstrate that an increase in the number of retrieved items is positively correlated with improved performance, indicating the effectiveness of the retrieval-augmented framework in black-box LLM personalization. However, in comparison to HYDRA, RAG-and PAG-based methods still yield suboptimal results. This is primarily attributed to the presence of potentially noisy and irrelevant retrieved behavior patterns, as well as the absence of shared general knowledge across the entire user group. Following the setup of existing work [50] , we utilize an encoder-only language model called DeBERTa-v3 [11] to encode the user's historical behaviors and the query into high-dimensional representations. We then compute the cosine similarity between the query and all historical items to evaluate their relevance. We select the top 20 items with the lowest relevance scores as irrelevant user history. As depicted in Figure 5 , our proposed HYDRA still outperforms the state-of-the-art baselines. This performance improvement may stem from the personalized reranker, which reranks the candidates based on their usefulness rather than just relevance. Additionally, the personalized adapter also adapts the model's generations to align with user preferences. In addition, prompt-based methods tend to underperform when there is a distribution shift between the few-shot demonstrations and user queries. This issue can become even more severe under personalized scenarios, where there might be a mismatch between the selected user history and the user query.We investigate the effect of different retrievers in Table 7 . Under the black-box LLM personalization scenario, dense retrievers that capture semantic relevance (e.g., Contriever [16] ) achieve similar performances to sparse retrievers (e.g., BM25 [39] in Table 1 ), which is aligned with observations in the existing works [40] .Table 7 : Additional experimental results with Contriever [16] as the retriever (k=4). Dataset (→) LaMP-2N LaMP-3 LaMP-4 Method (↓) Acc. ↑ F-1 ↑ MAE ↓ RMSE ↓ R-1 ↑ R-L ↑ BLEU ↑ ICL-We evaluate the effectiveness of different black-box LLM adapters in Table 8 . Due to the inaccessibility to model internal parameters, BBox-Adapter is the only black-box LLM adapter available, apart from our proposed HYDRA-Adapter. Based on Table 8 , it can be observed that HYDRA-Adapter outperforms BBox-Adapter when applied under the same personalization settings (as discussed in Section 3.4). This result demonstrates the effectiveness of our adapter module design, which captures both global and user-specific knowledge to further adapt to personalized model outputs. Dataset (→) LaMP-3 LaMP-4 LaMP-5 Method (↓) MAE ↓ RMSE ↓ R-1 ↑ R-L ↑ BLEU ↑ R-1 ↑ R-L ↑ BLEU ↑ Personalized BBox-To further validate the effectiveness of HYDRA, we conduct extensive experiments on two additional widely used personalization datasets [28] , MovieLens-1M and Recipe, focusing on predicting users' personal ratings for movies or recipes based on their historical rating patterns. The experimental results (Table 9 ) demonstrate that HYDRA outperforms the best-performing baselines by 8.0% on Movielens and 10.3% on Recipe, respectively.Assume that we have N train users in the training data and N test users in the test set. We adopt the transformer architecture, specifically the Longformer-base, as the reranker and the adapter base models. Consequently, the time complexity of all stages should be proportional to that of the attention mechanism in transformers, O(dL 2 ), where L indicates the sequence length and d indicates the hidden dimension. Additionally, the training process will go through the transformer for T epochs, while inference only requires one. For each user in the HYDRA-Reranker training data, we augment M random historical records from the user's corresponding profile. The retriever then retrieves the Table 11 illustrates two observed cases during the reranking process. In Case 1, the retriever accurately retrieves the text in the optimal order, meaning that the top outputs are the most relevant to the target. In this scenario, the reranker successfully generates scores that align with the retriever, preserving the correct relevance. In Case 2, when the retriever fails to retrieve the text in the optimal order by relevance, the reranker assigns higher scores to the more relevant terms, thereby mitigating the incorrect retrieval ordering. This emphasizes the significance of HYDRA-reranker by offering an additional measurement dimension that is dedicated to assessing usefulness.Table 12 presents three common error types observed in the reranker that account for most general errors: (1) Saturated Scores: These errors occur when the ranker outputs overconfident scores (near 1.0) regardless of different inputs. Consequently, it becomes challenging to distinguish the best output as the differences among the scores are minimal. (2) Incorrect Scores: These errors arise when the ranker assigns higher scores to less relevant terms. As a result, the likelihood of selecting the most suitable samples decreases, leading to suboptimal generation. (3) Divergent Scores: These errors occur when the ranker exaggerates the differences among the inputs. This may be attributed to the inputs being hard samples, where none of them are closer to the target compared to each other, making it difficult for the reranker to discern the most appropriate option.Table 11 : Case study for HYDRA-reranker. The "Target" column indicates the ground-truth categorization, while "Gen" represents the category of the retrieved text. "Score" is the score generated by the reranker, and "Order" denotes the retriever's ranking during the retrieval process.Case ID Source Retrieved text Target Gen Order ScoreEleven places will each get at least $1 million to reduce their jail population.At least eight people have died at Hampton Roads Regional Jail in the past 17 months. politics politics 1 0.3525But he still could be the Republican with the best shot at stopping her. politics crime 2 0.3031 "Some information that he might find embarrassing needs to get out. Just to be fair."politics crime 3 0.2867 Case 2The name speaks for itself. These are of the same high quality of all their products are and the sizing is true to the fit.So far so good with this one. I would say this is definitely worth the extra expense compare to those oval shaped cream colored ones made out of aluminum that I am sure you have seen in numerous places ... These are okay, I have had better quality sheets in the past for less or the same price. These wrinkle easy but if that does not bother you then it is all cool. I do have problems keeping these on my bed, ... Following the RAG-based framework [41] and the PAG-based framework [38] , we have implemented the prompt design for the RAG and PAG-based baselines. The details of the prompt design can be found in Table 13 and Table 14 , respectively. We follow the RAG framework in HYDRA. Additional examples of RAG framework for each task (k = 2) are available as follows.RAG Prompt Demo for LaMP-2N (k=2) the category for the article: "a vaccine but instead of fighting off disease it attracts dogs" is "entertainment", and the category for the article: "The president, and many of his European counterparts, had condemned Trump as dangerous during his run." is "politics" Which category does this article relate to among the following categories? Just answer with the category name without further explanation. categories: [women, religion, politics, style & beauty, entertainment, culture & arts, sports, science & technology, travel, business, crime, education, healthy living, parents, food & drink] article: The guitarist will not play with the band on its upcoming tour.RAG Prompt Demo for LaMP-2M (k=2) the tag for the movie: "In a dystopian future, a totalitarian regime maintains peace by subduing the populace with a drug, and displays of emotion are punishable by death. A man in charge of enforcing the law rises to overthrow the system." is "dystopia", and the tag for the movie: "Cobb, a skilled thief who commits corporate espionage by Table 12 : Three common error types of the reranker. The "Target" column indicates the groundtruth categorization, while "Gen" represents the category of the retrieved text. "Score" is the score generated by the reranker.Source Retrieved text Target Gen Score Saturated Scores great quality and durability, also a great price, i would buy again, i really recommend to all my friends and to anyone, looking for this kind product, Good price, durable, very good product, i bought this to make my son birthday cake, come out nice, and he love it it!! i would buy again, and recommend to any one. 5 5 0.9657 excellent shoes, great product, great price, fast delivery, unfortunately i had to return this product, because was ordered wrong size, but the return transaction with amazon was very prompt and smooth, i would buy again, over all i would recommend to all my friends and to anybody :) infiltrating the subconscious of his targets is offered a chance to regain his old life as payment for a task considered to be impossible: \"inception\", the implantation of another person's idea into a target's subconscious." is "sci-fi" Which tag does this movie relate to among the following tags? Just answer with the tag name without further explanation. tags: [sci-fi, based on a book, Given this author's previous articles, try to describe a template for their headlines. I want to be able to accurately predict the headline given one of their articles. Be specific about their style and wording; don't tell me anything generic.Given this author's previous publications, try to describe a template for their titles. I want to be able to accurately predict the title of one of the papers from the abstract.Only generate the template description, nothing else.comedy, action, twist ending, dystopia, dark comedy, classic, psychology, fantasy, romance, thought-provoking, social commentary, violence, true story] description: A ticking-time-bomb insomniac and a slippery soap salesman channel primal male aggression into a shocking new form of therapy. Their concept catches on, with underground \"fight clubs\" forming in every town, until an eccentric gets in the way and ignites an out-of-control spiral toward oblivion.RAG Prompt Demo for LaMP-3 (k=2) 5 is the score for "Luckily, I am married. But I have already sent HIGH PRIORITY messages to all my single girlfriends that this book is MUST reading for understanding the male mind and knowing how to deal with it...all wrapped up in an energetic, yet sympathetic package (how nice that Matt seems to actually like women!). I especially enjoyed the tip boxes, and co-author Fadal's female wisdom. Fun!", and 5 is the score for "I received this book direct from the author. This has no effect on my review.\n\nThis is book 3. If you've read this far it means you love the series, and loved this book. You cannot read this one without reading book 1 and 2. Otherwise you would be lost trying to follow what happens. I loved this book. I just wish there was an epilogue to show more of what happens after everything. I know there is a book just about Gabriel so maybe that's my follow up to the story. Happy reading!" What is the score of the following review on a scale of 1 to 5? just answer with 1, 2, 3, 4, or 5 without further explanation. review: Colin Dodds does it again with Another Broken Wizard. A genius at evoking a mood, scene, and character Dodd elegantly weaves all three together in this story with equal weight. I have long been a fan of Dodds' poetry, and this novel reaches many poetic points in both his use of language and the intoxicating (no pun intended) way he manages to paint scenes with words so that very specific moods and nuanced emotion come alive. Dodds is truly a wordsmith, and he has created a host of characters, especially the main protagonists, that will be very hard to forget -mostly because we know these people, either because they dwell within us or because they live among us. Besides, it a damn good noir story about one of those forgotten New England towns, and it's full of blood, guts, grit, love, and loss. Hard to put down -highly recommended.RAG Prompt Demo for LaMP-4 (k=2) "Social Media Gone Awry: Tips for Teens to Stay Safe" is the title for "Here are a few tips to keep your teen safe when using the Internet and other web-based technologies. If you think it's an awkward conversation; you can hand them this blog to read.", and "If You See Something, Please Do Something to Prevent Child Abuse" is the title for "Although the age of social media has dramatically lowered the threshold on privacy standards, many adults are still reticent about reporting their suspicions about child abuse and neglect." Generate a headline for the following article: That explains how the Confederate flag, contraception and clean water deregulation got linked to fighting mosquitoes, the senator said.RAG Prompt Demo for LaMP-5 (k=2) "Accurate Estimators for Improving Minwise Hashing and b-Bit Minwise Hashing" is the title for "Minwise hashing is the standard technique in the context of search and databases for efficiently estimating set (e.g., high-dimensional 0/1 vector) similarities. Recently, b-bit minwise hashing was proposed which significantly improves upon the original minwise hashing in practice by storing only the lowest b bits of each hashed value, as opposed to using 64 bits. b-bit hashing is particularly effective in applications which mainly concern sets of high similarities (e.g., the resemblance >0.5). However, there are other important applications in which not just of high similarities matter. For example, many learning algorithms require all pairwise similarities and it is expected that only a small fraction of the pairs are similar. Furthermore, many applications care more about containment (e.g., how much one object is contain by another object) than the resemblance. In this paper, we show that the estimators for minwise hashing and b-bit minwise hashing used in the current practice can be systematically improved and the improvements are most significant for set pairs of low resemblance and high containment.", and "b-Bit Minwise Hashing for Large-Scale Linear SVM" is the title for "In this paper, we propose to (seamlessly) integrate b-bit minwise hashing with linear SVM to substantially improve the training (and testing) efficiency using much smaller memory, with essentially no loss of accuracy. Theoretically, we prove that the resemblance matrix, the minwise hashing matrix, and the b-bit minwise hashing matrix are all positive definite matrices (kernels). Interestingly, our proof for the positive definiteness of the b-bit minwise hashing kernel naturally suggests a simple strategy to integrate b-bit hashing with linear SVM. Our technique is particularly useful when the data can not fit in memory, which is an increasingly critical issue in large-scale machine learning. Our preliminary experimental results on a publicly available webspam dataset (350K samples and 16 million dimensions) verified the effectiveness of our algorithm. For example, the training time was reduced to merely a few seconds. In addition, our tech can be easily extended to many other linear and nonlinear machine learning applications such as logistic regression. " Generate a title for the following abstract of a paper: We propose skewed stable random projectionsfor approximating the \u03b1th frequency moments of dynamic data streams (0 < \u03b1 \ufffd 2). We show the sample complexity (number of projections) k = G 1 \u01eb2 log ' 2 \u03b4 \u00b4 , where G ! \u01eb 2 log(1+\u01eb) = O (\u01eb) as \u03b1 ! 1, i.e., \u03b1 = 1 \u00b1 \ufffd with \ufffd ! 0. Previous results based on symmetric stable random projections (12, 16) required G = non-zero constant + O(\u01eb), even when \ufffd = 0. The case \ufffd ! 0 is practically important. For example, \ufffd might be the \"decay rate\" or \"interest rate,\" which is usuall y small; and hence one might view skewed stable random projectionsas a \"generalized counter\" for estimating the total value in the future, taking in account of the effect of decaying or inter accruement. We consider the popular Turnstile data stream model. The input data stream at = (i, It) arriving sequentially describes the underlying signal A, meaning At(i) = At 1(i) + It, i 2 (1, D). We allow the increment It to be either positive (i.e., insertion) or negative (i.e., del etion). By definition, the \u03b1th frequency moment F(\u03b1) = PD i=1 |At(i)| \u03b1. Our method only requires that, at the time t for the evaluation, ANeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide all the experimental details in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We set the temperature of LLMs to zero for all baselines in order to ensure stable experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We have properly cited all code, data, and models we used in the paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://aka.ms/oai/additionalusecase 
 
 Appendix:  


Decision: The paper proposes a model factorization framework for black-box LLM personalization. The proposed framework consists of a reranker to retrieve relevant user history records, and an adapter to align the output with user preferences. Both reranker and adapter are decomposed into a base model for capturing global knowledge and multiple user-specific heads for personalization. 

The reviewers noted and mostly agreed on the novelty of the approach, clarify of the presentation and extensive experiments. 

The reviewers raised concerns on dependency on high-quality data, handling dynamically changing user preferences, training and inference efficiency, clear description of the base model etc.. The authors answered these questions during rebuttal. The authors are strongly encouraged to incorporate the analysis and results provided during the rebuttal and any other changes/updates needed into the final version to address the raised issues and further improve the paper. 

From the additional results, it seems the training and inference cost of the proposed model is very high, e.g., training on 100 users can take a few hours. The size of the proposed model is also very large and not scalable with number of users, e.g., the personalized header for one single user requires 0.6M parameters. Both efficiency and model size can be big concerns for practical use of the proposed model. These issues need to be addressed in either the revised version or in an extended version of the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces HYDRA, a model factorization framework designed to personalize large language models (LLMs) without modifying their internal parameters. HYDRA addresses the challenge of personalizing inherently opaque, black-box LLMs through a retrieval-augmented workflow. This method enhances personalization by using historical data to effectively capture user-specific preferences. HYDRA consists of two primary components: a personalized reranker that selects relevant information from the user's history, and a personalized adapter that tailors the LLM’s outputs to individual preferences. By integrating both global knowledge and local user behaviors, HYDRA improves personalization effectiveness. Extensive testing shows that HYDRA surpasses other state-of-the-art methods in personalization tasks, proving its ability to deliver tailored experiences without altering the fundamental model parameters.
strength_weakness: strength_weakness: 1) Limited Evaluation Metrics: The evaluation primarily emphasizes improvements in accuracy and benchmark performance. However, the use of the LAMP dataset to measure personalization is quite limited and fails to convincingly demonstrate effectiveness.
2) Dependency on High-Quality Data: HYDRA's performance is significantly dependent on the quality of historical user data. If the data is sparse or noisy, the functionality of both the reranker and adapter could be undermined. Exploring and integrating methods to effectively manage such data limitations is crucial.
3) Scalability Concerns: The discussion in the paper about the model's effectiveness covers only a limited number of users. Conducting scalability tests with much larger datasets and more diverse user bases could help in affirmatively validating the framework's efficiency on a larger scale.
4) User Privacy Concerns: HYDRA uses user-specific data, raising potential privacy issues.
5) Handling Dynamically Changing Preferences: How does HYDRA manage continuously evolving user preferences? Is there an efficient mechanism in place to update user models without extensive retraining?
6) Limitations of Reranker and Adapter: What are the constraints on the learning capacity and efficiency of the reranker and adapter modules? Are there situations where these components fail to achieve optimal results?
7) Integration Challenges: What are the main challenges in integrating HYDRA into existing large-scale systems, especially those that already have personalization frameworks in place?
questions: questions: 1) Limited Evaluation Metrics: The evaluation primarily emphasizes improvements in accuracy and benchmark performance. However, the use of the LAMP dataset to measure personalization is quite limited and fails to convincingly demonstrate effectiveness.
2) Dependency on High-Quality Data: HYDRA's performance is significantly dependent on the quality of historical user data. If the data is sparse or noisy, the functionality of both the reranker and adapter could be undermined. Exploring and integrating methods to effectively manage such data limitations is crucial.
3) Scalability Concerns: The discussion in the paper about the model's effectiveness covers only a limited number of users. Conducting scalability tests with much larger datasets and more diverse user bases could help in affirmatively validating the framework's efficiency on a larger scale.
4) User Privacy Concerns: HYDRA uses user-specific data, raising potential privacy issues.
5) Handling Dynamically Changing Preferences: How does HYDRA manage continuously evolving user preferences? Is there an efficient mechanism in place to update user models without extensive retraining?
6) Limitations of Reranker and Adapter: What are the constraints on the learning capacity and efficiency of the reranker and adapter modules? Are there situations where these components fail to achieve optimal results?
7) Integration Challenges: What are the main challenges in integrating HYDRA into existing large-scale systems, especially those that already have personalization frameworks in place?
limitations: limitations: NA
paper_summary: paper_summary: This paper provides a black-box LLM personalization framework that explores global and local knowledge from user’s historical behaviour through model factorization.
strength_weakness: strength_weakness: Weaknesses:

1.	The performance is not convincing. For example, in Table 1, it’s interesting to see that ICL-Random seems to be perform on par with HYDRA (second-best baseline on LaMP-2M, LaMP-5), and even better than RAG / PAG baselines. It’s not clear to me why using random items from user behaviour history is doing much better; please give more details.
2.	From the design until experiments, the authors should provide more details (or I may missed it?) about the tasks that benefit by shared (global), the tasks that benefit individual (local) preference, and the tasks that benefit from both. I consider model factorization is a big factor, but not much explanation and analyses were given for that part. In that way, it will further strengthen the paper 
3.	In addition, in Table 2, for example, HYDRA -P.-Adapter&Reranker has the same performance as HYDRA. Section 4.3 should provide more analyses and explanation about that. 
4.	Even though the authors provided implementation details, I still believe the source code should be released or more stronger analyses should be given, given the performance results I mentioned above.
questions: questions: Please refer to my concerns above.
limitations: limitations: The authors did provide details in Appendix A.
paper_summary: paper_summary: The paper proposes HYDRA, a learning-based model factorization framework that captures both user-specific and shared behavior patterns to enable effective personalization within black-box LLMs. The framework involves training a reranker to prioritize the most useful information from top-retrieved relevant historical records, and training an adapter to align the output with individual user-specific preferences, thereby eliminating the reliance on access to inherent model parameters of black-box LLMs. The experimental results demonstrate that HYDRA outperforms existing state-of-the-art prompt-based methods by achieving an average relative improvement of 9.01% across five diverse personalization tasks in the LaMP benchmark.
strength_weakness: strength_weakness: 1. The Algorithm Details in Appendix C indicate that HYDRA requires multiple stages of training. However, the paper does not provide an analysis of HYDRA's efficiency.
2. The paper only describes the user-specific head as a single layer of a feed-forward network, without providing details about the base model.
3. HYDRA-Adapter Inference utilizes an LLM to generate b candidates and selects the one with the highest score as the final answer. This approach substantially increases the inference time.
4. As the number of users increases, the number and parameters of user-specific heads also increase accordingly. This not only raises concerns about computational expenses but also introduces storage issues, potentially posing challenges for practical applications.
5. The effectiveness of HYDRA heavily depends on the quality and relevance of historical data, which may not always be available in the real world.
questions: questions: 1. Could you provide the time complexity analysis, training runtime, and inference time?
2. Could you provide a more detailed introduction of the base model?
3. HYDRA-Adapter Inference uses LLM to generate b candidates, then selects the one with the highest score as the final answer. Considering the inherent randomness in LLM generation, is this fair to baselines?
4. How does HYDRA perform in scenarios where there are significant disparities in user behavior history, such as some users having dense histories while others have sparse ones?
limitations: limitations: 1. HYDRA-Adapter Inference utilizes an LLM to generate b candidates and selects the one with the highest score as the final answer. This approach substantially increases the inference time.
2. As the number of users increases, the number and parameters of user-specific heads also increase accordingly. This not only raises concerns about computational expenses but also introduces storage issues, potentially posing challenges for practical applications.
3. The effectiveness of HYDRA heavily depends on the quality and relevance of historical data, which may not always be available in the real world.

Comments: title: Thank You
Increase the score
A Gentle Reminder
Thank You
A Gentle Reminder
A gentle reminder
Acknowledgement
Additional Details of Time Complexity Analysis
comment: Dear Reviewer KhW9,

Thank you very much for taking the time to review our rebuttal and offering insightful feedback. We will update our paper with the additional results and discussions. In addition, we will provide a detailed README file to the source code upon release and a project page for detailed introduction. 

Many thanks again!

Best Regards,   
Authors
Thanks for your response in details. I read all the responses and revisit the paper. I'll increase the score to weak accept. On a side note, please add README file to the source code upon release for reproducibility. Thank you.
Dear Reviewer KhW9，

Thank you again for your valuable feedback. As we have mentioned in the general comments, we would like to kindly remind you that the author/reviewer discussion phase ends by Aug 13th. Aside from the responses we have offered previously, we have **also offered more comprehensive experimental results and rigorous analyses in the attached 1-page PDF file**. We sincerely hope that our responses have enhanced the paper's quality and addressed your concerns. If there are any additional suggestions or comments you would like to provide, please don't hesitate to share them. We look forward to engaging in a constructive discussion during the rebuttal phase. Thank you!

Best Regards,   
Authors
Dear Reviewer AVA9,
 
Thank you very much for taking the time to review our rebuttal and offering insightful feedback. We will update our paper with the additional results and discussions.
 
Best Regards,   
Authors
Dear Reviewer xKC3，

Thank you again for your valuable feedback. As we have mentioned in the general comments, we would like to kindly remind you that the author/reviewer discussion phase ends by Aug 13th. Aside from the responses we have offered previously, we have **also offered more comprehensive experimental results and rigorous analyses in the attached 1-page PDF file**. We sincerely hope that our responses have enhanced the paper's quality and addressed your concerns. If there are any additional suggestions or comments you would like to provide, please don't hesitate to share them. We look forward to engaging in a constructive discussion during the rebuttal phase. Thank you! 

Best Regards,  
Authors
Dear Reviewers,

Thank you again for your valuable feedback. We would like to kindly remind you that the author/reviewer discussion phase ends soon. We sincerely hope that our responses have enhanced the paper's quality and addressed your concerns. If there are any additional suggestions or comments you would like to provide, please don't hesitate to share them. We look forward to engaging in a constructive discussion during the rebuttal phase. Thank you!

Best Regards,

Authors
I thank the authors for their further explanation, and I would like to keep my score.
We conduct time complexity analysis as follows: 

Assume that we have $N_{train}$ users in the training data and $N_{test}$ users in the test set. We adopt the transformer architecture, specifically the Longformer-base, as the reranker and the adapter base models. Consequently, the time complexity of all stages should be proportional to that of the attention mechanism in transformers, $O(dL^2)$, where $L$ indicates the sequence length and $d$ indicates the hidden dimension. Additionally, the training process will go through the transformer for $T$ epochs, while inference only requires one.

- For each user in the HYDRA-Reranker training data, we augment $M$ random historical records from the user’s corresponding profile. The retriever then retrieves the top-$M$ ($M=20$ by default) relevant historical records to form training samples. Thus, for each user, we collect $M^2+1$ training samples.

- For each user in the HYPER-Adapter training data, we consider all the user historical records. For each record, we leverage model randomness to generate $k$ ($k=8$ by default) samples for the adapter to select. Consequently, we can have $\bar{H}k$ training samples per user, where $\bar{H}$ denotes the average number of histories per user. 

Therefore, the time complexity for the different stages is as follows:

| Method | Mode | Time Complexity | 
|:--------|:-------|:------------------|
| HYDRA-Reranker | Training | $O(N_{train}(M^2+1)Td(L^2))$ |
| HYDRA-Reranker | Fit New User | $O(N_{test}(M^2+1)Td(L^2))$ |
| HYDRA-Reranker | Inference | $O(N_{test}d(L^2))$ |
| HYDRA-Adapter | Training | $O(N_{train}\bar{H}kdT(L^2))$ |
| HYDRA-Adapter | Fit New User | $O(N_{test}\bar{H}kTd(L^2))$ |
| HYDRA-Adapter | Inference | $O(N_{test}kd(L^2))$ |

We then proceed to empirically examine the efficiency with respect to training and inference time consumption on 100 training users and 50 test users:

| Method | Mode | LaMP-2N | LaMP-2M | LaMP-3 | LaMP-4 | LaMP-5 |
|:--------|:-------|:------------------:|:----------------------:|:----------:|:-----------:|:-----------:|
| HYDRA-Reranker |  Training | 31m10s | 41m51s | 50m37s | 1h1m31s | 1h8m16s |  
| HYDRA-Reranker | Fit New User | 18m8s | 21m17s | 25m36s | 33m52s | 31m25s | 
| HYDRA-Reranker | Inference | 3m4s | 3m1s | 3m7s | 4m38s | 5m14s |
| HYDRA-Adapter | Training | 1h10m17s | 2h2m16s | 2h1m59s | 3h56m47s | 3h19m42s |
| HYDRA-Adapter | Fit New User | 28m15s | 1h7m27s | 1h19s | 2h23m10s | 1h59m2s | 
| HYDRA-Adapter | Inference | 4m16s | 4m17s | 4m17s | 5m53s | 5m59s | 

It is important to note that although current calculations are based on centralized computing, the fitting process for new users and inference can be distributed to user-end machines. This allows for simultaneous updates of different head models. The potential of parallelism and distributed training can further enhance the efficiency of HYDRA.