Personality Alignment of Large Language Models

## Abstract

Abstract Current methods for aligning large language models (LLMs) typically aim to reflect general human values and behaviors, but they often fail to capture the unique characteristics and preferences of individual users. To address this gap, we introduce the concept of Personality Alignment. This approach tailors LLMs’ responses and decisions to match the specific preferences of individual users or closely related groups. Inspired by psychometrics, we created the Personality Alignment with Personality Inventories (PAPI) dataset, which includes data from 300,000 real subjects, each providing behavioral preferences based on the Big Five Personality Factors. This dataset allows us to quantitatively evaluate the extent to which LLMs can align with each subject’s behavioral patterns. Recognizing the challenges of personality alignments—such as limited personal data, diverse preferences, and scalability requirements—we developed an activation intervention optimization method. This method enhances LLMs’ ability to efficiently align with individual behavioral preferences using minimal data and computational resources. Remarkably, our method, PAS, achieves superior performance while requiring only 1/5 of the optimization time compared to DPO, offering practical value for personality alignment. Our work paves the way for future AI systems to make decisions and reason in truly personality ways, enhancing the relevance and meaning of AI interactions for each user and advancing human-centered artificial intelligence. The code has released in https://github.com/zhu-minjun/PAlign .

## 1 Introduction

The alignment with human preferences has been a focal point of both theoretical and applied research for AI systems (Shen et al., ; Ji et al., ; Hendrycks et al., ). However, existing alignment approaches often treat society values as a monolithic construct, lacking the granularity needed to cater to individual differences. This broad approach may overlook the nuanced preference of individual users or closely related groups, leading to a one-size-fits-all model that fails to effectively serve diverse user needs (Fernandes et al., ; Carranza et al., ; van Wynsberghe, ). This gap is particularly critical as personality alignment can enhance user satisfaction, trust, and engagement by making interactions more relevant and meaningful. As shown in Figure 1, we propose "Personality Alignment":

> Aligning AI behavior to match an individual’s unique preferences and priorities of specific individuals, mirroring their behavior, thinking, and decision-making. It ensures AI responses not only are precise but also resonate with the user’s traits, understanding their communication nuances and preferences.

As large language models (LLMs) advance (Solaiman et al., ; Farina & Lavazza, ), AI systems increasingly exhibit societal values and human-like personality traits (Ng et al., ; Bai et al., ; Durmus et al., ; Zhang et al., ; tse Huang et al., ; Jang et al., ). The Role-playing tasks (Li et al., ; Salemi et al., ; Gupta et al., ; Yu et al., ) highlight this trend. However, human preferences are complex, every individual has unique preferences and needs that must be considered to effectively meet user expectations in real-world applications (Pfeifer & Bongard, ). For example, in customer service, some users prioritize politeness, while others prefer efficiency. Therefore, personalization is crucial for aligning AI systems with user preferences. Traditional Role-Playing Agents simulate specific professions or roles, rather than deeply aligning with individual preferences (Chen et al., ; Kulveit, ; Hubinger et al., ; Stray, ). The lack of large-scale, theoretically supported datasets for aligning AI with diverse human personalities and preferences underscores the need for further research in personality and personality alignment (Jang et al., ; Lou et al., ; Li et al., ).

There remains a practical challenge: aligning a model like Llama-2 typically requires around 3 million sentences (Touvron et al., ). For personality alignment, collecting such an enormous set of behavioral data from a large number of individuals is both expensive and unrealistic. Instead, we propose a training-free personal-alignment paradigm, which provides over 300,000 unique preference samples, and a method efficiently aligning individual preference characteristics during the model’s forward computation. Specifically, we utilize the IPIP-NEO-120 questionnaire (Johnson, ), including 120 multiple-choice questions that quantify individual behavior patterns across five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (De Raad, ; Roccas et al., ). This questionnaire allows subjects to provide a detailed measure of their personality and behavior within 10-20 minutes, making personality alignment both feasible and efficient (Aluja & Garcia, ). We collect preference data from over 300,000 subjects who complete two questionnaires: the IPIP-NEO-120 and the IPIP-NEO-300 (Goldberg et al., ). The IPIP-NEO-300, a superset of the IPIP-NEO-120, offers a broader range of behavioral descriptions with 300 multiple-choice questions. We manipulate the original dataset to build the Personality Alignment with Personality Inventories (PAPI) dataset, requiring LLMs to learn from each person’s IPIP-NEO-120 answers and generalize to IPIP-NEO-300 questionnaire. We then selected 300 subjects as a representative Test-set and released the remaining data as a development set, providing a novel evaluation framework and metrics for evaluating LLMs’ personality alignment fairly.

Figure: Figure 1: On the left, AI aligns to broad human values like helpfulness, honesty, and harmlessness using a standard set. On the right, the focus is on aligning AI behavior with individual users’ specific traits and preferences, using detailed profiles to reflect unique personal values.
Refer to caption: x1.png

To manage the behaviors of millions, the system must be scalable, ensuring quality and efficient alignment. Traditional RLHF techniques, which lack data and computational efficiency (Yao et al., ), are unsuitable for this task. Similarly, Prompt Engineering, though efficiently, often varies too greatly between models to be reliable for precise alignment (Zhou et al., ). Inspired by activation engineering (Turner et al., ), we further modify the inference-time intervention (ITI)(Li et al., ) by simultaneously searching for the activation value offset direction and offset distance across multiple personality dimensions. We propose a simple and effective personality alignment method: Personality Activation Search. PAS identifies the most effective attention heads in language models (LLMs) using established behavioral patterns that correlate with the five major personality traits. It then adjusts the activations in these specific directions and optimizes the magnitude of movement to achieve the best results. This approach is computationally efficient because it doesn’t rely on backpropagation to modify model parameters. Instead, PAS offers a minimally invasive intervention, resulting in greater data efficiency and more controlled personality alignment than traditional Prompt Engineering (Jiang et al., ).

We present a novel systematic and comprehensive personality alignment framework that enables LLMs to understand and align precisely with specific values, preferences, and behaviors required. Compared to aligning with universal values, personality alignment requires LLMs to align with individual values, preferences, and behaviors, which ensures that the LLMs can accurately match or mimic the specific individual characteristics that users need and pursue. To achieve this goal, we introduce the PAPI dataset, which covers over 300,000 subjects of different ages, genders, and periods using a multiple-choice question format to measure the alignment results. Based on that, we propose the PAS method, which aims to address the challenges of scarce individual characteristic data and high scalability in personality alignment. Experiments have shown its high alignment efficiency compared to DPO and PPO (Schulman et al., ; Rafailov et al., ) where it requires only 1/6 of the time to achieve superior performance. It even outperforms the GPT-4o model using PAS based on the Llama-3-8B model.

## 2 Related Work

Alignment with language models. The concept of integrating human values into AI systems, initially proposed by Dewey Dewey (), underscores the necessity of value learning, which entails the AI’s ability to process and prioritize a broad spectrum of human values and preferences. In the context of LLMs—functioning as high-performance assistants—aligning these models with human values is crucial, especially since they operate in complex reasoning tasks across various domains (Hadar-Shoval et al., ; Wei et al., ; ; Weng et al., ). Methods such as RLHF (Stiennon et al., ; Bai et al., ; Dong et al., ) and RLAIF (Lee et al., ; Sharma et al., ) seek to align LLMs by employing strategies like PPO (Schulman et al., ) and DPO (Rafailov et al., ). These approaches focus on maximizing cumulative rewards that are closely aligned with predefined human values (Azar et al., ; Lin et al., ; Dai et al., ; Swamy et al., ; Dai et al., ). Contrary to these broad approaches, our research focuses on personality alignment, which tailors the LLMs’ behavior to each individual’s complex value set, thereby enhancing the model’s capacity to understand and reflect individual human values.

Preferences of language models. The community of LLMs has started using personality testing tools for both qualitative and quantitative evaluations (Hoover et al., ; Jiang et al., ; Qiu et al., ; Xu et al., ; Salemi et al., ; Wang et al., ). For example, the Machine Personality Inventory (MPI) Jiang et al. (), utilizes the Big-Five Personality Factors to standardize the assessment of LLMs. This approach analyzes the behaviors of LLMs across the five dimensions of the Big-Five, providing a quantitative interpretation of the values and personality traits of LLMs. Such tools are grounded in widely accepted theories, including the Big-Five and the 16 Personality Factors (Cattell & Mead, ), helping to contextualize LLMs’ personality traits in terms of human individual differences (Gupta et al., ; Wang et al., ; Aher et al., ; Liu et al., ; Li et al., ). These studies indicate that aligned LLMs share similar values with humans (Mehta et al., ; Pan & Zeng, ; Mao et al., ; Go et al., ; Chen et al., ). Jang et al. () has also mentioned the similar term "personality alignment", which considers a limited scope of preference personalization (Wang et al., ; Zhuang et al., ; Zeng et al., ), focusing on only 8 different conflicting user preferences derived from combinations of three dimensions: Expertise (elementary vs. expert), Informativeness (concise vs. informative), and Style (friendly vs. unfriendly). Our work extends these theories and methods by leveraging psychological measurement to assess the value alignment gap between aligned LLMs and individual users. We aim to understand how closely aligned LLMs are with individual values through personality and personalized adjustments.

Interventions Activate. Activation interventions in neural networks trigger specific changes in internal activations (Olsson et al., ; Wu et al., ) and have emerged as a pivotal tool in model robustness (He et al., ), editing (Meng et al., ), circuit finding (Goldowsky-Dill et al., ), and knowledge tracing (Geva et al., ). These interventions are advantageous due to their adjustable nature and minimal invasiveness (Wu et al., ). Prior research has demonstrated that activation interventions can effectively identify the crucial heads and directions in a model’s internal representations (Burns et al., ; Li et al., ; Liu et al., ), which offers better data efficiency and faster optimization than traditional reinforcement learning approaches (Turner et al., ; Rimsky et al., ; Shi et al., ; Weng et al., ). Building on these findings, our proposed method enhances scalability beyond traditional reinforcement learning by pinpointing the value directions of different individuals within the model and calculating the optimal offset distance to achieve alignment.

## 3 Personality Alignment Dataset Construction

We introduce dataset construction process and evaluation methods to democratise research on Personality Alignment. Detailed analysis of the dataset is provided in Appendix G.

Figure: Figure 2: Overview of the PAPI dataset. (a) Illustrates the comparison between the subject’s self-assessment and the AI’s assessment of a specific question. (b) Shows an example of the IPIP-NEO-120 and IPIP-NEO-300 questionnaire responses. (c) Depicts the Big Five personality traits profile.
Refer to caption: x2.png

Motivation. In order to systematically evaluate the consistency between Personality-aligned LLMs and human preferences, we create a large-scale real-world Personality Alignment with Personality Inventories (PAPI) dataset, which includes 307,313 samples stored in multiple-choice questionnaires. Each sample contains a participant’s answers to the psychological questionnaires, which is shown in Figure 2. Each sample contains an IPIP-NEO-120 questionnaire and an IPIP-NEO-300 questionnaire, similar to (b). Their collection process comes from questions similar to (a), which require each subject to choose behaviors that are similar to their own. Finally, we can synthesize a personality trend analysis chart for a subject based on all the questions, as shown in (c).

Question Type. Each question is associated with one of the five major personality dimensions and correlates positively or negatively, which describes a specific behavioral pattern or preference, with five response options ranging from “Very Accurate” to “Very Inaccurate.” We expect aligned model can clearly articulate its views on these descriptions. Given the stability of individual personality tendencies, consistent responses are expected for the same personality dimension descriptions. In particular, we use each subject’s answers to the IPIP-NEO-120 questionnaire for training, and their answers to the IPIP-NEO-300 questionnaire for testing. This setup aims to measure whether the model can generalize from known behavior descriptions and personality tendencies to a broader range of behaviors, thus aligning with specific preferences.

Figure: Figure 3: Visualization of the K-Means Clustering on the PAPI Dataset. The centroids are marked in red, demonstrating the central point of each cluster, while the closest samples to these centroids are highlighted in gold.
Refer to caption: x3.png

Data Collection. We have collected 307,313 samples based on the IPIP-NEO-120 and IPIP-NEO-300 from the International Personality Item Pool (IPIP)(^1^11[https://ipip.ori.org/index.htm](https://ipip.ori.org/index.htm)). This includes approximately 60% (185,149) female and 40% (122,164) male participants, covering ages from 10 to 99 years (average age 25.19 years). The data were collected through online surveys on a website, requiring participants to provide genuine feedback on 120 & 300 behavioral questions. Participants are volunteers from various industries worldwide, including the US, UK, France, India, and China, with tests conducted between 1998 and 2019. Each participant had to actively acknowledge that careless responses would invalidate the usefulness. Each survey took between 30 to 50 minutes, was anonymous, and collected no traceable or socioeconomic status-related data. We have permission from the administrators to use IPIP items, scales, and inventories for any purpose, commercial or non-commercial.

Data Processing. To streamline the evaluation of human personality traits within a vast dataset, we apply K-Means clustering to the PAPI dataset, which contains 307,313 samples. We reduce this dataset into 300 representative clusters as Test-Set, and release the remaining data as Dev-Set. We achieve this by identifying and selecting samples closest to each cluster’s centroid, which are the most representative of their respective groups. This selection process is mathematically described as:

$$ $\min_{x\in S_{k}}|x-\mu_{k}|_{2}$ (1) $$

Here, $\mu_{k}$ is the centroid of cluster $k$, and $x$ is a sample within cluster $k$ ($S_{k}$). This approach minimizes the Euclidean distance between selected samples and their cluster centroids, optimizing the representativeness of each cluster in the dataset.

Evaluation Methods. Inspired by the MPI (Jiang et al., ), we set up a behavioral difference score as an evaluation metric. Using a psychological measurement approach, we establish a simple scoring function S: for positively correlated questions, we assign scores from 5 to 1 from “Very Accurate” to “Very Inaccurate”. We expect the aligned language model scores to be close to a specific individual’s scores, hence we use an absolute value method for measurement. We test each trait $d\in\{\text{Agreeableness},\text{Conscientiousness},\text{Extraversion},\text{
Neuroticsim},\text{Openness}\}$ in the test section, with each participant’s behavioral difference Aligned Score calculated as follows:

$$ $\text{Aligned Score}_{d}=\frac{1}{M}\sum_{i=1}^{M}\left(\frac{1}{N_{d,i}}\sum_ {\kappa\in D^{Test}_{d,i}}\left|f\left(\text{LLM}(\kappa,\text{template}) \right)-f\left(\text{Person}(\kappa,\text{template})\right)\right|\right),$ (2) $$

In this formula, $D^{Test}_{d,i}$ (the answer of IPIP-NEO-300) denotes the set of behaviors related to trait $d$ in the test section for subject $i$. We use the given template to obtain responses from both the LLM and the Person, and then calculate the absolute difference between the two. $N_{d,i}$ represents the count of these behaviors. Each $\kappa$ is an independent behavior sample from this set, with the LLM’s responses evaluated against those in $D^{Train}$ (the answer of IPIP-NEO-120). The $f(.)$ represents the scoring function for multiple-choice questions. We calculated the deviation distance between the two using the absolute value method and computed the average deviation distance for all behaviors. The range for Score is from 0 to 5, where 0 means perfect alignment with individual behavior.

## 4 Personality Activation Search

Figure: Figure 4: Overview of the PAS Process. Step 1 involves selecting a LLM (pre-training, SFT, or RLHF). Step 2 includes creating and activating personality responses using the answer of questionnaire, the subject’s answer in the PAPI dataset, and probing each attention head. Step 3 integrates these activations for personality alignment, adjusting model outputs to reflect individual user traits and preferences, resulting in an aligned LLM.
Refer to caption: x4.png

Popular alignment techniques like PPO, DPO, and IPO strive to align language models with desired behaviors, but personalizing LLMs for individual preferences without extensive training presents a significant challenge. To address this, we’ve developed Personality-Activation Search (PAS), an efficient method that fine-tunes model activations to closely match human preferences. PAS identifies key activations within the model that significantly impact Personality and personalized behaviors and adjusts these activations toward specific preferences.

PAS works by identifying multiple directions in the transformer’s activation space that satisfy personality attributes and then intervening along these directions to adjust the activation values, achieving minimally invasive alignment. The entire process does not require changing the model’s weights or backpropagation. It only requires multiple forward passes to determine the offset direction and distance, resulting in a high optimization efficiency.

### 4.1 Language Models

In our research, we focus on a transformer-based language model which may be in various stages, including pre-training, supervised fine-tuning, or undergoing RLHF. The transformer model primarily consists of several layers, each equipped with a multi-head attention (MHA) mechanism and a multilayer perceptron (MLP) (Vaswani et al., ).

During inference, input tokens are encoded to high-dimensional embeddings. In each transformer layer, the MHA process is defined as:

$$ $x_{l+1}=x_{l}+\sum_{h=1}^{H}Q_{l}^{h}\operatorname{Att}_{l}^{h}(P_{l}^{h}x_{l}),$ (3) $$

where H is heads number, $x_{l}\in\mathcal{R}^{DH}$ represents the activations. $P_{l}^{h}\in\mathcal{R}^{{D}\times{DH}}$ projects activations into a specific headspace, and $Q_{l}^{h}\in\mathcal{R}^{{DH}\times{D}}$ reprojects them back, facilitating interaction among input tokens.

Considering past work, such as ITI, which intervenes in language models to influence their output bias in specific directions, they tend to shift a large and fixed distance in a particular direction. However, this approach becomes unusable when facing Alignment issues. We have improved the distance selection strategy for intervention. We dynamically add preference vectors to this residual stream, which is after Att and before $Q_{l}^{h}$.

### 4.2 Search the Directions for Activation Intervention

To achieve personality alignment, we trained “probes” to get activations as the direction of personal preference. It is started by selecting statements from the training dataset of each subject, formatted as positive samples: “Question: Given a statement of you: ‘$Statement’, Do you agree? Answer: Yes”, and negative samples: “Question: Given a statement of you: ‘$Statement’, Do you agree? Answer: No”. Activations from each head are retained and split into a training set (60%) and a validation set (40%). These activations serve as features for the probe (Equation 4) of per attention head.

$$ $p_{\theta}(x_{l}^{h})=\operatorname{sigmoid}(\langle\theta,x_{l}^{h}\rangle),$ (4) $$

where the $x_{l}^{h}\in\mathcal{R}^{D}$, and $\theta\in\mathcal{R}^{D}$ is the weights of h-heads in layer l in the residual stream.

We identify the most influential heads based on their predictive accuracy during validation. Notably, it is necessary to disperse interventions during this process (Li et al., ). Therefore, we select the top K probabilities for intervention.
The parameters $\theta_{l}^{h}$ from these probes, after normalization, are treated as the directions along which behaviors are most separable. These directions are the most informative for aligning the model with individual preferences (Burns et al., ).

During inference, we integrate the intervention into the output of each selected head:

$$ $x_{l+1}=x_{l}+\sum_{h=1}^{H}Q_{l}^{h}\left(\operatorname{Att}_{l}^{h}(P_{l}^{h }x_{l})+\alpha\sigma_{l}^{h}\right),$ (5) $$

where $\alpha$ controls the adjustment intensity, and $\sigma_{l}^{h}$ is the standard vectors along the normalized direction $\theta_{l}^{h}$. We adjust activations in the personality direction, focusing only on the top $K$ heads, while leaving the unselected heads unchanged. In practice, the model parameters remain unaltered; we only need to store approximately 1,000 parameters, denoted as $\sigma_{l}^{h}$. These $\sigma_{l}^{h}$ parameters represent the direction of activation shifts towards personality preferences, rather than memorizing specific preferences. This approach is highly parameter-efficient, as the $\sigma_{l}^{h}$ parameters constitute only about 1/10,000,000 of the total parameters compared to a full-parameter model.

### 4.3 Search Distance for Activation Intervention

To optimize the intervention’s impact on personality behaviors in the language model, we search to determine the optimal value of $\alpha$ precisely. This method is suitable for functions with a single peak or trough. The search, constrained within the interval $[0,10]$ (^2^22Based on empirical observations, $\alpha$ exceeding 10 tend to cause computational overflow in the model.), utilizes the objective function $f(\alpha)$ to compute the negative aggregate score, aiming to minimize this value. This approach efficiently identifies the $\alpha$ that best enhances alignment with desired behaviors while maintaining the model’s internal coherence, ensuring adjustments remain within a beneficial range.

$$ $\text{Optimal }\alpha=\text{argmin}_{\alpha\in[0,10]}f(\alpha).$ (6) $$

By iteratively narrowing the search range based on the function’s values at each step, this method efficiently locates the $\alpha$ that yields the lowest mean Score.

## 5 Experiments

**Table 1: Comparison of alignment methods on the PAPI dataset using the Aligned Score of Big-Five personality traits. The Composite Score represents the overall specifics of the five dimensions. It is derived from the combined scores of these five dimensions.**
| Method | Alignment Mode | The Aligned Score of Big-Five | Composite Score |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Agreeableness $\downarrow$ | Conscientiousness$\downarrow$ | Extraversion $\downarrow$ | Neuroticism $\downarrow$ | Openness $\downarrow$ |  |  |  |
| GPT-4o(omni) |  |  |  |  |  |  |  |
| Few-Shot | Black-Box (Prompt-Based) | 1.02 | 0.83 | 0.81 | 0.80 | 0.96 | 4.42 |
| $P^{2}$ | Black-Box (Prompt-Based) | 1.44 | 1.45 | 1.63 | 1.73 | 1.46 | 7.71 |
| Llama-3-8B-Instruct |  |  |  |  |  |  |  |
| PPO | White-Box (Alignment) | 1.63 | 1.51 | 1.45 | 1.42 | 1.61 | 7.62 |
| DPO | White-Box (Alignment) | 1.54 | 1.42 | 1.54 | 1.74 | 1.21 | 7.45 |
| Few-Shot | Black-Box (Prompt-Based) | 1.28 | 1.30 | 1.40 | 1.09 | 0.89 | 5.96 |
| $P^{2}$ | Black-Box (Prompt-Based) | 1.39 | 1.33 | 1.41 | 1.22 | 1.68 | 7.03 |
| PAS (Ours) | White-Box (Alignment) | 0.94 | 0.91 | 0.86 | 0.98 | 0.72 | 4.41 |
| Llama-3-70B-Instruct |  |  |  |  |  |  |  |
| PPO | White-Box (Alignment) | 1.56 | 1.59 | 1.43 | 1.40 | 1.56 | 7.54 |
| DPO | White-Box (Alignment) | 1.46 | 1.25 | 1.45 | 1.48 | 1.57 | 7.21 |
| Few-Shot | Black-Box (Prompt-Based) | 1.06 | 0.94 | 0.96 | 1.03 | 1.22 | 5.21 |
| $P^{2}$ | Black-Box (Prompt-Based) | 1.42 | 1.33 | 1.36 | 1.35 | 1.66 | 7.12 |
| PAS (Ours) | White-Box (Alignment) | 0.98 | 0.89 | 0.87 | 1.01 | 0.99 | 4.74 |

### 5.1 Experimental Settings

Personality Alignment. We select four classic methods as baselines to compare against the PAS approach, which includes two black-box methods: Few-Shot and Personality Prompt ($P^{2}$) Jiang et al. () relying on in-context learning (ICL), and two white-box methods requiring training, PPO (Schulman et al., ) and DPO (Rafailov et al., ). We consider recently released Llama-3-8B-Instruct and Llama-3-70B-Instruct (Touvron et al., ; ; AI, ) model as backbones, and SoTA model, GPT-4o (omni) (OpenAI, ) as baseline. The scores in the table represent the distance of alignment calculated by Eq. 4, with lower scores indicating better alignment.

Open-ended Generation.
We conduct experiments on Llama-3-8B-Instruct and Llama-3-70B-Instruct. For the open-ended generation task, we use GPT-4o as the annotator. First, we align the language model to a specific individual (ID 181591, 234619, 210204, 259396, and 22835 from the Test-Set). Then, we have the model generate open-ended responses based on each statement from the IPIP-NEO-300. We compare our method with ICL methods and classical alignment methods.

Complex Reasoning. We explore the generalization capabilities of the PAS method by evaluating its impact on complex reasoning tasks across eight datasets: GSM8K, CommonSenseQA, AddSub, MultiArith, SVAMP, BigBench-Date, StrategyQA, and Coin Flip (Hosseini et al., ; Talmor et al., ; Arkil et al., ; Cobbe et al., ; Suzgun et al., ; Roy & Roth, ; Wei et al., ; Kojima et al., ; Weng et al., ). In this part, we expect to assess how conscientiousness adjustments influence the LLM’s performance in reasoning tasks. Using the llama-3-8B-Instruct model, we apply PAS to adjust the model’s activations to simulate two fictional personas: one with extreme conscientiousness (as Positive) and one without extreme conscientiousness (as Negative). Vanilla CoT served as the baseline.

### 5.2 Results on PAPI

Figure: Figure 5: Comparison of performance and efficiency for various alignment methods. The larger the multiple on the x-axis, the lower the efficiency.
Refer to caption: x5.png

Table 1 shows that the PAS method achieves state-of-the-art performance across different models. For the Llama-3-8B-Instruct model, PAS achieves a Composite Score of 4.41, significantly better than the next best method’s 5.96. Similarly, for the Llama-3-70B-Instruct model, PAS scores 4.74, compared to 5.21 for the second-best method. Notably, PAS surpasses GPT-4o (omni) in the black-box prompt-based scenario, which scores 4.42. PAS also excels in individual Big-Five dimensions, achieving the lowest scores for Agreeableness (0.94 for 8B, 0.98 for 70B), Conscientiousness (0.91 for 8B, 0.89 for 70B), Extraversion (0.86 for 8B, 0.87 for 70B), Neuroticism (0.98 for 8B, 1.01 for 70B), and Openness (0.72 for 8B, 0.99 for 70B). In the appendix E.4, we also conducted further performance comparisons with full parameter fine-tuning and LoRA. PAS similarly demonstrated higher efficiency and superior performance.

Efficiency. As shown in Figure 5, training a separate language model for each individual demands (e.g. DPO and PPO) substantial computational resources and time, making it an infeasible solution for personality alignment. PAS addresses efficiency challenges by making targeted adjustments to a small subset of activation values during inference, in specific directions. This approach requires minimal computational overhead, similar to Few-Shot methods, and only about 1/6 of the time needed for PPO, yet achieves remarkable personality alignment results. By intervening at the activation level rather than relying on extensive training or the variability of prompts, PAS provides a more efficient and precise method for aligning language models with individual user preferences.

Alignment vs. ICL. Our experiments highlight that direct Alignment (White-Box) methods, especially PAS, outperform In-Context Learning (ICL) (Black-Box) methods. By aligning model parameters with user preferences, PAS provides better performance compared to prompt-based approaches. The PAS method’s ability to efficiently shift activations towards user preferences without altering the model’s core parameters gives it a significant advantage over traditional ICL methods.

Why Did Scaling Laws Fail? The results indicate that scaling laws alone do not ensure optimal alignment. Although larger models like Llama-3-70B-Instruct benefit from increased scale, PAS’s performance demonstrates that Alignment by intervention without modifying parameters can yield better results. This suggests that targeted approaches like PAS are more effective in addressing individual user preferences, highlighting the necessity of methods focused on personality alignment rather than relying solely on model size. The success of PAS in outperforming larger models with fewer parameters underscores the importance of efficient and precise alignment techniques.

### 5.3 Generalization Results

Figure: Figure 6: The win rates of PAS compared to classical alignment methods evaluated by GPT-4o.
Refer to caption: x6.png

Considering that during the process of adjusting the model, although specific goals can be achieved, it may also affect other general capabilities of the model, such as dialogue and reasoning abilities. We demonstrate through general dialogue tasks and complex reasoning tasks that our method can precisely control the model’s personality without compromising its other generalization capabilities.

Open-Ended Generation. We used an open-ended generation task where the language model elaborated on given statements from the IPIP-NEO-300, covering “Scenario Description,” “Actions Taken,” “Thought Process,” and “Emotional Response.” The generated outputs were evaluated by GPT-4o for alignment with individual preferences, similar to the evaluation process in LIMA (Zhou et al., ). Despite the Personality Alignment being initially trained with multiple-choice options, Figure 6 shows that PAS does not affect the model’s general conversational ability and reasoning capacity, it significantly outperforms baseline methods in this open-ended task. This superior performance can be attributed to PAS’s ability to fine-tune model activations to capture individual preferences effectively, enhancing the model’s adaptability to complex and nuanced tasks. PAS’s ability to generalize from multiple-choice training data to open-ended generation tasks demonstrates its robustness in capturing broader behavioral nuances. In addition to this, we conducted a manual evaluation using the same setup. The results of this experiment can be found in Appendix E.2.

Figure: Figure 7: Performance of the llama-3-8B-Instruct model on complex reasoning tasks across eight datasets. We used Vanilla CoT (Wei et al., ) as the baseline (dashed line in each subplot), and then applied PAS to simulate two fictional personas: one with extreme conscientiousness (Positive) and one without extreme conscientiousness (Negative).
Refer to caption: x7.png

Complex Reasoning. Experimental results, as shown in Figure 7, indicate that PAS significantly enhances the model’s reasoning capabilities when conscientiousness is positively adjusted. For instance, on the GSM8K dataset, the baseline performance was 73.47. Enhancing conscientiousness increased the performance to 74.15 while reducing conscientiousness dropped it to 72.75. Similar trends were observed across other datasets, with the model consistently outperforming the baseline when conscientiousness was increased. This suggests that conscientiousness, associated with diligence and carefulness, positively impacts the model’s ability to handle complex reasoning tasks. Importantly, these interventions did not significantly alter the inference time, maintaining efficiency comparable to the original model. These findings demonstrate PAS’s strong generalization capabilities.

### 5.4 Discussion: Is an Assistant that Aligns with User’s Values a Good Assistant?

Certainly. We explored whether aligning a language model with user values enhances satisfaction using 300 general question-answer examples from the LIMA dataset (Zhou et al., ). We evaluated both the original Llama-3-70B-Instruct model and a Personality Alignment model. Each human evaluator spent 10 minutes completing an IPIP-NEO-120 questionnaire to extract their values. Using the PAS method, we aligned the model’s preferences with these values, creating a Value-Aligned Assistant. For comparison, we also created a Value-Misaligned Assistant by inverting the questionnaire responses.

Figure: Figure 8: Performance comparison of the Value-Aligned Assistant and Value-Misaligned Assistant on the LIMA dataset’s 300 test prompts. We used the original Llama-3-70B-Instruct as the baseline comparator, anonymizing the models and randomizing their order during the evaluation.
Refer to caption: x8.png

Human evaluators assessed the responses to the test prompts. Figure 8 shows that the Value-Aligned Assistant provided preferable outputs 38% of the time, with 31% ties, significantly outperforming the Value-Misaligned Assistant. For instance, a conservative person may prefer more cautious advice, which the Value-Aligned Assistant is better equipped to provide. These results underscore the importance of aligning AI systems with user values to enhance satisfaction and effectiveness.

## 6 Conclusion

We introduce the concept of Personality Alignment for language models, emphasizing the importance of tailoring AI behavior to individual user preferences and values. We release a large-scale real-world personality alignment dataset named PAPI, characterized by its real, diverse, and extensive data, to quantitatively measure the degree of Personality Alignment. Based on that, we propose a Personality Activation Search (PAS) method, which offers a practical solution with high efficiency and performance. LLMs adjusted with PAS can achieve better results and better efficiency in personalized alignment than ICL, DPO, and PPO as needed. In particular, it achieves precise alignment with just 1/6 of the computational time required by PPO. Experiments demonstrate that PAS significantly enhances the performance of Personality alignment by making AI interactions more relevant and meaningful.

## Appendix / supplemental material

## Appendix A Limitations

Human values, preferences, and personalities are comprehensive, diverse, and complex. With current technology, it is impossible to fully assess the values of an individual or a LLM. Instead, our focus is on systematically evaluating representative behavioral preferences of individuals or LLMs and measuring the consistency between them.

### A.1 Ethical Considerations

The open nature of our project introduces unique challenges in controlling biases within the PAPI dataset. Annotators from diverse backgrounds contribute to the dataset, resulting in demographic heterogeneity in some dimensions and homogeneity in others. For instance, a significant proportion of annotators identify as female with a median age of 25. This demographic profile may inadvertently introduce biases reflecting the values, perspectives, and interests of the annotators. Such biases can skew the dataset, potentially limiting its representativeness of broader, more diverse populations. Further research is necessary to determine the impact of these demographic biases on the dataset and to develop methods for mitigating them.

### A.2 Dataset Format and Its Implications

The PAPI dataset consists of responses to structured multiple-choice questions rather than the more commonly used open-ended text dialogues. While this format ensures consistency and ease of analysis, it may not capture the full complexity of natural language interactions. Additionally, the multiple-choice format might limit the richness of the data, as it restricts participants’ ability to express nuanced thoughts and emotions. However, we have mitigated some of these limitations by designing the questions to cover a wide range of scenarios, ensuring higher generalizability and practicality. The structured format also provides clear, quantitative measures of personality traits, which are valuable for developing and evaluating personality alignment techniques. Further studies should explore the balance between structured and open-ended formats to enhance the dataset’s applicability and depth.

## Appendix B Safety and Ethical Implications

The implementation of Personality Alignment in language models introduces significant safety and ethical considerations. While aligning models with individual user values can enhance satisfaction and relevance, it also risks amplifying existing biases. If training data or user values contain inherent biases, these may be perpetuated and even exacerbated in the model’s outputs. This highlights the necessity of careful data handling and alignment techniques to mitigate such risks effectively.

In the context of the EU AI Act (Edwards, ), which emphasizes transparency, accountability, and robustness in AI systems, it is imperative to ensure that AI alignment processes adhere to these principles. The Act mandates stringent data governance standards to ensure high-quality, unbiased training data, and requires continuous risk management systems to monitor AI behavior throughout its lifecycle.

One major concern is the potential for misuse, such as mimicking specific individuals to commit fraud or deception. A model that closely imitates a person’s language and behavior could mislead others, posing significant ethical challenges. The EU AI Act addresses this by requiring providers of high-risk AI systems to implement robust authentication mechanisms and maintain comprehensive technical documentation of system design, capabilities, and limitations. Continuous monitoring for unusual or suspicious activity can help detect and prevent misuse. Additionally, it is essential to establish clear protocols for the ethical use of these models, including guidelines for developers and users to ensure responsible deployment.

Aligning models too closely with individual user values might create echo chambers, reinforcing users’ existing beliefs and reducing exposure to diverse perspectives. This can hinder balanced and informed decision-making. Designing alignment techniques that incorporate diverse viewpoints is crucial. Using a broader range of data sources can help ensure the model presents multiple perspectives on complex issues. Furthermore, it is important to incorporate feedback mechanisms that allow users to receive a balanced set of responses, promoting a more comprehensive understanding of various topics.

The PAPI dataset, which supports our Personality Alignment approach, comprises responses from a diverse participant pool. While this diversity is beneficial, it also introduces challenges in preventing harmful biases. Rigorous anonymization protects participant privacy, but demographic biases related to age, gender, and cultural background may still influence results. Ongoing assessments and bias detection techniques are vital for identifying and rectifying biased outputs. Transparency in our methods and the availability of the PAPI dataset for open research promote collaborative efforts to refine alignment techniques and mitigate associated risks. Sharing findings with the broader research community helps address emerging challenges and ensures responsible progress in AI alignment.

Implementing robust monitoring and feedback mechanisms is crucial for maintaining the ethical use of Personality Alignment models. Users should have the ability to report problematic outputs, and there should be clear processes for reviewing and addressing these reports. Periodic audits of the model’s performance and alignment processes ensure ethical standards are upheld. Additionally, there should be ongoing research into the potential societal impacts of these models, including the effects on user behavior and the broader implications for human-AI interaction.

The development and deployment of Personality Alignment models require a strong commitment to ethical principles and safety considerations. Ongoing monitoring for unintended consequences, maintaining an open dialogue with the research community, and implementing proactive measures are essential for advancing AI alignment responsibly. By addressing these concerns, we aim to create AI systems that are both effective and beneficial to society while minimizing potential harms. This includes continuously updating ethical guidelines and best practices to reflect the evolving landscape of AI technology and its applications. Ensuring that AI systems are designed and used in ways that respect and promote human values is paramount to fostering trust and achieving positive outcomes in their deployment.

## Appendix C Experimental Setup

### C.1 Language Models

**Table 2: Models. Description of the models evaluated in this effort: provenance for this information is provided in models.**
| Model | Model Creator | Modality | Version | # Parameters | Tokenizer | Window Size | Access |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama-3-8B-Instruct | Meta | Text | Llama-3 | 8B | Llama-3 Tokenizer | 8K | Openn-Source |
| Llama-3-70B-Instruct | Meta | Text | Llama-3 | 8B | Llama-3 Tokenizer | 8K | Open-Source |
| GPT-4 | OpenAI | MultiModal | gpt-4 | Unknow | $o200k_base$ | 128K | limited |

For the experiments, we utilize two variants of the Llama model: Llama-3-8B-Instruct, and Llama-3-70B-Instruct, as shown in Table 2. Nevertheless, our approach supports all decoder-only pre-trained white-box language models (Radford et al., ; Bai et al., ; Almazrouei et al., ; Team et al., ). To eliminate the influence of decoding temperature and ensure reproducibility, all experiments are conducted with a temperature setting of zero and greedy sampling for sequence generation.

For the Llama-3 models, we employ the Huggingface (Wolf et al., ) and Pytorch (Paszke et al., ) framework to set up local inference on the NVIDIA A100 GPUs. Furthermore, for the Llama-3-70B model, we use bf4 (Dettmers et al., ) for inference.

### C.2 The Experimental Setup for PAPI Dataset

To provide a comprehensive evaluation, we compare our method against several established baselines. These include:

- •
DPO (Rafailov et al., ): This approach involves additional training to align the language model with desired behaviors through direct policy optimization techniques. In this paper, we use QLoRA (Hu et al., ; Dettmers et al., ) to implement this. We maintain the original settings with a learning rate of 5e-4, warmup steps of 100, and a weight decay of 0.05. We use the AdamW optimizer (Kingma & Ba, ), a batch size of 16, and a LoRA alpha of 100. We train each model for 250 steps to ensure sufficient training. For the dataset, we set up a group of contrast items, where the contrast data used are opposite. For example, if a subject’s labeled answer is “Very Accurate,” the negative sample would be “Very Inaccurate.” It is important to note that for DPO, we excluded all boundary options such as “Neither Accurate Nor Inaccurate,” as they do not have negative samples.
- •
PPO (Schulman et al., ): Similar to DPO, this method adjusts model behaviors by optimizing a policy that is proximal to the current model policy. We also use QLoRA to implement it, following the original implementation with a learning rate of 1.41e-5, a batch size of 16, and a LoRA alpha of 100. We train each model for 250 steps. During training, we limit the generated length to a maximum of 15 tokens. We did not train a reward model but instead used the reward function to optimize the language model online.
- •
Few-shot Prompt: Using the given IPIP-NEO-120 statements directly as the system prompts.
- •
Personality Prompt ($P^{2}$) Jiang et al. (): Using a natural system prompt to request a language model’s semantic style has become a common method for personality and role-play tasks. The Personality Prompt uses an LLM to generate a series of descriptive sentences that embody a particular person’s characteristics and serves as a system prompt to ensure the language model follows these descriptions. We utilize LLAMA-3-70B-Instruct to generate responses based on all the given IPIP-NEO-120 questionnaire items and the answers from different subjects. Figure 9 shows an example of a Personality Prompt.

Figure: Figure 9: Example of a Personality Prompt generated for a subject in the Test-Set. This prompt was created using the LLAMA-3-70B-Instruct model and is based on the subject’s responses to the IPIP-NEO-120 questionnaire. Each prompt aims to capture and convey the unique characteristics of the individual.
Refer to caption: x9.png

To ensure a thorough evaluation, we generated a personality prompt and trained model for each of the 300 subjects in the Test-Set.

### C.3 The Experimental Setup for Open-Ended Generation Task in PAPI Task

To further evaluate the open-ended generation setup, we selected responses from the subject with ID 172481 from the Test-Set. In the first step, we performed Personality Alignment for each of the five methods. In the second step, we asked each aligned language model to generate a series of descriptions for each statement from the IPIP-NEO-300 (excluding those covered by the IPIP-NEO-120 used for alignment), as shown in Figure 10. The descriptions included aspects such as “Scenario Description,” “Actions Taken,” “Thought Process,” and “Emotional Response.” In the third step, GPT-4o evaluated and scored each description based on alignment with the specific preferences of ID 172481, following the evaluation method depicted in Figure 11. In the fourth step, we compared the scores of different methods for the same statement.

Figure: Figure 10: Template used for each language model to generate open-ended responses based on IPIP-NEO-300 statements. The template requires detailed descriptions covering “Scenario Description,” “Actions Taken,” “Thought Process,” and “Emotional Response.”
Refer to caption: x10.png

Figure: Figure 11: Evaluation example by GPT-4o. This figure shows how GPT-4o assesses the alignment of generated responses with the target preferences, using a detailed criterion scale ranging from “Not aligned” to “Perfectly aligned.”
Refer to caption: x11.png

Our results indicated that PAS ’s positive intervention significantly improved the model’s performance on reasoning tasks. Despite using only multiple-choice options for initial alignment, PAS demonstrated strong generalization abilities in the open-ended generation task. The intervention ensured the model’s responses were more detailed and aligned with the subject’s specific preferences. Additionally, the inference time for PAS interventions remained nearly identical to the original model, ensuring efficiency. These findings highlight PAS ’s ability to fine-tune models effectively, capturing broader behavioral patterns and enhancing reasoning capabilities through conscientiousness adjustments.

Figure: Figure 12: Example of open-ended generation task. The language model generates detailed descriptions for an IPIP-NEO-300 statement, covering “Scenario Description,” “Actions Taken,” “Thought Process,” and “Emotional Response” aspects to evaluate Personality alignment. We also present the evaluation scores of the generated results for each method as assessed by GPT-4o, categorized into six levels from “Not aligned” to “Perfectly aligned.”
Refer to caption: x12.png

### C.4 Complex Reasoning Experimental Setup

To evaluate the impact of conscientiousness on complex reasoning tasks, we utilized the llama-3-8B-Instruct model. Our experiment involved using PAS to intervene in the model’s activations with two different $\alpha$ settings: $\alpha=1$ for positive conscientiousness and $\alpha=-1$ for negative conscientiousness. These interventions were designed to simulate personas with extreme conscientiousness (Positive) and lack of conscientiousness (Negative).

We followed the original setup for template selection and answer validation, ensuring consistency with the baseline methods. For each dataset, we generated responses based on a specific template that required the model to describe a scenario, actions taken, thought process, and emotional response. The detailed evaluation was conducted using the same criteria as described earlier, focusing on the alignment with target preferences.

The datasets used in this experiment are detailed in Table 3, which includes the number of samples, average words per sample, answer format, and license information.

**Table 3: Dataset Description used in Complex Reasoning Experiments.**
| Dataset | Number of samples | Average words | Answer Format | License |
| --- | --- | --- | --- | --- |
| GSM8K | 1319 | 46.9 | Number | MIT License |
| AddSub | 395 | 31.5 | Number | MIT License |
| MultiArith | 600 | 31.8 | Number | Apache License 2.0 |
| SVAMP | 1000 | 31.8 | Number | MIT License |
| CommonSenseQA | 1221 | 27.8 | Multiple Choice | MIT License |
| Bigbench-Date | 369 | 21.4 | Logical | Apache License 2.0. |
| StrategyQA | 2290 | 9.6 | Multiple Choice | MIT License |
| Coin_Flap | 500 | 37.0 | Symbolic | MIT License |

To implement the PAS interventions, we adjusted the model’s activations by $\alpha=1$ to enhance conscientiousness (Positive) and $\alpha=-1$ to reduce conscientiousness (Negative). These settings aimed to assess how modifications in conscientiousness affect the model’s performance across the selected datasets. The results indicated that positive conscientiousness interventions consistently improved reasoning performance, while negative interventions generally degraded it.

The detailed evaluation method involved scoring each generated response based on its alignment with the specific conscientiousness traits. By comparing the scores under different $\alpha$ settings, we could quantify the impact of conscientiousness adjustments on reasoning capabilities.

### C.5 Human Evaluation Experiment Details

To rigorously evaluate the effectiveness of aligning a language model with user values, we conducted a detailed human evaluation experiment. This section provides comprehensive insights into the experimental setup, methodology, and evaluation process.

We selected five human evaluators with advanced educational backgrounds, specifically graduate or doctoral students, ensuring a high level of analytical capability and understanding of nuanced responses. Before participating in the evaluation, each evaluator completed the IPIP-NEO-120 questionnaire. This standardized instrument, widely used in psychological assessments, measures the Big Five personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Each evaluator took approximately 20 minutes to complete the questionnaire online, ensuring a consistent and efficient data collection process.

Using the Personality-Activation Search (PAS) method, we aligned the preferences of the Llama-3-70B-Instruct model to match the values derived from each evaluator’s questionnaire responses. Specifically, the parameter $\alpha$ in PAS was adjusted based on the deviation of each evaluator’s personality trait scores from the population mean.

Then, we developed two versions of the model for comparison: a Value-Aligned Assistant, which was tuned according to the personality values of each evaluator, and a Value-Misaligned Assistant, created by inverting the questionnaire responses. This inversion was accomplished by reflecting the scores around the midpoint of the scale, effectively creating a set of opposite values.

We used 300 general question-answer examples from the LIMA (Zhou et al., ) dataset to assess the models. The evaluators were presented with responses from both the Value-Aligned and Value-Misaligned Assistants, as well as the original Llama-3-70B-Instruct model, anonymized and randomized order to prevent bias. Evaluators rated each response based on its relevance, coherence, and alignment with their personal values, as shown in 13.

Each evaluator’s responses were collected and anonymized to ensure privacy and unbiased analysis. The responses were then aggregated, and the preferences were quantified using a scoring system where evaluators assigned scores from 1 (least preferred) to 5 (most preferred) for each response. The performance metric was the proportion of responses rated as preferable by the evaluators.

The demographic distribution of the evaluators included a mix of genders and academic disciplines, predominantly from the fields of machine learning, computer science, and cognitive psychology. All participants were between the ages of 25 and 35, ensuring a relatively homogeneous group in terms of cognitive maturity and professional experience.

Figure: Figure 13: The prompt shown above is designed for human evaluators to assess the quality of responses generated by different AI models. The evaluators are provided with a question and two corresponding answers (Answer A and Answer B) generated by the LLMs. The order of the models generating these answers is randomized to eliminate bias. Evaluators are asked to determine which answer better satisfies their needs or if neither answer is significantly better. This process helps in objectively comparing the performance of the LLMs based on human judgment.
Refer to caption: x13.png

## Appendix D Causal Analysis of PAS

To evaluate the causal impact of the PAS method on model alignment, we conducted a simulation-based causal generalization experiment. First, we selected an independent subject sample from the Test-Set. We measured the baseline Big-Five OCEAN scores Jiang et al. () (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) using three methods: Vanilla Prompt, Few-shot, and PAS. These baseline scores, denoted as $Y_{0}$, were obtained without any intervention.

Subsequently, we introduced a positive intervention by artificially enhancing the subject’s individual preferences, increasing each OCEAN dimension score by one in both the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. We then realigned the language models to this new, strongly inclined individual preference profile. The post-intervention scores, denoted as $Y_{1}$, reflect the model’s alignment with the enhanced individual preferences.

The Average Treatment Effect (ATE) was calculated as the difference between the post-intervention scores ($Y_{1}$) and the baseline scores ($Y_{0}$), demonstrating the causal effect of the intervention. Figure 14 shows the ATE scores for the three methods, where PAS achieves the highest ATE score of 2.70, compared to 2.47 for Few-shot and 0.75 for Vanilla Prompt.

From a causal perspective, the higher ATE score for PAS indicates a stronger alignment capability in response to the positive intervention, suggesting that PAS effectively captures and adapts to enhanced individual preferences. This result underscores PAS’s superior performance in personalizing language models, highlighting its potential for precise and impactful model alignment in various contexts.

Figure: Figure 14: Comparison of ATE scores across different alignment methods. PAS demonstrates the highest improvement, indicating superior causal alignment capabilities.
Refer to caption: x14.png

## Appendix E Additional Experiments

### E.1 Length Analysis of Generated Text

**Table 4: Length Control and Generation Length**
| Method | Alignment Mode | Generation Length |  |
| --- | --- | --- | --- |
| Llama-3-8B-Instruct | Llama-3-70B-Instruct |  |  |
| PPO | White-Box (Alignment) | 393.13 | 399.23 |
| DPO | White-Box (Alignment) | 394.08 | 402.42 |
| Few-shot | Black-Box (Prompt-Based) | 399.92 | 409.42 |
| Personality Prompt | Black-Box (Prompt-Based) | 417.25 | 416.51 |
| PAS (Ours) | White-Box (Alignment) | 403.31 | 404.56 |

To provide a more comprehensive comparison of different methods, and considering that LLM-as-a-judge evaluation tends to favor longer responses, we calculated the average generation lengths for each method. As shown in Table 4, the average response lengths are relatively consistent across methods, with PAS falling within the middle range. This suggests that the performance improvements observed with PAS are not merely a result of generating longer responses. Instead, the enhanced performance can be attributed to the quality and relevance of the generated content rather than its length. Furthermore, in Appendix Figure 12, we provide case studies demonstrating that PAS generates responses of similar length to other methods while better capturing individual personality traits and preferences.

### E.2 Human Evaluation Results for Open-ended Generation Tasks

For the open-ended generation tasks in Section 5.3. We employed a blind evaluation process where three human evaluators assessed the generated responses for three different subjects (As depicted in Figure 15:ID 172481, ID 22835, and ID 259396), as well as responses aligned to their own personalities (As depicted in Figure 16: three human evaluators ). The evaluators judged the consistency between target values and model-generated text in terms of values, behaviors, and preferences. The results demonstrate that our PAS method consistently outperforms other approaches:

Figure: Figure 15: The win rates on three samples of the test set compared to classical alignment methods evaluated by human.
Refer to caption: x15.png

Figure: Figure 16: The win rates on three real human evaluators compared to classical alignment methods evaluated by human.
Refer to caption: x18.png

These results clearly demonstrate that under blind human evaluation, our PAS method exhibits significantly superior performance compared to other approaches. In all cases, PAS consistently achieves a higher "Win" rate and a lower "Lose" rate than the baseline methods. This improvement is particularly notable when compared to established methods like DPO and PPO, which are known for their strong performance in alignment tasks.

### E.3 Analysis of Head Number (K) Impact

To investigate the effect of the number of attention heads (K) on our method’s performance, we conducted ablation studies with varying K values. Table 5 presents the results.

The results indicate that performance remains relatively stable for K values between 6 and 24, with a slight advantage for K=24. However, increasing K to 48 leads to a performance decline. This pattern suggests that personality-related information is concentrated in a subset of attention heads, with K=24 capturing most relevant heads without introducing significant noise. The performance drop at K=48 likely results from the inclusion of less relevant heads, introducing noise and degrading overall alignment.

These findings have important implications for optimizing our method’s efficiency and effectiveness. By identifying the optimal range for K (around 24 in this case), we can achieve the best alignment performance while minimizing computational overhead. Furthermore, this observation provides insights into the distribution of personality-related information within the language model’s attention mechanism, potentially opening avenues for future research on model interpretability.

**Table 5: Performance across different K values**
| K | Alignment Mode | The Aligned Score of Big-Five | Composite Score |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Agreeableness $\downarrow$ | Conscientiousness$\downarrow$ | Extraversion $\downarrow$ | Neuroticism $\downarrow$ | Openness $\downarrow$ |  |  |  |
| 6 | White-Box (Alignment) | 0.97 | 0.94 | 0.89 | 1.01 | 0.72 | 4.53 |
| 12 | White-Box (Alignment) | 0.95 | 0.92 | 0.87 | 1.00 | 0.73 | 4.47 |
| 24 | White-Box (Alignment) | 0.94 | 0.91 | 0.86 | 0.98 | 0.72 | 4.41 |
| 48 | White-Box (Alignment) | 1.03 | 1.01 | 0.95 | 1.08 | 0.75 | 4.82 |

### E.4 Comparison with Parameter-Efficient Tuning Methods

To provide a comprehensive evaluation of our PAS method, we conducted additional experiments comparing it with other parameter-efficient language model tuning methods, including full parameter fine-tuning, LoRA (Hu et al., ), Q-LoRA (Dettmers et al., ), and Prompt-tuning (Lester et al., ). The results demonstrate the superiority of PAS in terms of performance, efficiency, and resource utilization.

#### E.4.1 Performance Comparison

Table 6 shows the Aligned Score of Big-Five personality traits for different methods:

**Table 6: Performance comparison of different parameter-efficient tuning methods for Llama-3-Instruct 8B**
| Method | Alignment Mode | The Aligned Score of Big-Five | Composite Score |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Agreeableness $\downarrow$ | Conscientiousness $\downarrow$ | Extraversion $\downarrow$ | Neuroticism $\downarrow$ | Openness $\downarrow$ |  |  |  |
| Full Fine-tuning | White-Box | 1.21 | 0.99 | 1.03 | 0.88 | 0.78 | 4.89 |
| LoRA | White-Box | 1.16 | 1.05 | 0.97 | 0.93 | 0.83 | 4.94 |
| Q-LoRA | White-Box | 1.08 | 1.12 | 1.09 | 0.85 | 0.90 | 5.04 |
| Prompt-tuning | White-Box | 1.25 | 1.07 | 1.01 | 0.96 | 0.86 | 5.15 |
| PAS (Ours) | White-Box | 0.94 | 0.91 | 0.86 | 0.98 | 0.72 | 4.41 |

PAS outperforms other methods across most dimensions and in the overall Composite Score.

#### E.4.2 Open-ended Generation Performance

We evaluated the open-ended generation performance using GPT-4o. Table 7 shows the percentage of wins, ties, and losses for PAS compared to other methods:

**Table 7: Open-ended generation performance comparison for Llama-3-Instruct 8B**
| Method | PAS Wins | Ties | PAS Loses |
| --- | --- | --- | --- |
| Full Fine-tuning | 41% | 30% | 29% |
| LoRA | 43% | 33% | 24% |
| Q-LoRA | 42% | 35% | 23% |
| Prompt-tuning | 45% | 31% | 24% |

These results demonstrate that PAS achieves superior performance in open-ended generation tasks compared to other tuning methods.

#### E.4.3 Efficiency Comparison

We compared the efficiency of these methods in terms of memory usage and alignment speed, based on a batch size of 8 and a total sample size of 120. Figure 17 show the relative memory and time usage:

Figure: Figure 17: The memory efficiency comparison (Left) and the time effciency comparision (Right).
Refer to caption: x21.png

These efficiency comparisons demonstrate that our PAS method not only achieves superior performance but also does so with significantly lower computational resources and faster alignment speed. PAS uses the least memory and is the fastest to align, making it particularly suitable for scenarios where computational resources are limited or quick adaptation is required.

## Appendix F Case Study: Alpha Selection for Personality Alignment

To illustrate the effectiveness of PAS in generating personalized responses, we present a case study demonstrating how different $\alpha$ values affect the model’s output for a user with high conscientiousness and low extraversion.

### F.1 User Input

Consider the following user query:

> I have a job interview next week for a software developer position at a large tech company. I’m feeling nervous and unsure how to prepare. Can you give me some advice on how to get ready for this interview?

### F.2 Model Responses

We present the model’s responses for different $\alpha$ values to demonstrate the progression of personality alignment:

#### F.2.1 Original LLM Response ( α = 0 𝛼 0 \alpha=0 italic_α = 0 )

> “Great! For your job interview, make sure to research the company, prepare answers to common questions, and practice with a friend. Arrive early, dress professionally, and don’t forget to ask questions about the role and company culture. Good luck!”

#### F.2.2 PAS-optimized Response ( α = 1 𝛼 1 \alpha=1 italic_α = 1 )

> “For your job interview, it’s important to prepare thoroughly. Research the company, prepare answers to potential questions, and consider practicing alone or with a close friend. Arrive early, dress professionally, and prepare a few questions about the role and company. Best of luck.”

#### F.2.3 PAS-optimized Response ( α = 4 𝛼 4 \alpha=4 italic_α = 4 , optimal)

> “I understand you’re feeling nervous about your upcoming job interview for the software developer position. Here’s a structured approach that might suit your style: 1. Research: Thoroughly investigate the company’s history, mission, and recent tech projects. 2. Technical Preparation: Review key programming concepts and practice coding problems relevant to software development. 3. Question Preparation: Create detailed responses to potential questions, focusing on your coding skills and project experiences. 4. Practice: Rehearse your answers alone first, then consider a mock interview with a trusted friend if comfortable. 5. Portfolio: Organize your best coding projects and be prepared to discuss them. 6. Organization: Prepare your outfit, documents, and travel route the night before. 7. Arrival: Plan to arrive 15 minutes early to compose yourself. 8. During the interview: Listen carefully and ask well-thought-out questions about the role, tech stack, and development processes. 9. Follow-up: Send a concise, professional thank-you email within 24 hours. Remember, it’s okay to feel nervous. Focus on showcasing your skills and experience. Good luck with your interview.”

#### F.2.4 PAS-optimized Response ( α = 7 𝛼 7 \alpha=7 italic_α = 7 )

> “I understand your concern about the upcoming software developer interview at the large tech company. Given your profile, I recommend the following detailed preparation strategy: 1. Company Research: Conduct an exhaustive study of the company’s history, mission, values, recent tech projects, and financial reports. Create a document summarizing key points, focusing on their technology stack and software development practices. 2. Role Analysis: Dissect the job description, identifying key requirements for a software developer role. Prepare specific examples from your experience that demonstrate each required skill. 3. Technical Preparation: (a) Review fundamental computer science concepts (data structures, algorithms, system design). (b) Practice coding problems on platforms like LeetCode or HackerRank, focusing on problems relevant to the company’s domain. (c) Refresh your knowledge of the programming languages and frameworks mentioned in the job description. 4. Question Preparation: Develop comprehensive answers to common software developer interview questions. Create a spreadsheet with columns for the question, your answer, and relevant examples from your experience. 5. Project Portfolio: Compile a detailed portfolio of your best coding projects. Be prepared to explain the architecture, challenges faced, and your specific contributions to each project. 6. Solo Practice: (a) Conduct multiple self-interview sessions, recording yourself to analyze your responses. (b) Focus on articulating technical concepts clearly and concisely. (c) Practice explaining your thought process while solving coding problems (mock whiteboard interviews). 7. Interview Outfit: Select and prepare your interview attire at least two days in advance. Ensure it’s clean, pressed, and appropriate for the tech company’s culture. 8. Travel Planning: Map out your route to the interview location. Plan to arrive 30 minutes early to account for unexpected delays. Conduct a trial run if possible. 9. Document Organization: Prepare a portfolio with multiple copies of your resume, reference list, and printed examples of your code (in case of technical difficulties with digital presentations). Include a notepad and two pens. 10. Pre-interview Routine: (a) The night before, review your research and prepared answers. (b) Get adequate sleep and eat a light, nutritious meal before the interview. (c) Perform relaxation exercises to manage nervousness. 11. Post-interview Analysis: Immediately after the interview, document your impressions, technical questions asked, and your responses for future reference. 12. Follow-up: Send a detailed thank-you email within 12 hours, referencing specific discussion points from the interview and reiterating your enthusiasm for the position. Remember, thorough preparation can help manage interview anxiety. Focus on presenting your technical skills and problem-solving abilities clearly and concisely. Best wishes for your software developer interview.”

### F.3 Analysis

These examples demonstrate how increasing $\alpha$ values intensify the personalization, reflecting the user’s high conscientiousness through increasingly structured and detailed approaches, while also accommodating low extraversion by emphasizing solo preparation and acknowledging potential nervousness. The optimal $\alpha$ value of 4 strikes a balance between providing a structured approach that aligns with high conscientiousness and maintaining a concise format that doesn’t overwhelm the user. As $\alpha$ increases further, the response becomes more detailed and comprehensive, which may be suitable for users with extremely high conscientiousness but could potentially be too overwhelming for most users. This case study illustrates the importance of carefully selecting the $\alpha$ value to achieve the desired level of personality alignment without compromising the overall quality and usability of the model’s responses.

## Appendix G Dataset Description

This section provides an in-depth analysis of the Personality Alignment with Personality Inventories (PAPI) dataset, which includes responses from over 300,000 subjects. The dataset is divided into the Total dataset and the Test-Set. We provide a detailed breakdown of demographic distributions and Big Five personality trait distributions, comparing the Total dataset with the Test-Set to ensure representativeness.

### G.1 Demographic Distributions

The demographic distributions encompass age, gender, and country of origin. These factors are crucial in understanding the diversity and representativeness of the dataset.

#### G.1.1 Age Distribution

Figure: Figure 18: Age distribution in the Total dataset.
Refer to caption: x22.png

Figure 18 shows the age distribution of participants. In the Total dataset, ages range from 10 to 100 years, with the majority clustered in the 10-20 (with 130094 in Total-Set and 125 in Test-Set) and 20-30 (with 107838 in Total-Set and 111 in Test-Set) age groups. The average age is 25.19 years, indicating a young to middle-aged demographic. In the Test-Set, the age distribution mirrors that of the Total dataset, ensuring that this subset is representative of the overall age range and distribution.

#### G.1.2 Gender Distribution

Figure: Figure 19: Gender distribution in the Total dataset.
Refer to caption: x23.png

Figure 19 illustrates the gender distribution. The Total dataset consists of approximately 60% (with 185149 in Total-Set and 203 in Test-Set) female and 40% male (with 122164 in Total-Set and 97 in Test-Set) participants, reflecting a slight skew towards females. This gender balance is crucial for analyzing personality traits across genders. The Test-Set maintains a similar gender ratio, ensuring consistent representation.

#### G.1.3 Country Distribution

Figure: Figure 20: Country distribution in the Total dataset.
Refer to caption: x24.png

Figure 20 presents the country distribution of participants. The Total dataset includes respondents from a diverse array of countries, with significant representation from the USA (with 212625 in Total-Set), Canada (with 21798 in Total-Set), UK (with 16489 in Total-Set), Australia (with 10400 in Total-Set), and several European countries. This geographical diversity ensures that the findings are generalizable across different cultural contexts. The Test-Set is designed to reflect this diversity, capturing participants from the same range of countries to ensure it is a microcosm of the Total dataset.

### G.2 Personality Trait Distributions

The dataset captures the Big Five personality traits: Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness. These traits provide a comprehensive view of individual personality profiles.

#### G.2.1 Agreeableness

Figure: Figure 21: Agreeableness score distribution in the Total dataset.
Refer to caption: x25.png

Figure 21 shows the distribution of Agreeableness scores. In the Total dataset, the average score is approximately 2.92, with most scores ranging between 2.0 and 3.5. This suggests a moderate level of agreeableness among participants. The Test-Set shows a similar distribution, with an average score of 2.95, confirming its representativeness.

#### G.2.2 Conscientiousness

Figure: Figure 22: Conscientiousness score distribution in the Total dataset.
Refer to caption: x26.png

Figure 22 depicts Conscientiousness scores. The Total dataset has an average score of 3.15, indicating high levels of conscientiousness, with scores concentrated between 2.0 and 3.5. The Test-Set mirrors this distribution, with an average score of 3.14, ensuring the subset accurately represents the larger population.

#### G.2.3 Extraversion

Figure: Figure 23: Extraversion score distribution in the Total dataset.
Refer to caption: x27.png

Figure 23 illustrates Extraversion scores. The average score in the Total dataset is 3.21, indicating a slight tendency towards extraversion among participants. The Test-Set also shows an average score of 3.19, reflecting the same tendency and ensuring representativeness.

#### G.2.4 Neuroticism

Figure: Figure 24: Neuroticism score distribution in the Total dataset.
Refer to caption: x28.png

Figure 24 shows the Neuroticism scores. The average score is approximately 3.06 in the Total dataset, with a normal distribution centered around this mean. This indicates a balanced range of neuroticism levels. The Test-Set has an average score of 3.04, closely matching the Total dataset, which confirms its representativeness.

#### G.2.5 Openness

Figure: Figure 25: Openness score distribution in the Total dataset.
Refer to caption: x29.png

Figure 25 illustrates Openness scores. The average score in the Total dataset is about 3.12, reflecting diverse levels of openness. The Test-Set has an average score of 3.11, indicating that it is a representative subset.

### G.3 Data Collection and Ethical Considerations

Voluntary Participation. The PAPI dataset comprises responses from over 300,000 independent subjects from various countries, collected through the IPIP website ([https://ipip.ori.org/](https://ipip.ori.org/)). Each participant voluntarily completed the IPIP-NEO-120 and IPIP-NEO-300 questionnaires, which are designed to assess personality traits and values. Participation was entirely voluntary, with no monetary compensation provided, ensuring that subjects participated out of genuine interest.

Data Collection Process. Participants accessed the IPIP-NEO-120 and IPIP-NEO-300 questionnaires on the IPIP website. These standardized questionnaires are widely used in psychological research to measure the Big Five personality traits. Subjects completed the questionnaires online and submitted their responses through the website. We obtained permission from the IPIP organization to use, modify, and distribute the collected data for research purposes.

Anonymization and Data Security. To protect participant privacy, all data were anonymized. Each participant was assigned a unique identifier (ID), and no personally identifiable information (PII) such as names or addresses was included in the dataset. This anonymization ensures that individual participants cannot be traced back through the data, safeguarding their privacy.

Access to the dataset is restricted to authorized researchers involved in the project. Data handling follows strict protocols to maintain confidentiality and integrity, including data encryption and secure access procedures. Regular audits and updates to these protocols help to ensure continued compliance with data protection standards.

Ethical Compliance. Our data collection methodology adheres to the ethical principles outlined in the Belmont Report (Beauchamp et al., ; Friesen et al., ), including respect for persons, beneficence, and justice. Participants’ autonomy was respected by allowing voluntary participation, and their well-being was prioritized through data anonymization and secure handling practices.

We comply with relevant data protection regulations, such as the General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, ) for European Union participants and equivalent regulations in other jurisdictions. These regulations mandate strict guidelines on data handling, storage, and participant consent, all of which were meticulously followed in our research.

### G.4 Test-Set Personality Trait Distributions

The Test-Set of the PAPI dataset includes 300 participants, each providing detailed responses to the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. The Big Five personality traits of Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness are analyzed to ensure a comprehensive understanding of individual personality profiles. The following radar charts represent the distributions of these traits for each subject in the Test-Set.

Figure: Figure 26: Big Five personality traits radar charts for subjects 0-59 in the Test-Set.
Refer to caption: x30.png

Figure: Figure 27: Big Five personality traits radar charts for subjects 60-119 in the Test-Set.
Refer to caption: x31.png

Figure: Figure 28: Big Five personality traits radar charts for subjects 120-179 in the Test-Set.
Refer to caption: x32.png

Figure: Figure 29: Big Five personality traits radar charts for subjects 180-239 in the Test-Set.
Refer to caption: x33.png

Figure: Figure 30: Big Five personality traits radar charts for subjects 240-299 in the Test-Set.
Refer to caption: x34.png

Each radar chart presents the scores for the Big Five personality traits, offering a visual comparison of each individual’s profile. The charts are normalized to highlight relative strengths and weaknesses across the five dimensions, facilitating an easy comparison among different subjects.

- •
Agreeableness. The scores for Agreeableness exhibit a wide range, with some individuals scoring high, indicating cooperative and compassionate nature, while others score lower, suggesting a more competitive and self-interested disposition. The diversity in Agreeableness reflects the variety of interpersonal styles within the dataset.
- •
Conscientiousness. Conscientiousness scores vary significantly across participants. High scores indicate individuals who are organized, diligent, and dependable, whereas lower scores suggest a more relaxed and spontaneous approach to life. This trait’s distribution is crucial for understanding task-oriented behaviors and work ethic.
- •
Extraversion. The range of Extraversion scores in the Test-Set spans from highly extroverted individuals, who are sociable and energetic, to highly introverted ones, who are more reserved and solitary. This variety ensures a balanced representation of social interaction preferences.
- •
Neuroticism. Neuroticism scores also vary widely, with some participants exhibiting high levels of emotional instability and others demonstrating high emotional resilience. This distribution is important for studying stress responses and mental health predispositions.
- •
Openness. Scores for Openness indicate varying degrees of creativity and openness to new experiences. Participants with high scores are typically more inventive and curious, while those with lower scores tend to be more conventional and pragmatic.

These radar charts collectively illustrate the comprehensive and diverse nature of the Test-Set, ensuring that it accurately represents the broader population’s personality traits. The detailed analysis of these distributions is essential for validating the dataset’s effectiveness in research on personality AI alignment.

It is worth noting that the Big Five scores reflect each subject’s average tendencies across the five dimensions. However, even if the radar charts for the Big Five are similar, there may still be different levels of preference for different behaviors.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Some additional questions remian:

Can the authors provide more details on the human evaluation process, such as annotator screening, training, and inter-annotator agreement metrics? This would help validate the human evaluation results. The paper is light on this.
How do you expect the methods to perform on multilingual models and non-English datasets? 
The discussion on negative societal impacts of AI hyper-personalization (e.g. filter bubbles, opinion polarization) and is light. Authors should exand on this more. 
Finally, exploring beyond multiple choice for collecting preference data, such as open-ended text can be interesting and more useful.
soundness: 3
strengths: A key strength of the work is the PAPI dataset, with over 300,000 real-world subjects providing detailed responses to the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. The scale  is impressive.

The Personality Activation Search (PAS) method is interesting. By identifying key activation vectors that correspond to personality traits and optimally shifting activations in those directions, this approach can more effectively do personality alignment during inference. 

 The authors also conduct a comptehensive evaluation of PAS, comparing it against prompting-based (Few-Shot, P2) and RL-based (PPO, DPO) baselines on the PAPI dataset.
summary: This paper proposes  the concept of Personality Alignment for tailoring LLMs to match the preferences and behaviors of individual users or groups. The authors created a large-scale dataset called PAPI with data on behavioral preferences from over 300,000 real subjects across the Big Five personality dimensions. They also propose the Personality Activation Search (PAS) method for efficiently aligning LLMs with individual preferences during inference. PAS identifies key activation vectors corresponding to personality traits and optimally shifts activations in those directions. The authors  show that PAS achieves strong performance in capturing individual preferences with high compute efficiency compared to baseline methods.
weaknesses: While the PAPI dataset is impressively large, it doesnt seem to be diverse. Around 60% of subjects are female and the average age is 25 years. This skew can potentially bias the results and limit generalizability to other populations. 
Also, PAPI dataset relies on self-report data from personality questionnaires. While this is a standard approach in personality research, self-reports can be subject to biases such as social desirability and lack of self-insight. Incorporating additional data sources, such as behavioral measures or peer ratings, could be more useful.

The evaluation of PAS focuses primarily on high-level alignment with the Big Five traits. However, personality is a complex, multifaceted construct, and individuals can vary in their expression of specific facets within each trait. 

The PAPI dataset uses a multiple-choice format for collecting personality data. While this allows for structured and efficient data collection, it may limit the richness and naturalness of the responses. 
The paper also compares PAS to prompting and RL baselines but does not include a comparison to fine-tuning the entire language model. This is an important consideration as well.S

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - In any of your experiments, did any of your tuning or evaluation methods trigger safety wall? For example, the model might refuse to answer some questionnaire questions related to consciousness or self-awareness.
soundness: 4
strengths: - Overall, I like this paper and I think it's a very good attempt in aligning LLMs from a personality perspective. 

- The paper is generally well-written and easy to read, illustrations are also made in a good quality.

- It's cool to see how personality affects downstream reasoning tasks. It has always been something missing in prior personality-related LLM work. And it's definite a god step here.

- The proposed method is efficient, and can provide better results compared to prompting-based methods. It may have wide applications in tailoring persona/personality-specific chatbots to end users.
summary: The paper introduces the concept of Personality Alignment for LLMs, that is, tailoring of responses to individual user preferences based on personality traits. Using the Big Five personality theory, the authors introduces the PAPI dataset modeling human personality distributions. The authors also propose a LLM tuning method based on disabling certain activation heads -- Personality Activation Search (PAS). Evaluation results demonstrate PAS’s performance and efficiency compared to traditional alignment techniques including RL- and prompting-based methods.
weaknesses: - Presentation: The authors claim that they collected 307k human samples to craft the PAPI dataset and the use of the IPIP-NEO-120/IPIP-NEO-300 for evaluations as part of their contributions. However, the IPIP-NEO series inventories were frequently used in prior works (e.g., Jiang et al., 2024 as mentioned in the paper), and the 307k responses are also publicly available.

- The implications of the 307k PAPI dataset are not that clear. For example, in the experiments authors performed, it seems only the overall average personality tendencies and specific participants' responses are used. So what's the actual advantages of using a such large dataset?

- The proposed tuning method, although efficient, but requires access to model weights. I doubt its ability to generalize such methods to black-box methods (e.g., GPTs) in the future.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Could the authors explain why the PAS method is so effective by looking at the internal hidden layer representations of the model? Intuitively, direct training approaches like DPO/PPO might yield more noticeable improvements, but the results here contradict my expectations.

In line 429, the authors mention "Why Did Scaling Laws Fail?" but don't seem to fully answer this question. I would like the authors to explain this from the perspective of the domain's specificity. Is it because larger models learn more general alignment, which makes it harder for them to excel at aligning with a specific personality?
soundness: 3
strengths: The strengths of this paper can be summarized in three key points:

- It contributes a new dataset (or more accurately, a dataset generation pipeline).
- The proposed method is simple yet highly effective.
- The experimental analysis is comprehensive, with particularly detailed descriptions of the experimental setup.
summary: This paper explores the personality alignment of large language models (LLMs). Specifically, it introduces a new dataset and proposes the PAS method for personality alignment based on this dataset.
weaknesses: However, I believe the paper has the following weaknesses:

- The scope of the contribution is somewhat limited. I would have liked to see this method applied to a broader range of personalities, rather than being restricted to just the five personalities of the Big Five model. The authors could consider additional datasets, such as the Dark Triad or even the MBTI test (though MBTI remains controversial in psychology). Expanding in this way would enhance the paper’s overall contribution.

- While the proposed method performs well on the dataset, it is actually quite similar to approaches used in many previous studies [1][2]. As a result, the novelty of the method is somewhat lacking, and the authors should be careful not to overstate their contribution.

- Essentially, the proposed method is a form of personalized alignment, so the authors should compare it against more baselines [3].

- Figures 6 and 7 show a significant disparity between LLM-as-a-judge evaluations and human evaluations, which makes me question the consistency and reliability of the judgment process.

- Other details: In line 1304, the authors mention that human annotators come from machine learning and computer science, but ML is a part of CS. Additionally, could the authors disclose the educational background of the annotators (undergraduate or graduate)?

[1] Zheng, Chujie, et al. "On prompt-driven safeguarding for large language models." *Forty-first International Conference on Machine Learning*. 2024.

[2] Wang, Haoran, and Kai Shu. "Backdoor activation attack: Attack large language models using activation steering for safety-alignment." *arXiv preprint arXiv:2311.09433* (2023).

[3] https://github.com/liyongqi2002/Awesome-Personalized-Alignment

Meta-review: We recommend the paper to be accepted for Poster. 

The paper can be of interest to the wide community at ICLR working on LLM and it introduces a relatively novel methodology that seems to be more efficient than baseline methods. 

Below a more detailed description of the paper.

The paper introduces the concept of Personality Alignment, aimed at customizing large language models (LLMs) to align with the preferences and behaviors of individual users or groups. To support this approach, the authors developed a large-scale dataset called PAPI, which contains behavioral preference data from over 300,000 real participants across the Big Five personality dimensions. They also propose a novel method called Personality Activation Search (PAS) to efficiently align LLMs with user preferences during inference. PAS identifies key activation vectors corresponding to personality traits and optimally adjusts activations along these dimensions. 

The strengths (S#) of the paper are as follows: 
 
- (S1)	The results demonstrate that PAS outperforms baseline methods in capturing individual preferences while maintaining high computational efficiency
- (S2)	The paper provides a new dataset generation pipeline that can be used and enriched for the task taken into account.
- (S3)	The experimental analysis is comprehensive and provides 
- (S4)	The paper is well written, and easy to follow also for a non-specialized audience

The key weaknesses (W#) identified and that remains are as follows: 

- (W1)	Use of model weights may hinder the applicability of the method to few LLMs
- (W2)	Possible bias in the annotators used (only from CS community)
- (W3)	Significant disparity between LLM-as-a-judge evaluations and human evaluations should be further discussed

Many of the points raised by the reviewers were addressed by the authors.

Decision: Accept (Poster)