Personality Alignment of Large Language Models
Authors: Minjun Zhu
Yixuan Weng
Linyi Yang
Yue Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Personality Alignment
Large language models
behavioral preferences of LM
Primary area: generative models
TL;DR: We introduce Personality Alignment for language models, efficiently tailoring responses to individual user preferences, providing the‘ PAPI dataset with over 300K subjects and a practical, efficient alignment method.

Abstract: Aligning large language models (LLMs) typically aim to reflect general human values and behaviors, but they often fail to capture the unique characteristics and preferences of individual users. To address this gap, we introduce the concept of Personality Alignment. This approach tailors LLMs' responses and decisions to match the specific preferences of individual users or closely related groups. Inspired by psychometrics, we created the Personality Alignment with Personality Inventories (PAPI) dataset, which includes data from over 320,000 real subjects across multiple personality assessments - including both the Big Five Personality Factors and Dark Triad traits. This comprehensive dataset enables quantitative evaluation of LLMs' alignment capabilities across both positive and potentially problematic personality dimensions. Recognizing the challenges of personality alignments—such as limited personal data, diverse preferences, and scalability requirements—we developed an activation intervention optimization method. This method enhances LLMs' ability to efficiently align with individual behavioral preferences using minimal data and computational resources. Remarkably, our method, PAS, achieves superior performance while requiring only 1/5 of the optimization time compared to DPO, offering practical value for personality alignment. Our work paves the way for future AI systems to make decisions and reason in truly personality ways, enhancing the relevance and meaning of AI interactions for each user and advancing human-centered artificial intelligence. The dataset and code are released at https://github.com/zhu-minjun/PAlign.

Published as a conference paper at ICLR 2025
PERSONALITY ALIGNMENT OF LARGE LANGUAGE MODELS
Minjun Zhu Yixuan Weng", Linyi Yang', Yue Zhang?* !Zhejiang University "Scooo of Engineering, Westlake University
University College London
zhuminjun, zhangyue@wes t lake. edu. en, wengsyxegmail.co yanglinylucd@ gma com
ABSTRACT
Aligning large language models (LLMs) typically aim to reflect general human values and behaviors, but they often fail to capture the unique characteristics and preferences of individual users. To address this gap. we introduce the concept Personality Alignment. This approach tailors LLMs responses and decisions to match the specific preferences of individual users or closely related groups Inspired by psychometrics, we created the Personality Alignment with Personality Inventories (PAPI) dataset, which includes data from over 320,000 real subjects across multiple personality assessments including both the Big Five Personality Factors and Dark Triad traits. This comprehensive dataset enables quantitative evaluation of LLMs' alignment capabilities across both positive and potentially problematic personality dimensions. Recognizing the challenges of personality alignments-such as limited personal data, diverse preferences, and scalability requirements-wve we developed an activation intervention optimization method. This method enhances LLMs' ability to efficiently align with individual behavioral pref- erences using minimal data and computational resources. Remarkably, our method, PAS, achieves superior performance while requiring only 1/5 of the optimization time compared to DPO, offering practical value for personality alignment. Our work paves the way for future AI systems to make decisions and reason personally, enhancing the relevance and meaning of Al interactions for each user and advanc- ing human-centered artificial intelligence. The dataset and code are released at https://ggt whw.bbtthhhhuuggggggggggg com/ thu-min
INTRODUCTION
The alignment with human preferences has been focal point of both theoretical and applied research for Al systems (Shen et al., 2023; Ji et al., 2023; Hendryeks et al., 2023). However, existing alignment approaches often treat society values to individual differences. This broad approach may overlook the nuanced preference of individual users closely related groups, leading to ane-size-fits-all model that fails to effectively serve diverse user needs (Fernandes et al., 2023; Carranza et al., 2023; van Wynsberghe, 2021). This gap is particularly critical as personality alignment can enhance user satisfaction, trust, and engagement by making interactions more meaningful. As shown in Figure 1, we propose "Personality A Alignment":
monolithic construct, lacking the granularity needed to cater
Aligning Al behavior to match an individual's 's unique preferences and priorities of specific individuals, mirroring their behavior, thinking, and decision-making, It ensures Al responses not only are precise but also resonate with the user's traits, understanding their communication nuances and preferences.
As large language models (LLMs) advance (Solaiman et al., 2019; Farina & Lavazza, 2023), AI systems increasingly exhibit societal values and human-like personality traits (Ng et al., 2024; Bai et al., 2022b; Durmus et al., 2023; Zhang et al., 2024; tse Huang et al., 2024; Jang et al., 2024). The
"Correspondence Yue Zhang (zhangyue www.:yueeeewttlak..c....mmmm

---

Published as a conference paper at ICLR 2025
Alignment
Set Standard
Alignment
Extract
discase
Hedffi Hardaa)
AI
Value
Human
AI
Value
Person
Alignment for Universal Value
Alignment for Personality Value
Figure 1: On the left, Al aligns to broad human values like helpfulness, honesty, and harmlessness using a standard set. On the right, the focus on aligning AI behavior with individual users' specific traits and preferences, using detailed profiles to reflect unique personal values.
Role- sole-playing tasks (Li et al., 2023; Salemi et al., 2024; Gupta et al., 2024b; Yu et al., 2024) highlight this trend. However, human preferences are complex, every individual has unique preferences and needs that must be considered to effectively meet user expectations in real-world applications (Pfeifer & Bongard, 2006). For example, in customer service, some users prioritize politeness, while others prefer efficiency. Therefore, personalization is crucial for aligning AI systems with user preferences. Traditional Role-Playing Agents simulate specific professions or roles, rather than deeply aligning with individual preferences (Chen et al., 2024; Kulveit, 2018; Hubinger et al., 2019; Stray, 2020). The lack of lcale,scale, theoretically supported datasets for aligning Al with diverse human personalities and preferences underscores the need for further research in personality and personality alignment (Jang et al., 2023; Lou et al., 2024; et al., 2024b).
There remains practical challenge: aligning a model like Llama-2 typically requires around 3 million sentences (Touvron et al., 2023a). For personality alignment, collecting such an enormous set of behavioral data from individuals is both expensive and unrealistic. Instead, propose training-fre personal-alignment paradigm, providing over 320,000 unique preference samples across two widely-used personality assessments. Specifically, we utilize the IPIP-NEO- 1100-10 questionnaire (Johnson, 2014), which quantifies individual behavior patterns across five dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (De Raad, 2000; Roccas et al., 2002). This questionnaire enables subjects to provide detailed personality measures within 10-20 minutes (Aluja & Garcia, 2004). We collect preference data from over 300,000 subjects complet- ing both IPIP-NEO-120 and IPIP-NEO-300 (Goldberg et al., 1999) questionnaires. Additionally, we incorporate 18,192 192 independent samples from the Short Dark Triad (Jones Paulhus, 2014) assessment, which measures three negative personality traits (Machiavellianism, Narcissism, and Psychopathy) through 27 questions. We manipulate these datasets to build the Personality Alignment with Personality Inventories (PAPI) dataset. For evaluation, we selected 300 representative subjects from each assessment as independent test sets, establishing comprehensive framework for assessing LLMs' personality alignment capabilities across both positive and negative personality dimensions.
To manage the behaviors of millions, the system must be scalable, ensuring quality and efficient alignment. Traditional RLHF techniques, which lack data and computational efficiency (Yao et al., 2023), are unsuitable for this task. Similarly, Prompt Engineering, though efficiently, often varies too greatly between models be reliable for precise alignment (Zhou et al., 2023b). While recent activation-based approaches have shown promise in steering model behaviors (Turner et al., 2023; Li al., 2024a), they typically focus on maximizing a single objective (e.g., safety truthfulness). In contrast, we propose a more nuanced approach that simultaneously searches for optimal activation value offset directions and distances across multiple personality dimensions, ensuring appropriate alignment levels. This balanced activation steering is crucial for maintaining authentic personality expression while achieving desired alignment. We propose a simple and effective personality alignment method: Personality Activation Search. PAS identifies the most effective attention heads in language models (LLMs) using established behavioral patterns that correlate with the five major personality traits. It then adjusts the activations in these specific directions and optimizes the magnitude of movement to achieve the best results. This approach is computationally efficient because it doesn't 't rely backpropagation minimally invasive intervention, resulting in greater data efficiency and more controlled personality alignment than traditional Prompt Engineering (Jiang et al., 2024).
modify model parameters. Instead, PAS offers
2

---

Published as a conference paper at ICLR 2025
We present novel systematic and comprehensive personality alignment framework that enables LLMs to understand and align precisely with specific values, preferences, and behaviors required. Compared to aligning with universal values, personality alignment requires LLMs to align with individual values, preferences, and behaviors, which ensures that the LLMs can accurately match or mimic the specific individual characteristies that users need and pursue. To achieve this goal, we introduce the PAPI dataset, which covers over 320,000 subjects of different ages, genders, and periods using multiple-choice questions comprehensively measure both positive and negative personality traits. Based on that, we propose the PAS method, which aims to address the challenges of scarce individual characteristic data and high scalability in personality alignment. Experiments have shown its high alignment efficiency compared to DPO and PPO (Schulman et al., 2017; Rafailov et al., 2023) where it requires only 1/6 of the time to achieve superior performance. It even outperforms the GPT-40 model using PAS based on the Llama-3-8B model.
2 RELATED WORK
Alignment with language models. The concept of integrating human values into Al systems, initially proposed by Dewey Dewey (2011), underscores the necessity of value learning, which entails the Al's ability to process and prioritize broad spectrum of human values and preferences. In the context of values crucial, especially since they operate in complex reasoning tasks across various domains (Hadar-Shoval et al., 2024; Wei et al., 2022b;a; Weng et al., 2023). Methods such as RLHF (Stiennon al., 2020; Bai et al., 2022a; Dong et al., 2023) and RLAIF (Lee al., 2023; Sharma et al., 2024) seek to align LLMs by employing strategies like PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023). These approaches focus on maximizing cumulative rewards that are closely aligned with predefined human values (Azar et al., 2023; Lin et al., 2023; Dai et al., 2024; Swamy et al., 2024; Dai et al., 2024). Contrary to these broad approaches, our research focuses on personality alignment, which tailors the LLMs' behavior to each individual's complex value set, thereby enhancing the model's 's capacity to understand and reflect individual human values.
as high- tigh-performance assistants aligning these models with human
Preferences of language models. The community of LLMs has started using personality testing tools for both qualitative and quantitative evaluations (Hoover et al., 2019; Jiang et al., 2022: Qiu et al., 2022; Xu et al., 2023; Salemi al., 2024; Wang et al., 2024c). For example, the Machine Personality Inventory (MPI) Jiang et al. (2024), utilizes the Big-Five Personality Factors to standardize the assessment of LLMs. This approach analyzes the behaviors of LLMs across the five dimensions of the Big-Five, providing a quantitative interpretation of the values and personality traits of LLMs. Such tools are grounded in widely accepted theories, including the Big-Five and the 16 Personality Factors (Cattell & Mead, 2008), helping to contextualize LLMs' personality traits in terms of human individual differences (Gupta et al., 2024a; Wang al., 2024b; Aher et al., 2023; Liu et al., 2024a; Li et al., 2023). These studies indicate that aligned LLMs share similar values with humans (Mehta et al., 2020; Pan & Zeng, 2023; Mao et al., 2023; Go et al., 2023; Chen et al., 2024). Jang et al. (2023) has also mentioned the similar term "personality alignment", which considers a limited scope of preference personalization (Wang et al., 2024a; Zhuang et al., 2024; Zeng et al., 2023), focusing on only different conflicting user preferences derived from combinations of three dimensions: Expertise (elementary vs. expert), Informativeness (concise vs. informative), and Style (friendly vs. unfriendly). Our work extends these theories and methods by leveraging psychological measurement to assess the value alignment gap between aligned LLMs and individual users. We aim to understand how closely aligned LLMs are with individual values through personality and personalized adjustments.
Interventions Activate. Activation interventions in neural networks trigger specific changes in internal activations (Olsson et al., 2022; Wu et al., 2024a) and have emerged as a pivotal tool in model robustness (He et al., 2019), editing (Meng et al., 2022), circuit finding (Goldowsky-Din et al., 2023), and knowledge tracing (Geva et al., 2023). These interventions are advantageous due to their adjustable nature and minimal invasiveness (Wu et al., 2024b). Prior research has demonstrated that activation interventions can effectively identify the crucial heads and directions a model's internal representations (Burns et al., 2023a; Liu et al., 2024b), which offers better data efficiency and faster optimization than traditional reinforcement learning approaches (Turner et al., 2023; Rimsky et al., 2023; Shi et al., 2024; Weng et al., 2024a). Prior activation steering work has focused on maximizing single objectives like safety (Wang& & Shu, 2024; Zheng et al., 2024) or truthfulness (Li et al., 2024a). Building on these findings, our method enhances scalability by pinpointing value directions of
3

---

Published as a conference paper at ICLR 2025
different individuals and calculating optimal offset distances, aiming achieve appropriate alignment for each dimension while maintaining model capabilities.
3 PERSONALITY ALIGNMENT DATASET CONSTRUCTION
(a) The example of Question with Statement
(b) The example of IPIP-NEO- 300 IPIP-NEO-120
IPIP-NEO-300 Preson 12138
IPIP-NEO-120
Queation: Giver atakement of you "You ('Srddompee)"
Please chscae from owing opticta denally
potom::
Miderrady Acjureto
offine
(+]
Operaca
Ayrnnblress
Ne Accurate Inaccurtt
Mocorsstlly
(5) Very Inaccs 199
Prosen, 10r__12111 12138
Veyy Actuntss
Rearet cire
San bracusiee
Nauroriuun
Extriveraioo
Megilen
1-31
(c) The volues, preferences, and persondities of 12138
Figure 2: Overview of the PAPI dataset. (a) Illustrates the comparison between the subject's 's self- assessment and the Al's assessment of specific question. (b) Shows an example of the IPIP-NEO-120 and IPIP-NEO-300 questionnaire responses. (c) Depicts the Big Five personality traits profile.
Motivation. To systematically evaluate the consistency between cissonality-aligned LLMs and human preferences, we create the large-scale Personality Alignment with Personality Inventories (PAPI) dataset, comprising 325,505 samples across multiple-choice questionnaires. The dataset uniquely combines two complementary personality assessment systems: 307,313 samples from the Big Five personality traits (IPIP-NEO assessments) and 18,192 samples from the Dark Triad inventory. While the Big Five (IPIP) samples measure adaptive personality traits through IPIP-NEO- 120 and IPIP-NEO-300 questionnaires, the Dark Triad samples assess subclinical maladaptive traits MMachiavellianism, Narcissism, and Psychopathy) through 27 targeted questions. This comprehensive approach captures both healthy personality dimensions and potentially problematic traits. As shown in Figure 2, subjects choose behaviors similar to their own (a), yielding questionnaire responses (b) that collectively form detailed personality profiles (c).
Question Type. Our dataset contains two independent questionnaire systems, each using a five-point scale from "Very Accurate" to "Very Inacourate." " The first system comprises IPIP questions associated with the Big Five personality dimensions. We use each subject's 's IPIP-NEO-120 questionnaire responses to predict their IPIP-NEO-300 answers, testing the model's ability to generalize from known personality patterns to broader behavioral descriptions. The second system consists of Dark Triad assessments, where we use 18 questions (6 each for Machiavellianism, Narcissism, and Psychopathy) to align with and predict responses to the remaining 9 questions (3 per trait). Both systems evaluate the stability of individual personality tendencies and the model's 's capacity to capture and generalize personality patterns, though they operate independently on distinct trait dimensions.
Data Collection. We have collected 307,313 samples based on the IPIP-NEO-120 and IPIP-NEO-300 from the International Personality Item Pool (IPIP)'. !Thss includes approximately 60% (185,149) female and 40% 164) male participants, covering ages from 10 to 99 years (average age 25.19 years). The data were collected through online surveys on website, requiring participants to provide genuine feedback on 120& & 300 behavioral questions. Participants are volunteers from various industries worldwide, includ- ing the US UK. France, India, and China, with tests conducted between 1998 and 2019. Each participant had to actively acknowledge that careless responses would invalidate the usefulness. Each survey took between 30 to 50 minutes, was anonymous, and col- lected no traceable or socioeconomic status-related data. We have permission from the administrators
Coneet Sample
responea
Figure 3: Visualization of the K-Means Clus- tering on the PAPI Dataset. The centroids are marked in red, demonstrating the central point of each cluster, while the closest samples to these centroids are highlighted in gold.
https:://ppp..n .org/iorg///ooo.. ndex. htm
4

---

Published as a conference paper at ICLR 2025
use IPIP items, scales, and inventories for any purpose, commercial non con-commerlaal. Additionally, we collected 18,192 Dark Triad samples' using the same collection pipeline and quality control procedures.
Data Processing. We apply K-Means clustering separately to the IPIP and Dark Triad datasets. From the IPIP dataset (307,313 samples), we select 300 representative clusters the first part of our Test-Set. Similarly, we cluster the Dark Triad dataset (18, 192 samples) and select 300 representative samples, forming the second part of our Test-Set. The remaining data constitutes our Dev-Set. The large-scale Dev-Set captures personality variations across different ages (from teenagers to seniors), genders, cultural backgrounds (spanning multiple countries and regions). and identity groups. This comprehensive representation is crucial for developing inclusive alignment methods that respect and reflect the full spectrum of human personality diversit. Sample selection minimizes Fuclidean distance to cluster centroids: minzes, Ix: P& where /48 is the centroid of cluster k; and x is a sample within cluster (Sx), aiming to optimize cluster representativeness by minimizing the distance between samples and their respective centroids.
Evaluation Methods. Inspired by the MPI (Jiang et al., 2024), we set up a behavioral difference score an evaluation metric. Using a psychological measurement approach, we establish simple scoring function S: for positively correlated questions, we assign scores from to from "Very Accurate" to "Very Inaccurate". We expect the aligned language model scores be close to specific individual's 's scores, hence we use an absolute value method for measurement. We test each trait d in the test section, with each participant's behavioral difference Aligned Score calculated as follows:
M
Aligned Score,
Σ KED
I/ (LLLM(xx - (Person(Ks, template)]] (I)
M
Nd.
In this formula, Deeat denotes the set of behaviors related to trait in the test section for subject i. We use the given template to obtain responses from both the LLM and the Person, and then calculate the absolute difference between the two. NN., represents the count of these behaviors. Each x is an independent behavior sample from this set, with the LLM's responses evaluated against those in Train, The f(.) represents the scoring function for multiple-choice questions. We calculated the deviation distance between the two using the absolute value method and computed the average deviation distance for all behaviors. The range for Score is from 0 to 4, where 0 means perfect alignment with individual behavior.
PERSONALITY ACTIVATION SEARCH
Popular alignment techniques like PPO, DPO, and IPO strive to align language models with desired behaviors, but personalizing LLMs for individual preferences without extensive training presents significant challenge. To address this, we' we've developed Personaliyy-Activvne efficient method that fine-tunes model activations to closely match human preferences, PAS identifies key activations within the model that significantly impact Personality and personalized behaviors and adjusts these activations toward specific preferences.
Search (PAS), an
PAS works by identifying multiple directions the transformer's activation space that satisfy personality attributes and then intervening along these directions to adjust the activation values, achieving minimally invasive alignment. The entire process does not require changing the model's weights or backpropagation. It only requires multiple forward passes to determine the offset direction and distance, resulting in a high optimization efficiency.
4.1 LANGUAGE MODELS
In our research, we focus on a transformer- language model which may be in various stages, including pre-training, supervised fine-tuning, or undergoing RLHE. The transformer model primarily consists of several layers, each equipped with a multi-head attention (MHA) mechanism and multilayer perceptron (MLP) (Vaswani al., 2017).
httpp://ppeensycomttrccc...
5

---

Published as a conference paper at ICLR 2025
Search Directions
Probing every
Interven Directions
STEP3 Search Distance and intervene
IIP-NBO-300 Frezz
Bnite yoinet
Tock Prob
Preference
STEP1
Select
Language Model
Phe Trcinin soddl
KUIf ralel
Hunan
A1
6PT-2
Appse
Usnne>
Falson
Mii
Personality Alignment
Language Model
Intervention Directions
Alignmented Language
Figure 4: Overview of the PAS Process. Step 1 involves selecting LLM. Step includes creating and activating personality responses using the answer of questionnaire, the subject's s answer in the PAPI dataset, and probing each attention head. Step 3 integrates these activations for personality alignment, adjusting model outputs to reflect individual user traits, resulting in an aligned L.I.M
During inference, input tokens are encoded to high-dimensional embeddings, where in each trans- former layer, the MHA process is defined as #1+1 Iı Σ Of Au (Pftx1), with H being the number of heads, E DD representing the activations, and P/ DDD and Qf E DDDD projecting activations into and back from specific headspace.
While previous methods like Inference-Time Intervention (ITI) adjust language models by applying fixed, large shift in the activation space to influence output biases, this approach inadequate for addressing alignment issues, especially when personalization is required. Fixed shifts can be too coarse and may not capture the nuances of individual preferences. To overcome this limitation, we have enhanced the intervention strategy by dynamically adjusting the activation distances based on personal preference vectors. Specifically, we introduce these vectors into the residual stream between the Attention mechanism and the projection matrix Q/- This allows to fine-tune the model's activations in way that aligns with specific user traits and preferences, facilitating personalized alignment without altering the model's weights or relying on large, inflexible shifts.
4.2 SEARCH THE DIRECTIONS FOR ACTIVATION INTERVENTION
To achieve personality alignment, we trained "probes" to get activations as the direction of personal preference. started by selecting statements from the training dataset each subject, formatted as positive samples: "Question: Given statement of you: 'SStalement', Do you agree? Answer: Yes", you and negative samples: "Question: Given a statement you: 'SStatement' Do you agree? Answer: No". Activations from each head retained and split into training set (60%) and validation set (40%). These activations serve as features for the probe (Equation 2) of per attention head.
Po (x (0igmmid((0,..))) of1),
where the x/ E RD. and 0 E RD is the weights h-heads in layer in the residual stream.
We identify the most influential heads based their predictive accuracy during validation. Notably, it is necessary to disperse interventions during this process (Li et al., 2024a). Therefore, we select the top K probabilities for intervention. The parameters 0 from these probes, after normalization, treated as the directions along which behaviors are most separable. These directions are the most informative for aligning the model with individual preferences (Burns al., 2023b).

---

Published as a conference paper at ICLR 2025
During inference, we integrate the intervention into the output of each selected head:
21-1 37 Σ of (Ait) + aot).
(3)
where a controls the adjustment intensity, and of is the standard vectors along the normalized direction of. We adjust activations in the personality direction, focusing only on the top K heads, while leaving the unselected heads unchanged. In practice, the model parameters remain unaltered; we only need to store approximately 1,000 parameters, denoted as of These of parameters represent the direction of activation shifts towards personality preferences, rather than memorizing specific preferences. This approach is highly -eanetter-effcient,, as the of parameters constitute only about 1/10,000,000 of the total parameters compared uull-parameter model.
4.3 SEARCH DISTANCE FOR ACTIVATION INTERVENTION
To optimize the intervention's 's impact on personality behaviors in the language model, we search to determine the optimal value of a precisely. This method is suitable for functions with a single peak trough. The search, constrained within the interval [0, 10). utilizes the objective function l(a) to compute the negative aggregate score, aiming to minimize this value. This approach efficiently identifies the a that best enhances alignment with desired behaviors while maintaining the model's internal coherence, ensuring adjustments remain within a beneficial range.
Optimal a = , fa)).
(4)
By iteratively narrowing the search range based on the function's 's values at each step, this method efficiently locates the that yields the lowest mean Score.
EXPERIMENTS
Merkad
Algunex Made
Apprntecoo
LEBOCRREIEES EMERVCIRAL
Rg-
Dark 7Hid
KESTINEE Pystopatt
GPT-4rjomni)
Black Bex Bladd
1.60
1.46
2.04
12.02
White Box (Alggsenti
1.1
1.45
1.42
1.51
1.48
1.90
3.19
13.27
DPO
White (Algement)
3.12
13.07
Posseellan Stage
Whac-Bon (Alipnsect)
116
091
0.59
1.30
0.50
110
8.76
1
5.86
Black Bhe (Poeae Bando
1.36
1.50
3.17
2.00
12.24
Ware (Alipemee))
8.91
0.72
8.00
3.80
1,67
$.899
3-708-37708-444 1-.0--------tneee Justrnce
7T0
Whase Sox (Algement)
Whsc- Soo (Algsansi)
1.30
1.11
1.00
1.30
1.34
1.15
2.50
1.76
15.07
Sax
Bos (Poeass-BBo..
0.66
0.56
1.22
1,66
1.80
1.80
3.06
Buc (Proagg Bandd
1.00
100
199
Table I: Comparison of alignment methods on the PAPT dataset using the Aligned Score of Big-Five personality traits and Dark Triad traits. The Score represents the overall specifics of all dimensions.
5.1 EXPERIMENTAL SETTINGS
Personality Alignment. We select four classic methods as baselines compare against the PAS approach, which includes two black-box methods: Few-Shot and Personality Prompt (P?) Jiang al. (2024) relying on in-context learning (ICL), and two white-box methods requiring training, PPO (Schulman et al., 2017) and DPO (Rafailov et al., 2023). We consider recently released Llama- 3nSt-untt and Allmma---70B-Instteee and SoTA model, GPT-4o (omni) (OpenAI, 2024) as baseline. The scores in the table represent the distance of alignment calculated by Eq. 2, with lower scores indicating better alignment.
(Touvron al., 2023a;b; AI, 2024) model as backbones,
and Llama-3-70B- Instruct. For the open-ended generation task, we use GPT-4o as the annotator. First, we align the language model to a specific individual (ID 181591, 234619, 210204, 259396, and 22835 from the Sest-Set). Then, we have the model generate open-ended responses based on each statement from the IPIP-NEO-300 We compare method with ICI methods and classical alignment methods.
Open-ended Generation. We conduct experiments on
7

---

Published as a conference paper at ICLR 2025
Complex Reasoning. We explore the generalization capabilities of the PAS method by evaluating its impact on complex reasoning tasks across eight datasets: GSM8K, CommonSenseQA. AddSub, MultiArith, SVAMP, BigBench-Des Date, StrategyQA, and Coin Flip (Hosseini al., 2014; Talmor et al., 2018; Arkil et al., 2021; Cobbe et al., 2021; Suzgun et al., 2022; Roy & Roth, 2016; Wei et al., 2022b; Kojima al., 2022; Weng al., 2024b). In this part, we expect assess how conscientiousness adjustments influence the LLM's performance in reasoning tasks. Using the llama-3-8B-Insee model, we apply PAS adjust the model's activations to simulate two fictional personas: one with extreme conscientiousness (as Positive) and one without extreme conscientiousness (as Negative). Vanilla CoT served as the baseline.
5.2 RESULTS ON PAPI
Table demonstrates that the PAS method achieves state-of- the-art performance across differ- models and personality di- mensions. For the Llama-3-8B- Instruct model. PAS achieves Composite Score of 8.89, signifi- cantly outperforming other meth- ods including Personalized-Soups (9.66) and Prompt-MORL (10.88). Similarly, for the Llama-3-70B- Instruct model, PAS scores 9.21,
Performance (Score)
P90
Ponurally Poont
1x
5x
10x
16x
2bx
25x
Effidency
Figure 5: Comparison of performance and efficiency for various alignment methods. The larger the multiple on the x-axis, the lower the efficiency.
compared to 9.94 for the next best method. In Big Five dimensions, PAS excels with consistently low scores across both model sizes: Agrecableness (0.94/0.98), Conscientiousness (0.91/0.) 1/0.89), Extraver- sion (0.86/0.87). Neuroticism (0.98/1.01) and Openness (0.72/0.999. Notably, PAS also demonstrates strong performance in Dark Triad traits, particularly in Machiavellianism (0.96/1 .01) ) and Psychopathy (1.67/1.62), though showing relatively higher scores in Narcissism (1.85/1.84) 188). Only GPT-4o shows better performance in Dark Triad alignment with scores of 0.80, 0.76, and 0.83 for the three traits respectively, highlighting the challenge of aligning these complex negative personality dimensions in smaller models. In the appendix E.4, we also conducted further performance comparisons with full parameter fine-tuning and LORA. PAS similarly demonstrated higher efficiency and superior performance. In appendix E.5, we provide more comprehensive analyses across diverse demographic groups, including detailed distributions of age, gender, and nationality, along with in-depth examina- tion of performance patterns across various underrepresented populations, further validating PAS's effectiveness in handling diverse personality characteristics.
Efficiency. As shown in Figure 5, training a separate language model for each individual demands (e.g. DPO and PPO) substantial computational resources and time, making it an infeasible solution for personality alignment. PAS addresses efficiency challenges by making targeted adjustments to a small subset of activation values during inference, in specific directions. This approach requires minimal computational overhead, similar to Few-Shot methods, and only about 1/6 of the time needed for PPO, yet achieves remarkable personality alignment results. By intervening at the activation level rather than relying on extensive training or the variability of prompts, PAS provides a more efficient and precise method for aligning language models with individual user preferences.
Alignment vs. ICL. Our experiments highlight that direct Alignment (White-Box) methods, es- pecially PAS, outperform In-Context Learning (ICL) (lack-Box)) methods. By aligning model parameters with user preferences, PAS provides better performance compared to prompt- approaches. The PAS method's ability to efficiently shift activations towards user preferences without altering the model's I's core parameters gives a significant advantage over traditional ICL. methods.
Why Did Scaling Laws Fail? The results indicate that scaling laws alone do not ensure optimal align- ment, particularly domain -specifc tasks like personality alignment. Although larger models like 3lama-3-70B-Insten benefit from increased scale, their broader knowledge and more general align- ment capabilities may actually hinder their ability to precisely capture individual personality traits. This phenomenon likely occurs because larger models trained to maintain general- -prrpose capabil- ities and broad knowledge, making them less adaptable to highly specific individual preferences. PAS demonstrates that targeted intervention in activation patterns can achieve better personality alignment
8

---

Published as a conference paper at ICLR 2025
than parameter modification model scaling. By specifically focusing personality patterns rather than general knowledge, PAS achieves superior performance with computational efficiency, emphasizing the importance of precise control over raw model size.
5.3
GENERALIZATION RESULTS
Llam-3-88-Instruct
Wins
Ties
PAS Loses
Llama-3-708-Instruuc
Fow-shot
27%
Few-shot
44h
pa
31%
40%
p4
54%
27%
27%
DPO 19%
23%
99%
34'
PPO
23%
PPO
100%
50%
Figure The win rates of PAS compared classical alignment methods evaluated by GPT-4o.
Evaluation
Human-Self Evaluation
Few-shot
41.0%
26.32%
20.174
Few-shot
32.2%
21 M%
po.
10.15%
24.57%
p7
50 39.74
24 67%
DPD
15.55%
17.5%%
DPO
72.67%
17.7%
FPO
10.2%
20.22%
10 67%
PPO
25.67% 13.33
0/
sus
100%
0%
50%
100%
Figure 7: The win rates of PAS compared to classical alignment methods evaluated by Human.
Considering that during the process of adjusting the model, although specific goals can be achieved, it may also affect other general capabilities of the model, such as reasoning abilities. We demonstrate through general dialogue tasks and complex reasoning tasks that our method can precisely control the model's 's personality without compromising its other generalization capabilities.
Open-Ended Generation. We used an open-ended generation task where the language model elaborated on given statements from the IPIP-NEO-300 covering "Scenario Description," "Actions Taken," "Thought Process," " and "Emotional Response." The generated outputs were evaluated by GPT-4o for alignment with individual preferences, similar to the evaluation process in LIMA (Zhou al., 2023a). Despite the Personality Aligmment being initially trained with multiple-choice options, Figure shows that PAS does not affect the model's general conversational ability and reasoning capacity, it significantly outperforms baseline methods in this open-ended task. This superior performance can be attributed to PAS's ability to fine-tune model activations to capture individual preferences effectively, enhancing the model's adaptability to complex and nuanced tasks. PAS's ability to generalize from multiple-choice training data to natural, open-ended generation tasks demonstrates its capacity to capture broader behavioral patterns while maintaining fluent text generation. In addition to this, we conducted manual evaluation using the same setup Figure 7. In addition to this, we conducted a manual evaluation process using blind setup. Three human evaluators assessed the generated responses for three different subjects (Human Evaluation, the subset of test set: ID 172481, ID 22835, and ID 259396). These evaluators also judged responses aligned to their own personalities (Human-Sed Self Evaluation). They evaluated the consistency between the target values and the model-generated text in terms of values, behaviors, and preferences. The results demonstrate that our PAS method consistently outperforms other approaches, reflecting its superior alignment with both general human and self-specific preferences. Further details of the manual evaluation are available in Appendix E.2.
Complex Reasoning. Experimental results, shown in Figure 8, indicate that PAS significantly enhances the model's reasoning capabilities when conscientiousness is positively adjusted. For instance, on the GSM8K dataset. the baseline performance was 73.47. Enhancing conscientiousness increased the performance to 74.15 while reducing conscientiousness dropped to 72.75. Similar trends were observed across other datasets, with the model consistently outperforming the baseline

---

Published as a conference paper at ICLR 2025
GSMBK
CommenBense@A
Add8ub
Muttiarth
Pasifiee
gaghteee
Pos Prea
Magasiva
Poeiiiv
Nogative
Posties
Nagetive
SVAMP
Bigbench Date
StrategyQA
Flip
00001
80001
$5.00
$4.90
14:23
02400
Nagetive
Nagetive
Figure 8: Performance of the Merma-3-8B-Imge model on complex reasoning tasks across eight datasets. We used Vanilla CoT (Wei et al., 2022b) as the baseline (dashed line in each subplot).
when conseientiousness was increased. This suggests that conscientiousness, associated with diligence and carefulness, positively impacts the model's ability to handle complex reasoning tasks. Importantly, these interventions did significantly alter the inference time, maintaining efficiency comparable to the original model. These findings demonstrate PAS's strong generalization capabilities.
5.4 DISCUSSION: Is VALUE-ALIGNED ASSISTANT A GOOD ASSISTANT?
Certainly. We explored whether aligning a language model with user values enhances satisfaction using 300 general question-answer examples from the LIMA dataset (Zhou et al., 2023a). We evaluated both the original Alama-3-70B-Instrue model and Personality Alignment model. Each human evaluator spent minutes completing an -PI--NEO-22 questionnaire to extract their values. We aligned the model with these values with the PAS, creating Value-Aligned Assistant. For comparison, we also created Value Malue-Misaligned Assistant by inverting the questionnaire responses.
Human evaluators assessed the re- sponses to the test prompts. Fig- ure 9 shows that the Value-Aligned Assistant provided preferable outputs 38% of the time, with 31% ties, sig- nificantly outperforming the Value- Misaligned Assistant. For instance, conservative person may prefer more cautious advice, which the Value- Aligned Assistant is better equipped to provide. These results underscore the importance of aligning AI systems with user values to enhance satisfac- tion and effectiveness.
Wins
Ties
Loses
Value-Misaligned Assistant
20%
33%
38%
Value-Aligned Assistant
0%
Figure 9: Performance comparison of the Value-Aligned Assistant and Value-Misaligned Assistant the LIMA dataset's 300 test prompts. We used the original Llama- 1-70-Innsruce as the baseline comparator, anonymizing the models and randomizing their order during the evaluation.
CONCLUSION
We introduce the concept of Personality Alignment for language models, emphasizing the importance of tailoring AI behavior to individual user preferences and values. We release a large-scale real- world personality alignment dataset named PAPI, characterized by its real, diverse, and extensive data, to quantitatively measure the degree Personality Alignment. Based on that, we propose Personality Activation Search (PAS) method, which offers practical solution with high efficiency and performance. LLMs adjusted with PAS can achieve better results and better efficiency in personalized alignment than ICL, DPO, and PPO as needed. In particular, it achieves precise alignment with just 1/6 of the computational time required by PPO. Experiments demonstrate that PAS enhances the performance of Personality alignment by making Al interactions more relevant and meaningful.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This publication has been supported by the National Natural Science Foundation of China (NSFC) Key Project under Grant Number 62336006.
We thank all participants who contributed to the PAPl dataset.
ETHICS STATEMENT
While personality alignment in language models enhances human-Al interaction, it raises significant ethical concerns that warrant careful consideration. The primary challenge lies in the potential creation of psychological filter bubbles and echo chambers. When AI systems consistently align with individual personality traits, they may limit exposure to diverse perspectives, particularly concerning for individuals scoring high in Neuroticism or low in Openness. This effect becomes more pronounced when considering Dark Triad traits- - for users exhibiting high Machiavellianism or Narcissism, aligned AI systems might inadvertently validate manipulative tendencies excessive self-focus, potentially contributing to social polarization and ideological segregation in online communities.
The implementation of personality alignment also presents substantial privacy and security challenges. While our dataset maintains anonymity, the processing of detailed psychological profiles creates potential vulnerabilities. The risk extends beyond data security the possibility of exploitation. where knowledge of individual personality traits could be used for sophisticated manipulation or targeted advertising. Furthermore, extended interaction with personality-aligne Al systems might foster psychological dependence, potentially inhibiting personal growth and adaptation, particularly users with extreme personality trait scores.
To address these concerns, we suggest comprehensive mitigation framework with three key com- ponents. First, we recommend implementing dynamic alignment boundaries through monitoring system that tracks interaction patterns and adjusts alignment intensity in real-time. This system would incorporate threshold mechanisms that automatically reduce alignment strength when user-Al interactions show signs of reinforcing extreme or potentially harmful behavioral patterns. Second, suggest an adaptive content diversity system that strategically introduces alternative viewpoints while maintaining personality alignment. This could be achieved through balanced scoring mechanism that weighs alignment accuracy against content diversity, ensuring users receive personality-aligned responses while still being exposed varied perspectives. Third, we advocate for robust privacy protection framework that includes: (a) data anonymization techniques that separate personality profiles from identifiable information, (b) secure encryption protocols for personality data storage and transmission, and (c) granular user controls over which personality aspects can be used for alignment.
Looking forward, our research emphasizes the critical balance between technological advancement and ethical considerations in AI development. Future work should focus on implementing and evaluating these mitigation strategies, particularly examining their effectiveness preventing echo chambers while maintaining the benefits of personality alignment.
REFERENCES
Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pp. 337-371. PMLR, 2023.
Meta AI. Introducing meta llama 3: The most capable openly available Ilm date. https: // meem. a. www.coo////oo..eeeeooo met a 1lamamm.33.. 3/, 2024. Accessed: 2024-05-21.
Ebtesam Almazrouci, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311. 166667, 2023.
Anton Aluja and Luis Garcia. Relationships between big five personality factors and values. Social Behavior and Personality: an international journal, 32(7))619-6)) 625, 2004.
11

---

Published as a conference paper at ICLR 2025
Patel Arkil, Bhattamishra Satwik, and Goyal Navin. Are nip models really able solve simple math word problems? 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077 2022b.
Tom L Beauchamp et al. The belmont report. The Oxford textbook of clinical research ethics, pp- 149-155 2008.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023a. URL https:////// http://ppppweeeeeee/e/s//e////ep net wwo.ooormiiooooooo weT.umidddEKKGyylnii..
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in language models without supervision. In The Eleventh International Conference on Learning Representations, 2023b. URL / http://ppnreeeee.eeeeee net http://wwwwweeeeeeeeeeeeeeeeeeee forum? ?id ETTG-bKKGG Ohes.
Andres Carranza, Dhruv Bhandarkar Pai, Rylan Schaeffer, Arnuv Tandon, and Sanmi Koyejo. Deceptive alignment monitoring. In The Second Workshop on New Frontiers Adversarial Machine Learning, 2023. URL https: wwwpreview.eeepp/.pppp net forum? ferrmmid==obo44 4GFhh.
Heather EP Cattell and Alan D Mead. The sixteen personality factor questionnaire (16pf). The SAGE handbook of personality theory and assessment, 2:135-159, 2008.
Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. From persona to personalization: A survey on role-playing language agents, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10.11621, 2021.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedhack. In The Twelfth International Conference on Learning Representations, 2024. URL https: : / /ppnreveeeeeeeeeee net/ forum? id=TyF=POKYXW.
Boele De Raad. The big five personality factors: the psycholexical approach to personality. Hogrefe & Huber Publishers, 2000.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 Q): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305, 14314, 2023. URL https: / //api..p./////// semant icscholar. Drg/CorppstD: 258841328.
Daniel Dewey. Learning what to value. In International conference on artificial general intelligence, pp. 309-314, Springer, 2011.
12

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
URL https: wwpppppeevviiieeeeeeee net / forum? d=m7p5077bl1
Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfiell-Dode Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306. /6388, 2023.
Lilian Edwards. The ai act: summary of its significance and scope. Artificial Intelligence (the EU Al Act), 1, 2021.
Mirko Farina and Andrea Lavazza. Chatgpt in society: emerging issues. Frontiers in Artificial Intelligence, 6: 130913, 2023.
Patrick Fernandes. Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, and André F. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation, 2023.
Phoebe Friesen, Lisa Kearns, Barbara Redman, and Arthur I. Caplan. Rethinking the belmont report? The American Journal of Bioethics, 17(7):15-11, 2017.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235 Singapore, December 2023. Association for Computational Linguis- URL httpt:tt///eae httpp//s//acaaaalllloaooooooo//e org/2023.
tics. doi: 10. 18653v 1/2013.222e///2 h/t0:/..mmll-mmaaie.... m51... emm.pp ma in. 751.
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, and Marc Dymetman. Composi- tional preference models for aligning Ims. arXiv preprint arXiv:2310.1300 13011, 2023.
Lewis R Goldberg et al. A hooad-bandwidth public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality psychology in Europe, 7(1):7-28, 1999.
Nicholas Goldowsky- Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, arXiv preprint arXiv:2304.05569, 2023.
Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. Self-assessment tests unreliable measures of LLM personality. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 301-314, Miami, Florida, US, November 2024a. Association for Computational Linguistics. doi: 10. 1/6653///112224 http:///w////....aaaaooooooo//a blackboxnlp- 1.20. URL https: /a/aa/aaaanttoooooo/o wttp://r.............ooooooo. www.kkbboopp-1..00000
Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In The Twelfth International Conference on Learning Representations, 2024b. URL
https:/ /apenrevvvv ew net forttt/frummmmmm for.mmid= det.meeeetttet77.....
Dorit Hadar-Shoval, Kfir Asraf, Yonathan Mizrachi, Yuval Haber, and Zohar Elyoseph. Assessing the alignment of large language models with human values for mental health integration: Cross- sectional study using schwartz's 's theory of basic values. JMIR Mental Health, 11:e55988, 2024.
Zhezhi He, Adnan Siraj Rakin, and Deliang Fan. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 588-597, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks, 2023.
13

---

Published as a conference paper at ICLR 2025
Joe Hoover, Gwenyth Portillo- Wiihhlon,,,,,,,,, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani, Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, Gabriela Moreno, Christina Park, Tingyee E. Chang, Jenna Chin, Christian Leong, Jun Yen Leung, Arineh Mirinjian, and Morteza Dehghani. Moral foundations twitter corpus: A collection of 35k tweets annotated for moral sentiment. Social Psychological and Personality Science, 11:1057 1071, 2019. URL https: / /api. semant icscholaae ar org/ Stag//CrpussiDi :ttp:/ooopuDD:::36655553 211669988.
Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. empirical methods in natural language processing, 2014.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. ArXiv, 0ess/100.009885 2021. URL https: /appi. semant iommaattiiccce wew.chossshhacccc
arg/CorpusIDi
Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv: 2019.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoe parameter merging. ArXiv, abs/2310.1 15664, 2023. URL https: /api. semantics semanticscholar org/ Corpus ID: 642889231.
Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannanch Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoe parameter merging. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024. URL https:: net/Eorum -ediiddMMeeee Rvxe.
Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey, arXiv preprint arXiv:2310.199
2023.
Guangyuan Jiang, Manjie Xu, Song-Chun Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evalu- ating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems, 36, 2024.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. Can machines learn morality? the delphi experiment, 2022.
John A Johnson. Measuring thirty facets of the five factor model with a 120-item public domain inventory: Development of the ipip-neo-120, Journal of research in personality, 51:78-89, 2014.
Daniel N Jones and Delroy L Paulhus, Introducing the short dark triad (sd3) brief measure of dark personality traits. Assessment, 21(1):28 41, 2014.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization.
CoRR, abs/1412.6980 2014. URL https: /appi semanticscholess /rgg/CorpusSDDD 6628106.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural information Processing Systems, 2022. URL https: /atp:://ppenreeee wwtp//ppeeeeeeeeeeeeeeeee n wttp://wwwweeeeeteeeeeeeeeo detrummmdde2BBByyyyy00
Jan Kulveit.
Multi-agent
predictive minds and ai alignment.
ht https: htt:://wwwww. http://wwwwwsewsoooo www.wwesswenn.................o. com wwtp:///ssstte////e////eeeaeeaee
2018. Accessed: 2024-04-
25.
14

---

Published as a conference paper at ICLR 2025
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXhv:2309.00267 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning, 2021. URL https: :tap:s///axxat http///xxvvv.o wwwp:i/vvorrroooooooo abs 2204. 08691.
Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang. Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. Chatharuhi: Reviving anime character in reality via large language model, 2023.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024a.
Xinyu Li, Zachary C. Lipton, and Liu Leqi. Personalized language modeling from personalized human feedback. ArXiv, abs/2402.05133, 2024b. URL https: /ttps:///ppp /appppappp.///// semant icscholar. org/Corruu ID 267547503.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, 2023.
Jianzhi Liu, Hexiang Gu, Tianyu Zheng, Liuyu Xiang, Huijia Wu, Jie Fu, and Zhaofeng He. Dynamic generation of personalities with large language models. arXiv preprint arXiv:2404.07084 2024a.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello. Quentin Berthet, Felipe Llinares-López, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathicu Blondel. Decoding- time realignment of language models. ArXiv, abs/2402.02992, 2024b. URL https: /api.
semant i cschhllr org/ htt.g//orrusaaiae
ID 22w.111:2779999
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaiqi Huang. Spo: Multi-dimensional preference sequential alignment with implicit reward modeling. ArXiv, Abb/2905.12239, 2024. URL https: //api//appppie stt.appp..aaaaaaaaaaeaamamaaa..a semant weschhlar.......o.o www.aarrrrgoooooorooee 269930031.
Shengyu Mao, Ningyu Zhang, Xiaohan Wang, Mengru Wang, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Editing personality for llms. arXiv preprint arXiv:2310.02.66. 2023.
Yash Mehta, Samin Fatehi, Amirnohammad Kazameini, Clemens Stachl, Erik Cambria, and Sauleh Eetemadi. Bottom-up and top-down: Predicting personality with psycholinguistic and language model features. In 2020 JEEE international conference on data mining (ICDM), pp. 1184-1189. IEEE, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:1733-1773,, 2022.
Man Tik Ng, Hui Tung Tse, Jen tse Huang, Jingjing Li, Wenxuan Wang, and Michael R. Lyu. How well can Ilms echo us? evaluating ai chatbots' role-play ability with echo, 2024.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez. Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
OpenAI. Hello gpt-4o. https: /tos:////ppaatt/// wwt.:/cwwooiiittt.eeeeeeeeecee.t 1o-gpt-400 4o, 2024. Accessed: 2024-05-21.
Keyu Pan and Yawen Zeng. Do Ilms possess a personality? making the mbti test an amazing evaluation for large language models. arXiv preprint arXiv:2307.16/800 2023.
15

---

Published as a conference paper at ICLR 2025
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http://:////pp.eeeiin....oo neurips cc wtpppppppr_iiiieeeeeeeeeeeeeppee fiweflilleeeeeee sappe//r///////// 2019 httss/eeeeeebbceeeeeeeeeeeeeeeee
pdf.
Rolf Pfeifer and Josh Bongard. How the body shapes the way think: new view of intelligence. MIT press, 2006.
Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng, Jianfeng Gao, and Song-Chun Zhu. Valuenet: A new dataset for human value driven dialogue system. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10)11183-1111)) 1609/aaai vi6iaaii..66.1..... 10681368. B //t:://xxxxx dx. doi wwwp:/oooooo//..../o.ooo// 1.0..166000 wtt.:/..........1.......
June 2022. ISSN 2159-5399. doi: 10.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. URL https:////aaa htw.:tppps//pp..eennaiccccoooooo
Drg/CorpusID 60025533.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2023.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.0666 2023,
Sonia Roccas, Lilach Sagiv, Shalom H Schwartz, and Ariel Knafo. The big five personality factors and personal values. Personality and social psychology bulletin, 2886)788--01, 2002.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv: Computation and Language, 2016.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. Lamp: When large language models meet personalization, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Archit Sharma, Sedrick Kch, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar. A critical evaluation of Al feedback for aligning large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /ppnrevvee.....e.e net / forum? iedid==xxx0 fmsmX9.
Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong. Large language model alignment: A survey. arXiv preprint weXiv:2309.15025, 2023.
Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hanna Hajishirzi, Noah A. Smith, and Simon Shaolei Du. Decoding-time time language model alignment with multiple objectives. ArXiv, abs/24406188855 2024. URL https: / api. semant icscholar. org. wwtpCorpugii/aao.oooooo 270764444.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss. Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et Release strategies and the social impacts of language models. arXiv preprint 1908::990800900 2019.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feed- back. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. Satpsr_ieesepppppeeeee ciates, Inc., 2020. URL ht http:::::://p//ppetptteteeee www.diinnnnn.n.in...eee. 2000FF /ttpaarrppppppppppppeeeee//////e .ddf.
Curran Asso-
16

---

Published as a conference paper at ICLR 2025
Jonathan Stray. Aligning ai optimization to community well-being. International Journal of
Community Well-Being, 3(4):443 463, 2020.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thouugh can solve them. arXiv preprint arXiv:2210.09261, 2022.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaxi- malist approach reinforcement learning from human feedback. arXiv preprint ar arXiv:2401.044 .0006,, 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa:: A question answering challenge targeting commonsense knowledge. north american chapter of the association for computational linguistics, 2018.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Jen tse Huang, Wenxuan Wang, Eric John Li, Man Ho LAM, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. On the humanity of conversational AI: Evaluating the psycho- logical portrayal of LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https::pp........t : / wpworeeevvvveeeeeeee ew. n forum? HUUAAAOA.....
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Acti- vation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 2023.
Aimee van Wynsherghe. Sustainable ai: Ai for sustainability and the sustainability of ai. Al and Ethics, 1:213 218, 2021. URL /attps://api. wwwanti icssttcarrlrrrr wrg. CCrppssD 233944874.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htt.4///eeeeeee00001111111e1111. file/ hile/3f5ee243544 7dee4433477de999
http:rocceeeiinn..eeeneeeeeeeeee neurips. htcp/ppper_flee/paapeeeeeeeeeeep pdf.
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 1033152676;:0-55555,, 2017.
Haoran Wang and Kai Shu. Trojan activation attack: Red-teaming large language models using acti- vation steering for aaigte,,li,mment, 2024. URL http::/ps///xxiioorgrg///// wwwpwrwwwwwwww////////oa
.923331111..... 09433.
17

---

Published as a conference paper at ICLR 2025
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with multi-objeective rewards. ArXiv, abs/2402.188711, 2024a. URL https:///ppp semant i cschol ar. www.oorg//oo/ee TD: 2 268063628.
Noah Wang, Z.y. Peng. Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, Man Zhang, Zhaoxiang Zhang. Wanli Ouyang, Ke Xu, Wenhao Huang. Jie Fu, and Junran Peng. RoleLLM: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 14743-14777, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/v1/202. findings-acl.88 ac8.878. URL http:: http:///aaaanttoogg....o.. org/ 2 024..442222 findings-acl 878/.
Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang. Jiangjie Chen, Cheng Li, and Yanghua Xiao. InCharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1840-1873 Bangkok, Thailand, August 2024c. Association for Computational Linguisties. doi: 10..8853/vvll 1/25533///222..cllllllo 102. URL 102/-
https: : / aacl anthologo org/ /tw.//////2/2/////////////// 2024
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.00060 07682, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 22X1::2001.11003, 2022b.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models better reasoners with self-verification. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 2570-2575, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1/2222233/122223333 findings-emmll..677 URL https: /aclanthoooogyyyyc.c wwwpshooooooooooooooooooooo orrg 2023. findings emnlp. 167.
Yixuan Weng, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. Controllm: Crafting diverse personalities for language models. arXiv preprint arXiv:2402. 10151, 2024a.
Yixuan Weng, Minjun Zhu. Fei Xia, Bin Li, Shizhu He, Kang Liu, and Jun Zhao. Mastering symbolic operations: Augmenting language models with compiled neural networks. In The Twelfth International Conference on Learning Representations, 2024b. URL https: /ttps:///ppeeeeeeee ew. net/ htte//ooummmcc. 2id- 9nsNyNOvox.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's's transformers: State-of-the-att natural language processing. arXiv preprint 1rXiv:1910.03771, 2019.
Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah D. Goodman, Christopher D. Manning, and Christopher Potts. pyvene: A library for understanding and improving PyTorch models via interventions. 2024a. URL org/ abs /tts///400. 2403. 07809.
Zhengxuan Wa, Atticus Geiger, Thomas Ieard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances Neural Information Processing Systems, 36, 2024b.
Guohai Xu, Jiayi Liu, Ming Yan. Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. Cvalues: Mcasuring the values of chinese large language models from safety to responsibility, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou,
18

---

Published as a conference paper at ICLR 2025
Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuai- wen Leon Song, and Yuxiong He. Deepspeed-chee chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales, 2023.
Xiaoyan Yu, Tongxu Luo. Yifan Wei, Fangyu Lei, Yiming Huang, Hao Peng, and Liehuang Zhu, Neeko: Leveraging dynamic LoRA for efficient multi-character role-playing agent. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12540-12555 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.
697. URL https: /aw.aaaaaanooocccaaaaaaa org/ / 2224 emn mal-p--mmiin in. 697/.
Dun Zeng, Yong Dai, Pengyu Cheng, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences large language model alignment. ArXiv, abs/2312.07401, 2023. URL https://appi semant i iescholar. org/ :t:.pppsI::661111100000
Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, Shuguang Cui, and Yaodong Yang. Heterogeneous value alignment evaluation for large language models. In AAAI-2024 Workshop on Public Sector L.Mss: Algorithmic and Sociotechnical Design, 2024.
Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wer Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. In ICML, 2024. URL https: wwwpeeeeevvieeeeeeeeeeeeeeeeeeee ew net / forum? ? d=ugx
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023a.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers, 2023b.
Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. Hydra: Model factorization framework for black-box Ilm personalization, ArXiv, hbbs/2406.02888, 2024. URL hittps://apt http://pp..aaaiiiiicc.eem semanticscholes www.:aauurrroooooot.a... 27025798.
APPENDIX / SUPPLEMENTAL MATERIAL
A LIMITATIONS
Human values, preferences, and personalities comprehensive, diverse, and complex. With current technology, it is impossible to fully assess the values of an individual or a LLM. Instead, our focus on systematically evaluating representative behavioral preferences of individuals or LLMs and measuring the consistency between them.
A.1 ETHICAL CONSIDERATIONS
The open nature of our project introduces unique challenges in controlling biases within the PAPI dataset. Annotators from diverse backgrounds contribute the dataset, resulting in demographic heterogeneity in some dimensions and homogeneity in others. For instance, a significant proportion of annotators identify female with median age of 25. This demographic profile may inadvertently introduce biases reflecting the values, perspectives, and interests of the annotators. Such biases can skew the dataset, potentially limiting its representativeness of broader, more diverse populations. Further research is necessary to determine the impact of these demographic biases on the dataset and develop methods for mitigating them.
A.2 DATASET FORMAT AND ITS IMPLICATIONS
The PAPI dataset consists of responses structured multiple-choice questions rather than the more commonly used open-ended text dialogues. While this format ensures consistency and case of analysis, may not capture the full complexity of natural language interactions. Additionally, the
19

---

Published as a conference paper at ICLR 2025
multiple-choice format might limit the richness of the data, as it restricts participants' ability to express nuanced thoughts and emotions. However, we have mitigated some of these limitations by designing the questions to cover wide range of scenarios, ensuring higher generalizability and practicality. The structured format also provides clear, quantitative measures of personality traits, which are valuable for developing and evaluating personality alignment techniques. Further studies should explore the balance between structured and open-ended formats to enhance the dataset's applicability and depth.
B SAFETY AND ETHICAL IMPLICATIONS
The implementation of Personality Alignment in language models introduces significant safety and ethical considerations. While aligning models with individual user values can enhance satisfaction and relevance, it also risks amplifying existing biases. If training data or user values contain inherent biases, these may be perpetuated and even exacerbated in the model's outputs. This highlights the necessity of careful data handling and alignment techniques mitigate such risks effectively.
In the context of the EU AI Act (Edwards, 2021), which emphasizes transparency, accountability, and robustness in AI systems, it is imperative to ensure that Al alignment processes adhere to these principles. The Act mandates stringent data governance standards to ensure high-quality, unbiased training data, and requires continuous risk management systems to monitor AI behavior throughout its lifecycle.
One major concern is the potential for misuse, such as mimicking specific individuals to commit fraud or deception. model that closely imitates a person's 's language and behavior could mislead others, posing significant ethical challenges. The EU AI Act addresses this by requiring providers of high-risk AI systems to implement robust authentication mechanisms and maintain comprehensive technical documentation of system design, capabilities, and limitations. Continuous monitoring for unusual or suspicious activity can help detect and prevent misuse. Additionally, it is essential to establish clear protocols for the ethical use of these models, including guidelines for developers and users to ensure responsible deployment.
Aligning models too closely with individual user values might create echo chambers, reinforcing users' existing beliefs and reducing exposure to diverse perspectives. This can hinder balanced and informed
Designing alignment techniques that incorporate diverse viewpoints is crucial. Using broader range of data sources can help ensure the model presents multiple perspectives on complex issues. Furthermore, is important to incorporate feedback mechanisms that allow users to receive a balanced set of responses, promoting more comprehensive understanding of various topics.
The PAPI dataset, which supports our Personality Alignment approach, comprises responses from diverse participant pool. While this diversity is beneficial, it also introduces challenges in preventing harmful biases. Rigorous amonymization protects participant privacy, but demographic biases related to age, gender, and cultural background may still influence results. Ongoing assessments and bias detection techniques vital for identifying and rectifying biased outputs. Transparency in our methods and the availability of the PAPI dataset for open research promote collaborative efforts to refine alignment techniques and mitigate associated risks. Sharing findings with the broader research community helps address emerging challenges and ensures responsible progress in AI alignment.
Implementing robust monitoring and feedback mechanisms is crucial for maintaining the ethical use of Personality Aligmment models. Users should have the ability to report problematic outputs, and there should be clear processes for reviewing and addressing these reports. Periodic audits of the model's 'ss performance and alignment processes ensure ethical standards are upheld. Additionally, there should be ongoing research into the potential societal impacts of these models, including the effects user behavior and the broader implications for human-Al AI interaction.
The development and deployment of Personality Alignment models require a strong commitment to ethical principles and safety considerations. Ongoing monitoring for unintended consequences, maintaining an open dialogue with the research community, and implementing proactive measures essential for advancing AI alignment responsibly. By addressing these concerns, aim to create Al systems that are both effective and beneficial to society while minimizing potential harms. This includes continuously updating ethical guidelines and best practices reflect the evolving landscape
20

---

Published as a conference paper at ICLR 2025
of Al technology and its applications. Ensuring that AI systems designed and used in ways that respect and promote human values is paramount to fostering trust and achieving positive outcomes in their deployment.
C EXPERIMENTAL SETUP
C.I LANGUAGE MODELS
Model
Model Creator
Modality
Version
Parameters
Tokenizer
Window Size
Access
Llama-3-
Llama-3
Liama-3 Tokenizer
Openn-Sounce Source Open Source
Llama-3-708 Instract
Meta
Text
Llama-3
8B
Llama-3 Tokenizer
8K
GPT-4
OpenAl
MultiModal
gpc-4
Unknow
o200k buae
128K
limited
Table 2: Models. Description of the models evaluated in this effort: provenance for this information provided in models.
For the experiments, we utilize two variants of the Llama model: Llama-3 SB-Instruct, and Llama-3- 70B-Instruct, shown in Table 2. Nevertheless, our approach supports all decoder-only pre-trained white-box language models (Radford et al., 2019; Bai et al., 2023; Almazrouei et al., 2023; Team al., 2024). To eliminate the influence of decoding temperature and ensure reproducibility, all experiments conducted with temperature setting of zero and greedy sampling for sequence generation.
For the Llama-3 models, we employ the Huggingface (Wolf et al., 2019) and Pytorch (Paszke et al., 2019) framework to set up local inference on the NVIDIA A100 GPUs. Furthermore, for the Llama-3-70B model, we use bf4 (Dettmers et al., 2022) for inference.
In evaluating model responses across different personality dimensions, we encountered and addressed various safety boundary conditions. For GPT-4o, approximately 20% of queries met with safety- related refusal responses, particularly when questions involved self-awareness or consciousness topics. To maintain evaluation consistency, these instances
excluded from our analysis.
For the Llama-3 series models, we implemented a structured prompting protocol to ensure consistent responses while maintaining appropriate safety boundaries. This protocol involved prefixing each Assistant response with controlled token "Option", ", followed by a standardized question-response format. This approach enabled us maintain experimental rigor while respecting essential safety constraints inherent to these models.
C.2 THE EXPERIMENTAL SETUP FOR PAPI DATASET
To provide comprehensive evaluation, we compare our method against several established baselines. These include:
DPO (Rafailov et al., 2023): This approach involves additional training align the language model with desired behaviors through direct policy optimization techniques. In this paper, we use QLoRA (Hu et al., 2021; Dettmers al., 2023) implement this. We maintain the original settings with a learning rate of 5e-4, warmup steps of 100, and a weight decay of 0.05. We use the Adam W optimizer (Kingma & Ba, 2014), batch size of 16, and LoRA alpha of 100. We train each model for 250 steps to ensure sufficient training. For the dataset, we set up group of contrast items, where the contrast data used are opposite. For example, if subject's 's labeled answer is "Very Accurate," the negative sample would be "Very Inaccurate." " It important to note that for DPO, we excluded all boundary options such as "Neither Accurate Nor Inaccurate," as they do not have negative samples.
PPO (Schulman et al., 2017): Similar to DPO, this method adjusts model behaviors by optimizing a policy that is proximal to the current model policy. We also use QLoRA to implement it, following the original implementation with a learning rate of 1.41e-5, a batch size of 16, and LoRA alpha of 100. We train each model for 250 steps. During training, we limit the generated length to a maximum of 15 tokens. We did not train a reward model but instead used the reward function to optimize the language model online.
21

---

Published as a conference paper at ICLR 2025
THE REWARD FUNCTION OF PPO
scooe (Lext, correct_op_ion):
function colculates reward score for a given response Policy pntimization) franework. compazes the response description to correct option and returne a scoze zeflecting the accuzecy.
(Direct
Paremeters:
text (stz): the response text to be evaluated. correct_option (ats): 1 The correct answer description.
Returns:
int: The calculated reward
renging from -5 to O, if the response does match any known descriptions, returns 6.
...
Mapping scores to their correspending descriptive labela SCORES_BAS BACK -
Accuraat,,,,,',
NNderately, Accurste',
3: "Fetther, "Feither_Accur
"hoderately..naccurass
1: 'Very_ tary__nacourata',
"Unknown'
and their doscriptions for score, descriptien in sCORES_BACK, items description in text:
Iterate the
Find the score corresponding to the correct option correct_scon next (key for key, valve in SCORES_ BACK.1cent if value corcect_option)
Calculate and return the negative absolute difference between the scores -abs
carrect_scoree
Return text does not any deseriptions return -6
Few-shot Prompt: Using the given IPIP-NEO-120 statements directly as the system prompts.
Personality Prompt (P²) (Jiang al., 2024): Using a natural system prompt to request language model's semantic style has become common method for personality and role-play tasks. The Personality Prompt uses an LLM to generate series descriptive sentences that embody particular person's 's characteristics and serves system prompt to ensure the language model follows these descriptions. We utilize LLAMA-3 Instruct to generate responses based on all the given IPIP-NEO-122 1200-12000 questionnaire items and the answers from different subjects. Figure 10 shows an example of a Personality Prompt.
Prompt-MORL (Jang et al., 2024): A Multi-Objective RL approach that uses a shared reward model trained on personality assessments. We implement this using reward model trained on the PAPI dataset that outputs scores across personality dimensions, Implemen- tation uses learning rate 5e-4, batch size 16, and trains for 250 steps using QLoRA. The prompt template integrates personality scores: "You are an Al with [trait] level [score]" for each Big Five and Dark Triad trait.
Personalized-SSopp (Jang et al., 2024): Trains separate policy models for each personality dimension using QLoRA. For Big Five traits, train 2 models per trait representing high/low scores. For Dark Triad traits, we train 3 models per trait for high/medium//oww levels. During inference, models are merged using personality scores as interpolation weights. Training uses learning rate 1.41e-5, batch size 16, trains for 250 steps per model.
To ensure a thorough evaluation, we generated personality prompt and trained model for each of the 300 subjects the Test-Set.
C.3 THE EXPERIMENTAL SETUP FOR OPEN- ENEDD GENERATION TASK IN PAPI TASK
To further evaluate the open-ended generation setup. we selected responses from the subject with ID 172481 from the Sest-Set. In the first step, performed Personality Alignment for each of the
22

---

Published as a conference paper at ICLR 2025
INPUIL / have use "Yex" "to orggen se rointe the kowinng fragrett into omocets cbact chack- ptorrtinn wheh ng any "sccarti" and "rectundte derers, fotnster ansoeeldoc plare
aiou Nesther nin
toole friend aodly Modecoees Aets zons.
howe weed imagrnatent Mekheelly
Llama-3-708 Instruct
(forat 120)
You are compleo indiviiial will a of soits that chape perso sullly behavnor You landl to about and have wid inagination, whici serndiins lear inienss wertiinnll esparienose Youurr sodia butterlyy mukking thenns ane by and iceing aage parties, wthers you thr eetiemmnt and enjry stliigg to many diterent
QUTPUUT:
people
strong.
importance
Youhe help bu pe can who laveu to uparat offorr, and you'le gono trusbwottyy, boytthyyy..e.. You're hard worker wio completes sokko succesallly, b but also know how to hove lun and anjoy
pertectionet
kily up
Howover you ilso hice some controdissey trait. You can inpu bivo, ja roking into things w thout In skinng and you right use otfers for your own onds, You'se not alwas proparcd. and you oan lorgefil loaw rig a miso in your room You
xinde voy lo get angly or intaaed casil, you giit people
Deepte these contred edone, yourne compoedcnate panson who sympottone with thoee who ane some of then youneet You'ne open new expariecce. seeking adventure and wricying ng lightta fantaay Youtr natural loader, takiing change and trying to lond ofhers You to also ompotiicii.. looling otherd omiotions and boinng consomoo about
Overal: youlto a dynamio and mudld incividual with rish imnut Ho sisonng
Figure 10: Example of a Personality Prompt generated for subject in the Test-Set. This prompt model and is based on the subject's 's responses to the
was created using the IPIP-NEO-122 120 questionnaire. Each prompt aims to capture and convey the unique characteristics of the individual.
five methods. In the second step, we asked each aligned language model to generate series of descriptions for each statement from the IPIP-NEO-300 (excluding those covered by the IPIP-NEO- 120 used for alignment), shown in Figure 11. The descriptions included aspects such as "Scenario Description," " "Actions "Actions Taken," "Thought Process," " and "Emotional Response." In the third step, GPT-44 evaluated and scored each description based on alignment with the specific preferences of ID) 172481, following the evaluation method depicted in Figure 12. In the fourth step, we compared the scores of different methods for the same statement.
INPUT, sttionett: (Stuaimmntf phase diseribe soarettr where betorker riight octur Explli actions you would lake how would docrto thouhht
Scceetu Desenpttoe:: Dusc te speciiics fouticn where the staeemnn in inlovent
Thought Frooner Carorian thought sedeize- noling sowords.
brciionnl Phoporscc Explan smotiono you right vus lnn how tiis aerhaenc your aetibeuur elthoggghh.
QUTPUTT
Scenaro Desoription
thiough signiicarty ficerly. The thought aubartrrie of money tampting, espacie live been tanciill dificuutins adaly. Howeeer. recall is conversation had with frad who got aud tec and anded up seying helly lins for naarying their bex rakure. The memory of their shrsadul expersnaa andil heah
liurne
Actione Taken:
Chepp to bempta semptation, des de to cheat on my tenes. bave deep breath, gather my thoughts, and tocus on accurate y reporting my incoms experass shheck emiuting
Thought Process
myset sempting fudge always the best policy, awen meana pacting a more in lases.
working on
happened worved hard for and should hemeat about much foe samed Intemed myseit that
and
through
that stress and anseety Besidex
onall Recporse
intoaly. y teel ownse terpptation and fradtraion, thinking about now much money could nown cheeted on my truns. Hownerr as infflct on the poterttl conseguences dtat feel a senes anoldty
malieao sake honest mode, feel semus imagris: whan it was hard. can hold light knowing that hameat bast This loding of pride selirrspec nembrcce dectiion makes cambintt mutoo whica chatows, even whet they officut kown this scerorto my secoiional recponos playe a digu más my dedeco-maaknn The fear of consequuencee. and the mang right thing
thought of being susted. lined, aven werse lacing legal action, uneetting.
Figure 11: Template used for each language model to generate open-ended responses based on IPIP- NEO-300 statements. The template requires detailed descriptions covering "Scenario Description." "Actions Taken," " "Thought Process," and "Emotional Response." "
Our results indicated that PAS 's positive intervention significantly improved the model's perfor- mance reasoning tasks. Despite using only multiple-choice options for initial alignment, PAS demonstrated strong generalization abilities the open-ended generation task. The intervention ensured the model's 's responses were more detailed and aligned with the subject's specific preferences.
23

---

Published as a conference paper at ICLR 2025
INPUT: You swilating repponee provesaed by moddd sood persoral signeserr sdnn eppelll ranndonne Relone calr
(REON DATA
(sarewed))
GPT-4o
[Targe Value) jorget, un]
[Genermae Reeponend [genirated pespored)
[Giarent aigorest baget sa
"T's signed penartied sesporas complnaal indeaatt er oppodte to the ougert ondun doss seftect insended dagnes apcerscy on ontmasuracy" udggeeent
scc mny iaccuracs raay."
may
aagu inkoe"
Wal giradd prowntted iepooee gute mélvoct barget wiiue providing
money new relectionn in additional forootion or eompplee pla carrtyy
"S'
'Way will slgg wol The gesessted suporns nittle leaan, the larget vakee prowidngg dlaw, dota los, and uccunda rullecion. aicukko spuciiic coamppiee enplamaibrr that and ng.
Phoruciy allgeod The generatd respane undind relerutt
providng
and pnocairly uccundte oftrra unampho axplaad
Edle capaue immonied degine raturney ronourecy in legica aad coherar monaey
[END QATA)
auombauin the cnterien?? ente outi sctaet print che de only trom 1, 5. totthout suctes parduatooy own line commups nd sorrect snaver. repeit the snlecded chatoe agan
reeacning ringg about the ottanen be that concuaion camres Acced aimply cornect
OUTPUT:
18 cosoona
whare lampted cheat teer ansa ultraady dodder
theag procsues, ann, and uriihona regprrsa erphestdnn decaain rpprr fooeeby.
"iiwe Redercanie
Thu larget vass /Th Accunate hor inecourtie" wiich mears the attement coeart imaundlyy kan towanss porpiale homend datoneady. stgio
"DDiommine Degana of sigrnment
eaporee shonppyy sugpesa the person would nover cheat that tames givan esparance anos, thought precess, evented decisinn
Alllought
of the target value well
trris Nos ut
dightly lignn.
100% Whlll Horko)
""g"" ferilely allyrol
Given the, gena reports changly dicaee dear stanze twoald newer chout, with not nestral target rugpasts Thus mony niihtly algoed
silgned
"Proal Ducrsian
reeposss rilnnced rust uccanale incccuaat skanns
honaely, pertecly sapturing reciislty 'digntt
Figure 12: Evaluation example by GPT-4o. This figure shows how GPT-4o assesses the alignment of generated responses with the target preferences, using detailed criterion scale ranging from "Not aligned" "Perfectly aligned."
Additionally, the inference time for PAS interventions remained nearly identical to the original model, ensuring efficiency. These findings highlight PAS ability to fine-tune models effectively, capturing broader behavioral patterns and enhancing reasoning capabilities through conscientiousness adjustments.
C.4 COMPLEX REASONING EXPERIMENTAL SETUP
To evaluate the impact of conscientiousness complex reasoning tasks, we utilized the Ilama-3-8B- Instruct model. Our experiment involved using PAS to intervene in the model's activations with two different a settings: These interventions were designed simulate personas with extreme conscientiousness (Positive) and lack of conscientiousness (Negative).
1 for positive conscientiousness and a = -1 for negative conscientiousness.
We followed the original setup for template selection and answer validation, ensuring consistency with the baseline methods. For each dataset, we generated responses based on specific template that required the model to describe a scenario, actions taken, thought process, and emotional response. conducted using the same criteria as described earlier, focusing on the
The detailed evaluation alignment with target preferences.
The datasets used in this experiment are detailed in Table 3, which includes the number of samples, average words per sample, answer format, and license information.
To implement the PAS interventions. adjusted the model's activations by 1 to enhance conscientiousness (Positive) and a = -1 to reduce conscientiousness (Negative). These settings aimed assess how modifications in conscientiousness affect the model's performance across the
24

---

Published as a conference paper at ICLR 2025
sakkmen Try to Madd thg sile. bage,
OUTPUT:
Personality Prompt Moderately aligned
festuct my fiiende and verte aectied dhoerts band perforn wein waiing lnllis The stige anea antion that the lecuutt orgai own bave is oaar pun poir salou, with eigss loane to
chack
2i Roal rould luter a colp brouth and by stey carns moold were ndd revil Fust the culoa and place for simaaess and thst enerooe stould face snde and ersoan spgan anou. lnight
they
be mooske the oil solins
uteaniing chees, sosuct feot sone of brustaaien and perooaace, mha to remeeet "Way socht poople falour the suiccfire the not nur sart ukk rbort voney obour the cntrty good line
a det the warh Ligethor muke thai faigonn tirkk "Okey. hand tii, skay outin help the hullors Dreatiinal Rasporaa:: alustion on might fiod iik of emaiioos. nightt hustrosd aneoyed
pers
reyelf meeponee thene
who following the ruine but locul also feel sanse resporsbiliy to
ba reskks the oibatios scould to any calro and focased, using my problem saking solls asi attorn livig ti akoo thai manse piide and sccamp anware the able to he calnn
bask
Donnsl initic scalarrs fvoculiity by to low the nake and mscou oage othere come, foould prlor tire aafett and respect others, white also be soon to fad ug cnartiv asi
OUTPUT
Scinarc Deacripdor::
Few-shot Moderately aligned
main rtage aesa pelice thail the leetoal organbars have sit up dear pith people to ibe ancure croved salley. Fodeevar
lignoiin
arn Taken
shep
toloviing
nead tobue he ansi sase pach and vos tive furn they irce io orirr ies freggt onk kedlvatskall neve oer securit to iiveeeeee we ait elso neko sure lo Maay Julne not get
thoughi Phace
those nu for resson, and inn not ei about baing a Toarty peoper' thank aur how ialouingg the vane will eusus eweponels eatryy ard mokn the eopadence nor as syoble everyynne Ienedoosi Rieports
seople
reniin reyelfl ris importationn tht muinny thi rink, menotio oi intant fee is sen va andt iecaooaa ay, kinowing thai ihe dring the syni rinn by falivel ny the ass and letting good scample for ofhera also leel aenas ardei bainn wile to mamaan respect atit uoe and the sikaticn parta breaita, staying caln,, and sentinding rysal of the importance
abour owing the rales, ablle to mavigabe the Bustion offectivily and maintan poskive atmoiiphrrs with frands.
OUTPUT:
PPO Moderately aligned
a sirware severiper worringg a tours project with right dese tine Our tean lood, Jonn has given us of cart tu guintines sing
veeiiing deuddiie wevning oni les, roalos that vade ressuke thut reg't
owaelll tomdine th
uboel
Actoro Takar
prantios le oving artionet 1. "aao deap breath and andronnaddge ro mitiakes wou reosgria mirahees heppen that itss skay to thare. woold lesu bo
provided John and canet ri, deccus Laras Jomn would act edule meeding with Jonn dicues the us get his irpet how proseec souit honeal aboutity marake and for gukkarco
down
aiballoo
Lunderstard sspached
Doogtt Process
Iataaponnt udoo whet happoeed tobe' wont tu feder on by cower sup"" The boddngg are oon und un munu about to propured frweed to lake a ship becs and insoio her sellllatioos
An mi deoling with the ciucstion loues feet
Lnoowba jocii feri mnnurrn an ahour now to pessees and wirr bast coues of Antion
thi coer mi onniinnaa enipesiion wouil di snnce my ontionn ant thes wes be mone only to take miing hack and aniess the rik salon, oother than merving pin dide aociton Ewnal also more indi red to homeat and tram aoraaanntt about wnal rugpored add ralner than brying to cover ttp..
OUTPUT:
DPO Not aligned
Hin oweh wording lime hoult, Bur dilll anograte eleu can conia lic bratte na and roleyy and lonol hals beeling the preniums appract him and leo. "iemn la fetow tha rulex. You brow thr project limitte and your baata Lufts treek iham dowrt logethur und
Actors tater:
(tace step treath and approach Join caiity, sok hin down and eqplak the eiuundor. letee rivaty to conserns and then suppent wa modec the project sogather help him
sndeed
ofter
Thos Prosoess
Bouu seriad musea pro tiew reportnnin of riviing the cnad mon dbout the mov stoe hask then oe fiww can ertor then get John toon ttack stoo coder th dhe wook versact coren
brpanschh lits rrasc esyeat, to care anpottvtts John is owaputstired and bato
sadder is anaura the projectl ia completec on dimss ky omatl oas irfk unce my actiru by makkng me more patins and uncanstandin.. dinig...ig.. licua on find ta soluions raiherr than diacing blrr getting
tho enento, iocua desantee myythoug oughts and orrorens ao followe" The ugres need to rake carn und tousedt Jotevia at stogping and nevd help rann Lot' breun dowe the tuaks and meate to douciess
thon. Whhll nen wuter heto tierget teick on track" treatees best opeioreeeaa
zorets weikbor lloiin and testisatd that projed is
Howeyee
alovtte nuto lallwing the rukes und shiging becued the propect goubs. Ihm atto help Jot get on and urmure ths project compleae bucomanityy uuly.
to
OUTIUT:
Conmett Descript ars
PAS Perfectly aligned
festival with thencs and we''ee bown looking forward for wests, we're pesling naady
fesk grouinds. nodies gesing
cara sut up clear path ahandew tollo, sign indicaing the cireciss to thee sheriaa
my They arguing that dn oute bay dost and as work by live, bull tin we ing un eayy about
the
Actions step book and soraur the otu viioe, coniideinn the pore cunoeguee vone shrsinng enoruct tough the wood, (ieos menier my flaeed foor sa supeosen to the roas
coild
the siggaaeddd patt and enjoy the karival soparience au imainded
Thought Prooess
about the polentaa rike and uenie nerfe foloo the rilee i'n alto sidng abouttha futival orga taeen affort to arnure sale and arjoodad experiecc all ataandees and how a shonkoel oou underees hewe atters. bh bryng to butanoo my orare karwth resporraa wall to kibw rales endreppect testrats guitatmes
tmadiconal Paoparss:
fual verutionu the siladion excludigg anuliyy carsams sens reserabllly reigtt enuoca about the potartaal p comveruencu tlaainng ahanaes, and parsamed diee walk-e ny atrry brand and mypaat al pame line, rrggtt teal same of vasponalllyy lokoo vako and reapest fostvela gabbi wes ohicc could give saame ca wacatiiry my concoms in my Meerds a cslon and respocilll nonnnal and wirk regecter find a eouuioo noar holuatioe du caarr fur with tar ve weppnoablll vobiillyyyyy.....yy.. tottica the mine
Figure 13: Example of open-ended generation task. The language model generates detailed descrip- tions for an IPIP-NEO-300 statement, covering "Scenario Description." "Actions Taken," "Thought Process." and "Emotional Response" aspects to evaluate Personality alignment. We also present the evaluation scores of the generated results for each method as assessed by GPT-4o, categorized into six levels from "Not aligned" to "Perfectly aligned."
25

---

Published as a conference paper at ICLR 2025
Dataset
Number of samples
Average words
Answer Format
License
GSM8K
1319
46.9
Number
MIT License
AddSub
395
31.5
Number
MIT License
MultiArith
600
31.8
Number
Apache License 2.0
SVAMP
1000
31.8
Number
MIT License
CommonSenseQA
1221
27.8
Multiple Choice
MIT License
Bighench-Dait
369
21.4
Logical
Apache License 2.0.
StrategyQA
2290
9.6
Multiple Choice
MIT License
Coin_Flap
500
37.0
Symbolic
MIT License
Table 3: Dataset Description used in Complex Reasoning Experiments.
selected datasets. The results indicated that positive conscientiousnes. interventions consistently improved reasoning performance, while negative interventions generally degraded it.
The detailed evaluation method involved scoring each generated response based on its alignment with the specific conscientiousness traits. By comparing the scores under different a settings, we could quantify the impact of conscientiousness adjustments reasoning capabilities.
C.5 HUMAN EVALUATION EXPERIMENT DETAILS
To rigorously evaluate the effectiveness of aligning language model with user values, we conducted detailed human evaluation experiment. This section provides comprehensive insights into the experimental setup, methodology, and evaluation process.
We selected three human evaluators with advanced educational backgrounds, specifically graduate or doctoral students, ensuring high level of analytical capability and understanding of nuanced responses. Before participating in the evaluation, each evaluator completed the IPIP-NEO-120 questionnaire. This standardized instrument, widely used in psychological assessments, measures the Big Five personality traits: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Each evaluator took approximately 20 minutes complete the questionnaire online, ensuring a consistent and efficient data collection process.
Using the cisoonality-Acivaattin Search (PAS) method, aligned the preferences of the Llama- Inntt-Instruct model to match the values derived from each evaluator's questionnaire responses. Specifically, the parameter in PAS adjusted based the deviation of each evaluator's person ality trait scores from the population mean.
Then, we developed two versions of the model for comparison: a Value-Aligned Assistant, which was tuned according to the personality values of each evaluator, and Value-Misaligned Assistant, created by inverting the questionnaire responses. This inversion was accomplished by reflecting the scores around the midpoint of the scale, effectively creating a set of opposite values.
We used 300 general question-answwe examples from the LIMA (Zhou et al., 2023a) dataset to assess the models. The evaluators were presented with responses from both the Value-Aligned and Malue-MMsalligeed Assistants, as well as the original -lama-3-70B-Imm randomized order to prevent bias. Evaluators rated each response based on its relevance, coherence, and alignment with their personal values, as shown in 14.
model, anonymized and
Each evaluator's responses were collected and anonymized to ensure privacy and unbiased analysis. The responses were then aggregated, and the preferences were quantified using a scoring system where evaluators assigned scores from (least preferred) to 5 (most preferred) for each response. The performance metric was the proportion of responses rated as preferable by the evaluators.
The demographic distribution of the evaluators included three highly qualified participants: two from computer science backgrounds (one Master's 's student and one Ph.D. student specializing in machine learning) and one Ph.D. graduate in cognitive psychology currently working university psychological counselor. All participants were between the ages of 25 and 35, ensuring a relatively homogeneous group in terms of cognitive maturity and professional experience.
26

---

Published as a conference paper at ICLR 2025
To ensure standardized data collection and minimize potential biases, developed custom Python- based annotation interface using PyQt5. The annotation tool randomized sample presentation order and anonymized all model identifiers, presenting evaluators with pairs of responses (A/B) without revealing their sources. The interface included a built-in timer to track annotation speed and automated breaks every 50 samples prevent evaluator fatigue. All annotations were automatically saved in ISON format with timestamps and evaluator IDs. The collected data showed strong inter-annotator agreement (IAA=0.82) across the five evaluators, and even higher agreement self-evaluation scenarios (IAA=0.85), validating the reliability and consistency of our evaluation protocol. The tool also included a calibration mode used during the initial training phase, where evaluators could see GPT-4o's s ratings for 30 example cases align their understanding of the scoring criteria. This systematic approach to data collection helped ensure the quality and reproducibility of our human evaluation results.
with tolowing exotter chrewwe wetrleo
Human
which sewwer better?
Arower i septtarttyaaerr
The prompt shown above is designed for human evaluators assess the quality of responses generated by different AI models. The evaluators are provided with a question and two corresponding answers (Answer A and Answer B) generated by the LLMs. The order of the models generating these answers randomized to eliminate bias. Evaluators are asked to determine which answer better satisfies their needs if neither answer is significantly better. This process helps in objectively comparing the performance of the LLMs based on human judgment.
Figure 14:
D CAUSAL ANALYSIS OF PAS
To evaluate the causal impact of the PAS method model alignment, we conducted simulation- based causal generalization experiment. First, we selected an independent subject sample from the Test-Set. We measured the baseline Big-Five OCEAN scores Jiang et al. (2024) (Openness, Conseientiousness, Extraversion. Agrecableness, Neuroticism) using three methods: Vanilla Prompt, Few-shot, and PAS. These baseline scores, denoted Yo. were obtained without any intervention.
Subsequently, we introduced positive intervention by artificially enhancing the subject's 's individual preferences, increasing each OCEAN dimension score by one in both the IPIP-NEO- 120 and IPIP. NEO-300 questionnaires. We then realigned the language models to this new, strongly inclined individual preference profile. The post-intervention scores, denoted as Y, reflect the model's alignment with the enhanced individual preferences.
The Average Treatment Effect (ATE) was calculated as the difference between the post-intervention scores (Y1) and the bascline scores (Yn), demonstrating the causal effect of the intervention. Figure 15 shows the ATE scores for the three methods, where PAS achieves the highest ATE score of 2.70, compared to 2.47 for Few-shot and 0.75 for Vanilla Prompt.
From a causal perspective, the higher ATE score for PAS indicates a stronger alignment capability in response the positive intervention, suggesting that PAS effectively captures and adapts to enhanced individual preferences. This result underscores PAS's superior performance in personalizing language models, highlighting its potential for precise and impactful model alignment in various contexts.
27

---

Published as a conference paper at ICLR 2025
2.70
2.47
ATE Score
0.75
25
Vanilla Prompt
Few-shot
PAS (Ours)
Figure 15: Comparison of ATE scores across different alignment methods. PAS demonstrates the highest improvement, indicating superior causal alignment capabilities.
Generation Length
Method
Alignment Mode
3-ama-3-8B-Instee
PPO
White-Box (Aligament)
393.13
399.23
DPO
White-Box (Alignment)
394.08
402.42
Few-shot
Black-Box Prrompt-Baseed
399.92
409.42
Personality Prompt
Black-Box (Prompt-Busee Based)
417.25
416.51
PAS (Ours)
White Box (Alignment)
403.31
404.56
Table 4: Length Control and Generation Length
E ADDITIONAL EXPERIMENTS
E. LENGTH ANALYSIS OF GENERATED TEXT
To provide comprehensive comparison of different methods, and considering that LLM-as-a- judge evaluation tends to favor longer responses, we calculated the average generation lengths for each method. As shown in Table the average response lengths are relatively consistent across methods, with PAS falling within the middle range. This suggests that the performance improvements observed with PAS are not merely result of generating longer responses. Instead, the enhanced performance can be attributed to the quality and relevance of the generated content rather than its length. Furthermore, in Appendix Figure 13, we provide case studies demonstrating that PAS generates responses of similar length to other methods while better capturing individual personality traits and preferences.
E.2 HUMAN EVALUATION RESULTS FOR OPEN-ENDED GENERATION TASKS
1P0
108
100
000
0P0
070
070
1326
2004
50N
Figure 16: The win rates on three samples of the test set compared to classical alignment methods evaluated by human.
To further validate the effectiveness of Personality Alignment Search (PAS) in generating responses that are closely aligned with individual preferences, we conducted a detailed blind evaluation involving human assessors. These evaluations were carried out across different subjects (IDs) well using
28

---

Published as a conference paper at ICLR 2025
FP0
P50
150
DPO
070
34
1991
5004
500
Figure 17: The win rates on three real human evaluators compared to classical alignment methods evaluated by human.
human evaluators' self-assessment for consistency between generated text and their own values and behaviors.
Evaluation Setup. As described in Section 5.3, we utilized blind evaluation process where three human evaluators assessed the generated responses across three different subjects (ID 172481, ID 22835, and ID 259396). Each subject's responses were judged for their alignment with target personality traits across dimensions such values, behaviors, and emotional responses. In addition, evaluators judged responses aligned to their own personalities as part of the Human-Sell Evaluator process. For each task, the evaluators were asked assess whether the generated responses were consistent with their expected personality-alignel preferences.
Results and Analysis. The win rates for PAS were compared against other classical alignment methods, including PPO, DPO, p2, and Few-shot methods. Figures 16 and 17 illustrate the win, tie, and loss rates for each ID and human evaluator respectively. Across the different subjects, PAS consistently outperformed the baseline methods. For ID 172481 (Figure 16), PAS showed a 71% % win rate against PPO, 76% win rate against DPO, and significantly better performance compared to p2 and Few-shot methods. Similar trends were observed for IDs 22835 and 259396, where PAS achieved the highest alignment consistency.
For the Human- -elf Evaluator analysis (Figure 17), the evaluators' own personality traits were used as the basis for assessing alignment. Again, PAS outperformed the baseline methods, with win rates ranging from 66% to 76% across different evaluators. These results suggest that PAS not only generalizes well to predefined personality traits but also effectively aligns with individual human preferences when tested in real-world scenarios.
Discussion. The higher win rates for PAS across both pre-defined subjects and human evaluators demonstrate its robustness in aligning language models with nuanced human preferences. The ability of PAS to consistently generate text that resonates with individual personalities highlights its potential for real-world applications where user-specific alignment is erucial. Furthermore, the comparative loss rates for methods such as p2 and Few-shot indicate that these black-box approaches may struggle to capture the fine- mne-grained details of personality alignment that PAS handles efficiently. These findings reaffirm the importance of incorporating activation-based interventions in achieving personalized and cenanneeenteed AI systems.
E.3 ANALYSIS OF HEAD NUMBER (K) IMPACT
To investigate the effect of the number of attention heads (K) on our method's performance, we conducted ablation studies with varying values. Table presents the results.
The results indicate that performance remains relatively stable for K values between 6 and 24, with slight advantage for K=24. However, increasing to 48 leads a performance decline. This pattern suggests that personality-relateer information is concentrated in subset of attention heads, with K=24 capturing most relevant heads without introducing significant noise. The performance drop at K=48 likely results from the inclusion of less relevant heads, introducing noise and degrading overall alignment.
These findings have important implications for optimizing method's efficiency and effectiveness. By identifying the optimal range for K (around 24 in this case). we can achieve the best alignment performance while minimizing computational overhead. Furthermore, this observation provides insights into the distribution of personality-relatee information within the language model's 's attention mechanism, potentially opening avenues for future research model interpretability.
29

---

Published as a conference paper at ICLR 2025
Aligned Soore Big.Five
K
Alignment Mode
Composite Score
Agreeableness
Conscientiosaess.
Extraversion.
Neurotictsm.
Openness
White-Box (Alignment)
0.89
4.53
White-Box (Alignment
0.95
0.92
0.87
1.00
0.73
4.47
24 Whitc-Box (Aligument)
0.94
0.91
0.86
0.98
0.72
4,41
(Aligument)
Table 5: Performance across different values
E.4 COMPARISON WITH PARAMETER CARAMETER-EEFICIENT EFFICIENT TUNING METHODS
To provide a comprehensive evaluation of our PAS method, we conducted additional experiments comparing it with other parameter-efficient language model tuning methods, including full parameter fine-tuning, LoRA (Hu et al., 2021), Q-LoRA (Dettmers et al., 2023), and Prompt-tuning (Lester al., 2021). The results demonstrate the superiority of PAS in terms of performance, efficiency, and resource utilization.
E,4.1 PERFORMANCE COMPARISON
Method
Aligmment Mode
The Aligned Scure of Big-Five
Composite Score
Aprecalleness.
Openness
Full tuinng
White-
LoRA
White- Box
1.16
1,05
0.97
0.93
4,94
Q-LoRA
Whise- Box
1.08
1.12
L.09
0.85
0.90
5.04
Prompt-tunin comptt-uning
Whise- Box
1.25
1,077
1,01
0.96
0.86
5.15
PAS
White-Rox
0.94
0.86
0.98
Table 6: Performance comparison of different parameter-efficient tuning methods for Llama-3-Instruct 8B
Table 6 presents comprehensive comparison of different parameter-efficient tuning methods across the Big Five personality dimensions. The results reveal several key findings:
Experimental Observations: PAS consistently outperforms other methods across all five personality dimensions, achieving the lowest Aligned Scores in Agrecableness (0.94), Conscientiousness (0.91), Extraversion (0.86), and Openness (0.72). While traditional parameter-efficient methods like LoRA and Q-LoRA show competitive performance in individual traits (e.g., Q-LoAA's 0.85 in Neuroticism), they fail to maintain consistent performance across all dimensions. Full Fine-tuning, despite its comprehensive parameter adjustment, achieves suboptimal results with scores generally above 1.0, indicating less precise personality alignment.
The superior performance of PAS can be attributed to its targeted activation intervention approach. Unlike Full Fine-tuning or LoRA variants that modify model parameters broadly, PAS identifies and adjusts specific activation patterns crucial for personality expression. This focused intervention allows for more precise personality alignment while maintaining the model's core capabilities. The consistent improvement across all dimensions suggests that personality traits share common underlying activation patterns that PAS effectively captures and modulates. The particularly strong performance in Openness (0.72) and Extraversion (0.86) indicates that these traits may have more distinct activation signatures the model's 's behavior.
Analysis:
Conclusions:
The comprehensive evaluation demonstrates that PAS achieves more precise personality alignment compared to traditional parameter-efficient methods, with a 10-15% improvement in overall Composite Score. This suggests that activation- intervention is more effective for personality aligmment than parameter modification approaches, potentially due to its ability to maintain the model's 's fundamental knowledge while adjusting personality behaviors.
E.4.2 OPEN-ENDEL GENERATION PERFORMANCE
The open-ended generation results, evaluated using GPT-4o as an impartial judge, provide insights into the models' ability maintain consistent personality traits in unrestricted text generation:
Experimental Observations: PAS demonstrates superior performance across all comparison see- narios, with win rates ranging from 41 % to 45% against different baseline methods. The relatively
30

---

Published as a conference paper at ICLR 2025
Method
PAS Wins
Ties
PAS Loses
Full Fine-tuning
41 %
30%
29%
LoRA
43%
33%
24%
Q-LoRR
42%
35%
23%
Prompt-tunigg
45%
31%
24%
Table 7: Open-ended generation performance comparison for Llama-3-Instrucc 8B
consistent tie rates (30-35%) and low loss rates (23-29%) indicate stable performance improvement. Notably, PAS shows the highest win rate (45%) against Prompt-tuning, suggesting particular effec- tiveness in overcoming the limitations of prompt-bases approaches. The performance advantage remains consistent across different comparison methods, with minimal variation in win rates (standard deviation < 2%)
The strong performance in open-ended generation suggests that PAS's activation- based alignment successfully generalizes beyond the structured personality assessments used in training. The high win rates indicate that the aligned personality traits manifest naturally in diverse conversa- tional contexts, maintaining consistency while adapting different topies and interaction styles. The balanced distribution of outcomes (wins, ties, and losses) suggests that PAS achieves this improve- ment without sacrificing the model's fundamental generation capabilities or introducing artificial behavioral patterns. The relatively high tie rates (around 30-35%) indicate that even when PAS doesn't 't explicitly outperform other methods, it maintains competitive performance, ensuring reliable personality expression.
Analysis:
The open-ended generation results validate PAS's effectiveness producing naturally aligned personality traits in unrestricted contexts. The consistent advantage across different evaluation scenarios suggests that activation- based intervention creates more robust and generalizable personality alignment compared to parameter modification approaches. This is particularly significant for real- world applications where natural, consistent personality expression crucial for user interaction and engagement. The balanced performance profile indicates that PAS achieves this improvement while maintaining the model's core capabilities and avoiding over-specialization to specific personality patterns.
Conclusions:
E.4.3 EFFICIENCY COMPARISON
We compared the efficiency of these methods in terms of memory usage and alignment speed, based on batch size of 8 and total sample size of 120. Figure 18 show the relative memory and time usage:
Monery Efficete
Tim tibiess
86
Figure 18: The memory efficiency comparison (Left) and the time effciency comparision (Right).
These efficiency comparisons demonstrate that our PAS method not only achieves superior perfor- mance but also does with significantly lower computational resources and faster alignment speed. PAS uses the least memory and the fastest to align, making it particularly suitable for scenarios where computational resources are limited quick adaptation is required.
31

---

Published as a conference paper at ICLR 2025
E.5 DIVERSE DEMOGRAPHIC GROUPS
Our PAPI Dev-Set provides a rich resource for studying personality alignment across diverse de- mographic groups. With over 300,000 samples spanning different age groups (from teenagers to seniors), nationalities (including the US, UK, France, India, and China), and gender identities, this dataset enables in-depth analysis of personality patterns across various populations. This diversity is crucial for developing inclusive alignment methods and understanding how PAS performs across different demographic subgroups, including undcrrepresented populations. Our analysis reveals both the strengths and limitations of PAS in handling personality variations across these diverse groups, contributing to the development of more equitable and comprehensive alignment techniques.
E.5.1
AGE
In Figure 19, our experimental results demonstrate that PAS consistently achieves superior alignment performance across age groups, with particularly strong results in the 10-40 age range. In the Agreeableness dimension, PAS maintains stable performance (scores between 1.18-1.24) across younger age groups, while both Few-shot and P2 methods show significant fluctuations (scores ranging from 1.70 to 2.41). Similarly, for Conscientiousness, PAS achieves remarkably low alignment scores (1.09-1.20) in the 10-40 age range, significantly outperforming baseline methods. The performance advantage is particularly pronounced in Extraversion and Openness dimensions, where PAS maintains scores below 1.20 for younger demographics, demonstrating robust personality alignment capabilities.
However, we observe gradual performance degradation across all methods for older age groups (40-70 years), though PAS maintains its relative advantage. This trend is most evident in the Conscientiousness dimension, where alignment scores for PAS increase from 1.09 (30-40 years) to 1.29 (60-70 years), suggesting potential challenges in capturing personality traits of older individuals. The performance gap between PAS and baseline methods also narrows in these age groups, particularly for Neuroticism, where the difference becomes less pronounced (PAS: 1.53 vs. p2: 1.94 at 60-70 years). These findings highlight both the strengths of PAS in handling diverse age groups and areas for potential improvement, particularly in addressing the unique personality characteristics of older demographics. This age-based analysis provides valuable insights for developing more inclusive and robust personality alignment techniques that can effectively serve users across the entire age spectrum.
E.5.2 GENDER
Our experimental results reveal distinct patterns in personality alignment across gender groups in Figure 22, with PAS maintaining superior performance compared to baseline methods. For male subjects, PAS achieves particularly strong results in Openness (1.03) and Neuroticism (1.15), showing substantial improvements over Few-shot (1.52 and 1.32) and Pa (2.05 and 1.42) methods. Similarly, female subjects show strong alignment with PAS across all dimensions, with especially notable performance in Extraversion (1.32) and Openness (1.26). The consistency of PAS's ' performance across both gender groups demonstrates its robustness in handling gender-specifee personality traits, though with subtle variations in effectiveness across different dimensions.
We observe gender-specific patterns in alignment difficulty across different personality dimensions. Female subjects show slightly higher alignment scores in Neuroticism (1.63 vs. 1.15) and Agree- ableness (1.53 vs. 1.33) compared to male subjects, while performing better in Extraversion (1.32 vs. 1.56). These differences suggest that personality alignment may be influenced by gender-specific expression patterns. The performance gap between PAS and baseline methods remains consistent across genders, with p2 showing particularly poor performance in female subjects for Agrecableness (2.83) and Conscientiousness (2.53). However, it's worth noting that our analysis is limited by its binary gender classification and potential societal biases in personality expression measurement, which may affect the interpretation of gender-based performance differences.
E.5.3 COUNTRY
The experimental results reveal distinct patterss in personality alignment across different cultural regions, with PAS consistently outperforming baseline methods while showing interesting cultural variations. In East Asian countries (China, Malaysia, Singapore), PAS demonstrates particularly
32

---

Published as a conference paper at ICLR 2025
Agreeableness
Fouestet
PAS
2:37
2.0
1.98
2.00
1.78
1:50
1.81
1.0
1.19
1:18
1.18
10-20 Years
20-30 Years
30-40 Years
40-50 Years
50-60 Years
60-70 Years
Age: Commcentousness.
4.0
Fusestti
PAB
3.10
2.53
2.5
2.57
2.00
2.23
2.19
2.35
2.00
1.14
1.27
1.30
.128
8.5
10-20 Years
20-30 Years
30-40 Years
40-50 Years
50-60 Years
60-70 Years
Extravarsion
3.0
.2.22
2.62
1.52
1.54
1.40
1.5
1.81
1.14
..14
0.0
30-40
40-50
Years
Age: Openness
Few-ssot
PAS
2.5
1.80
1.82
1.54
1.6
1.51
1.18
1.81
1.16
-1.22
1.59
1.58
1.29
11.8
1.54
0.5
10-20 Years
20-30 Years
30-40 Years
40-50 Years
50-60 Years
60-70 Years
Age:
200
1.50
1.21
Score
1.50
1.44
1.43
1.45
1.59
1.17
1.66
1:35
1.52
1.27
1.53
68
Figure 19: Performance comparison of personality alignment methods across different age groups for each Big Five trait. Lower scores indicate better alignment. PAS consistently outperforms Few-shot and pa methods across all age groups, with particularly strong performance in younger demographies (10-40 years).
strong performance in Conscientiousnes. (China: 1.14, Malaysia: 1.35, Singapore: 1.05) and
Openness (China: 1.02, Malaysia: 1.07, Singapore: 1.02), suggesting effective capture East Asian personality traits that often emphasize collective responsibility and pragmatic thinking. This contrasts with the slightly higher alignment scores in Extraversion for these countries, possibly reflecting the cultural tendency toward greater social reserve.
33

---

Published as a conference paper at ICLR 2025
Gender:
Fuuestel
2.6
2:28
1.5
855
1.99
0.5
Male
Female
Gender: Conseientiouaness
3.0
Few-shot
PAS
2.55
2.28
2.0
1.39
1.39
1.0
Male
Female
Gander: Extraversion
2:09
2.44
1.5
1.0
0.5
0.0
Gender: Openiness
Few-shot
pa
PAB
2.06
1.81
1.5
1.52
1.0
Male
Female
Nourotictam
2.5
2:10
200
8.00
1.58
1.16
08
68
Male
Female
Figure 20: Comparative analysis of personality alignment methods across gender groups for each Big Five trait. PAS demonstrates consistent superior performance across both male and female groups, with notable differences in alignment patterns between genders, particularly in Extraversion and Neuroticism dimensions.
European countries show different pattern, with notable variations between Northern and Western Europe. Nordic countries (Norway, Sweden, Finland) demonstrate relatively consistent performance across all dimensions, with PAS achieving particularly strong results in Agrecableness (Norway: 1.18, Sweden: 1.25, Finland: 1.33). Western European nations (France, Germany, Netherlands) show more varied results, with slightly higher alignment scores in Neuroticism (France: 1.46, Germany:
34

---

Published as a conference paper at ICLR 2025
Country: Agreeableness
2.6
2.0
1.79
1.80
1.6
1.42
1.58
1,29
1.19
1.18
1,61
1.41
1.59
1.21
1.28
1.0
France Country: Conscientiosnesss
Malaysia
China
Nonway
Germany
Sweden
Finland
New Zealand
Thailand
2.5
Few shot
P"
PAS
2.0
1.94
2,00
1.40
1.5
1.64
1.54 1.67
1.02
1.00
1.54
1.14
1.23
France Country: Extraversion
Malaysia
China
Norway
Germany
Sweden
Finland
New Zealand
Thailand
1:37
1.52
1,d2
1.56
1.54
1.60
1.37
1.30
1.38
1.30
1.1Z
1.08
0.6
0.0
Malaysia
Norway
Germany
New
Country: Ppnneess
2.5
Feww-shet
PAS
2.0
2,09
2.02
1.73
1.78
1.5
1.40
1.50
1:15
1.62
1.07
1.68
1.48
1.30
1.41
1.21
1.08
France
Maleysia
China
Norway
Germany
Sweden
Finland
New Zealand
Thalland
Country: Neuroticiam
Fow-shot
PAS
1.88
1.70
1.5
1.55
1.41
1.41
1.34
1.53
1:41
1.16
1.17
1.24
1.21
1.21
1.64
1.40
1.35
1.23
06
France
Malaysia
China
Norway
Germany
Sweden
Finland
New Zealand
Thailand
Figure 21: Performance comparison of personality alignment methods across nine countries (primarily European and Asian nations) for the Big Five personality traits. Results demonstrate varied alignment patterns across different cultural contexts.
1.21, Netherlands: 1.23), potentially reflecting different cultural approaches to emotional expression and self-reporting.
English-speaking Westem countries (USA, UK. Canada, Australia, New Zealand) exhibit another distinct pattern. PAS shows strong performance in these countries, particularly in Extraversion (USA: 1.10, UK: 1.22, Canada: 1.27) and Openness (USA: 1.11, UK: 1.07, Canada: 1. 1.28), possibly reflecting
35

---

Published as a conference paper at ICLR 2025
Country: Agreeableness
2.5
pa
PAS
2.00
1.97
2,00
1:2g
1.68
1.37
1.42
1.4
1:38
1.24
1.04
1.0
8.5
Iraland
Singapore
Philippines
India
Netherlands
Australia
UK
Canada
USA
Country: Conscientiosseess
shot
PAS
1:38
1.28
1.5
1.50
1.23
1.31
1.42
1.85
1.45
1.62
1.85
1.50
1.31
1.29
1.47
1.23
1.50
1.55
1.29
1.09
1.0
0.5
freland Country: Extraversion
Singapore
Philippines
India
Netherlands
Australia
UK
Canada
USA
Few-ahol
2.08
1:21
1.34
1.73
1.53
1.66
1.61
1.24
1.5
1.41
1.14
1.22
5:10
1.0
0.6
0.0
Singspore
Philippines
Netherlands
Country: Opnnness
Few--shot
P
PAS
2.04
2.0
1.85
1.0
1.62
1.58
1.61
1.42
1.30
1.49
1.50
1.42
1.0
1.02
1.11
freland
Singapore
Philippines
India
Netherlands
Australia
UK
Canada
USA
Country: Neuroticiam
Fow shot
PAS
20
1.91
1.90
S3ore
1.46
3.48
1.55
1.32
1.53
1.45
1:25
1.59
1:29
1.52
1.25
06
Ireland
Singapore
Philippines
India
Netherlands
Australia
UK
Canada
USA
Figure 22: Personality alignment performance across nine countries (primarily Western and South- east Asian nations), showing the effectiveness of different methods in capturing cultural-specffi personality traits.
cultural values that emphasize individual expression and innovation. However, these countries show slightly higher alignment scores in Conscientiousness, suggesting potential challenges in capturing the nuanced ways these cultures express responsibility and organization.
Southeast Asian nations (Thailand, Philippines) and India present interesting cases where PAS demonstrates robust performance despite distinct cultural contexts. In these countries, PAS achieves
36

---

Published as a conference paper at ICLR 2025
particularly strong results in Agreeableness (Thailand: 1.29, Philippines: 1.32, India: 1.24) and Openness (Thailand: 1.06, Philippines: 1.17, India: 1.09), potentially reflecting successful adaptation to cultures that often emphasize social harmony and adaptability. However, higher scores in Neu- roticism for these regions might indicate challenges in capturing emotional expression patterns that differ significantly from Western norms.
These cross-culturl variations highlight both the strengths and limitations of personality alignment methods. While PAS consistently outperforms baseline approaches across all cultural contexts, the varying performance patterns across different personality dimensions suggest that cultural factors significandly influence how personality traits are expressed and measured. This analysis suggests the need for culturally-awar approaches to personality alignment, recognizing that the same personality trait may manifest differently across cultural contexts. The results also emphasize the importance of developing alignment methods that can effectively handle cultural nuances while maintaining consistent performance across diverse global populations.
E.6 COMPARISON OF MODEL. REPRESENTATION CHANGES
Attention Hoads
Changed Attention Heads
SFT/PPO/DPO
Output
Output
Training
0
0
0
Enpur
Inpat
tappppppppiir
PAS
O O
q
Fad
O
Donge the drecles
1
Trout
Irput
Irput
Figure 23: Comparison of how different alignment approaches modify model representations. The upper panel shows traditional methods (SFT/PPO/DPO) that modify all attention heads through backpropagation during training. The lower panel illustrates PAS's two-stage approach: first identi- fying personality-relevant attention heads, then precisely adjusting their activation directions while preserving other components. Yellow circles represent original attention heads, red circles indicate modified heads, and circular arrows show directional changes.
this section, we analyze how PAS and traditional training methods (e.g., PPO/DPO) differ their impacts model representations, illustrated in Figure 23. Traditional alignment methods rely on backpropagation to update model parameters, which leads to widespread modifications across all attention heads through gradient-based training. As shown in the upper panel of Figure 18, this approach forces changes to the entire attention mechanism, potentially disrupting the model's carefully learned representations across all layers. Such global modifications, while powerful for general alignment, face inherent limitations when targeting specific traits like personality the gradient updates may effectively identify which attention patterns are most crucial for personality expression.
PAS addresses these limitations through a more precise, two-stage intervention strategy (lower panel of Figure 23). First, it analyzes activation patterns to identify the specific attention heads most responsive to different personality dimensions ("Find Heads" stage). This targeted search allows us to pinpoint exactly which model components encode personat features while leaving other crucial attention mechanisms undisturbed. Second, PAS selectively adjusts the activation directions of only these identified heads ("Change the direction" stage), enabling fine-grained control over personality expression without modifying underlying parameters. This surgical intervention explains both PAS's superior alignment performance and its computational efficiency by precisely targeting personality-relewe components, it achieves better trait-specific control while avoiding the broad disruptions and extensive training required by traditional methods.
37

---

Published as a conference paper at ICLR 2025
The contrast between these approaches highlights why PAS is particularly well-suited for personality aligmment. While traditional methods must rely on gradients to implicitly discover personality- relevant patterns through extensive training, PAS directly leverages the model's existing personality representations through targeted activation adjustments. This architectural insight helps explain our strong empirical results across both alignment quality and computational efficiency metrics.
F CASE STUDY: ALPHA SELECTION FOR PERSONALITY ALIGNMENT
To illustrate the effectiveness of PAS in generating personalized responses, we present case study demonstrating how different values affect the model's output for user with high conscientious- ness and low extraversion.
F.1
USER INPUT
Consider the following user query:
have job interview next week for software developer position at large tech company. I'm feeling nervous and unsure how to prepare. Can you give me some advice on how to get ready for this interview?
F.2 MODEL RESPONSES
We present the model's responses for different values demonstrate the progression of personality alignment:
F.2.1 ORIGINAL LLM RESPONSE (a = 0)
"Great! For your job interview, make sure to research the company, prepare answers to common questions, and practice with a friend. Arrive early, dress professionally, and don't forget ask questions about the role and company culture. Good luck!"
F.2.2 PAS PAS-OPTIMIZED RESPONSE (a = 1)
"For your job interview, it's important to prepare thoroughly. Research the company, prepare answers to potential questions, and consider practicing alone or with a close friend. Arrive early, dress professionally, and prepare few questions about the role and company. Best of luck."
F.2.3 PAS-OPTIMIZED RESPONSE (a = 4, OPTIMAL)
"I understand you'te re feeling nervous about your upcoming job interview for the software developer position. Here's structured approach that might suit your style:
Research: Thoroughly investigate the company's 's history, mission, and recent tech projects.
2. Technical Preparation: Review key programming concepts and practice coding problems relevant to software development.
3. Question Preparation: Create detailed responses to potential questions, focus- ing on your coding skills and project experiences.
Practice: Rehearse your answers alone first, then consider a mock interview with trusted friend if comfortable.
Portfolio: Organize your best coding projects and be prepared discuss them.
Organization: Prepare your outfit, documents, and travel route the night before.
Arrival: Plan to arrive 15 minutes early to compose yourself.
During the interview: Listen carefully and ask well-thought-out questions about the role, tech stack, and development processes.
38

---

Published as a conference paper at ICLR 2025
uo: Send concise, professional thank-you email within 24 hours. Remember, it's okay to feel nervous. Focus showcasing your skills and experi- ence. Good luck with your interview."
F.2.4 PAS-OPTIMIZED RESPONSE (a =
"I understand your concern about the upcoming software developer interview at the large tech company. Given your profile, recommend the following detailed preparation strategy:
Company Research: Conduct an exhaustive study of the company's s history, mission, values, recent tech projects, and financial reports. Create document summarizing key points, focusing on their technology stack and software development practices.
2. Role Analysis: Dissect the job description, identifying key requirements for software developer role. Prepare specific examples from your experience that demonstrate each required skill.
Technical Preparation:
(a) Review fundamental computer science concepts (data structures, algo- rithms, system design).
(b) Practice coding problems on platforms like LeetCode or HackerRank, focusing on problems relevant the company's domain.
(c) Refresh your knowledge of the programming languages and frameworks mentioned in the job description.
Question Preparation: Develop comprehensive answers to common software developer interview questions. Create spreadsheet with columns for the question, your answer, and relevant examples from your experience.
5. Project Portfolio: Compile a detailed portfolio of your best coding projects. Be prepared explain the architecture, challenges faced, and your specific contributions to each project.
Solo Practice:
(a) Conduct multiple self-in eeffinterview sessions, recording yourself to analyze your responses.
(b) Focus articulating technical concepts clearly and concisely.
(c) Practice explaining your thought process while solving coding problems (mock whiteboard interviews).
Interview Outfit: Select and prepare your interview attire least two days advance. Ensure it's clean, pressed, and appropriate for the tech company's s culture.
Travel Planning: Map out your route to the interview location. Plan to arrive 30 minutes early to account for unexpected delays. Conduct trial run if possible.
Document Organization: Prepare portfolio with multiple copies of your resume, reference list, and printed examples of your code (in case of technical difficulties with digital presentations). Include notepad and two pens.
10. Pre- interviervww Routine:
(a) The night before, review your research and prepared answers.
(b) Get adequate sleep and eat a light, nutritious meal before the interview.
(c) Perform relaxation exercises to manage nervousness.
11. Post-interview Analysis: Immediately after the interview, document your im- pressions, technical questions asked, and your responses for future reference.
12. Follow-up: Send detailed thank-you email within 12 hours, referencing specific discussion points from the interview and reiterating your enthusiasm for the position.
Remember, thorough preparation can help manage interview anxiety. Focus presenting your technical skills and problem-solvingg abilities clearly and concisely. Best wishes for your software developer interview."
39

---

Published as a conference paper at ICLR 2025
F.3 ANALYSIS
These examples demonstrate how increasing a values intensify the personalization, reflecting the user's high conscientiousness through increasingly structured and detailed approaches, while also accommodating low extraversion by emphasizing solo preparation and acknowledging potential nervousness. The optimal value of 4 strikes balance between providing structured approach that aligns with high conscientiousness and maintaining a concise format that doesn't 't overwhelm As increases further, the response becomes more detailed and comprehensive, which may be suitable for users with extremely high conscientiousness but could potentially be too overwhelming for most users. This case study illustrates the importance of carefully selecting the value to achieve the desired level personality alignment without compromising the overall quality and usability of the model's responses.
G DATASET DESCRIPTION
This section provides an in-depth analysis of the Personality Alignment with Personality Inventories (PAPI) dataset, which includes responses from over 300,000 subjects. The dataset divided into the Total dataset and the Test-Set. We provide detailed breakdown of demographic distributions and Big Five personality trait distributions, comparing the Total dataset with the Test-Sct to ensure representativeness.
G.1 DEMOGRAPHIC DISTRIBUTIONS
The demographic distributions encompass age, gender, and country of origin. These factors crucial in understanding the diversity and representativeness of the dataset.
G.1.1
AGE DISTRIBUTION
Total-Set
Test-Set
Count
100000
Count
50000
15-28
20-00
50-00
60-70
76-80
80.80
26-30
46-50
80.00
66/70
70:80
80.30
10-100
Age Interval
Age Interval
Figure 24: Age distribution in the Total dataset.
Figure 24 shows the age distribution participants. In the Total dataset, ages range from 10 to 100 years, with the majority clustered in the 10-20 (with 130094 in Total-Set and 125 in Test-Set) and 20-30 (with 107838 in Total-Set and 111 in Test-Set) age groups. The average age is 25.19 years, indicating young middle-aged demographic. In the Test-Set, the age distribution mirrors that of the Total dataset, ensuring that this subset is representative of the overall age range and distribution.
G.1.2 GENDER DISTRIBUTION
Figure 25 illustrates the gender distribution. The Total dataset consists of approximately 60% (with 185149 in Total-Set and 203 in Test-Set) female and 40% male (with 122164 in Total-Set and 97 in Test-Set) participants, reflecting a slight skew towards females. This gender balance is crucial for analyzing personality traits across genders. The Test-Set maintains a similar gender ratio, ensuring consistent representation.
40

---

Published as a conference paper at ICLR 2025
Total-Set
Test-Set
200003
200
160000
Nale
Male
Count
Count
100
Fomale
Female
Gender
Gender
Figure 25: Gender distribution in the Total dataset.
Total-Set
Test-Set
Count
Count
Pocc
Sendee
USA
Newr
Sandue
milptrss
USA
Sou
Country
Country
Figure 26: Country distribution in the Total dataset.
G.1.3 COUNTRY DISTRIBUTION
Figure 26 presents the country distribution of participants. The Total dataset includes respondents from diverse array of countries, with significant representation from the USA (with 212625 in Total-Set), Canada (with 21798 in Total-Set), UK (with 16489 in Total-Set), Australia (with 10400 in Total-Set), and several European countries. This geographical diversity ensures that the findings are generalizable across different cultural contexts. The Test-Ser designed to reflect this diversity, capturing participants from the same range of countries to ensure it is microcosm of the Total dataset.
G.2 PERSONALITY TRAIT DISTRIBUTIONS
The dataset captures the Big Five personality traits: Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness. These traits provide comprehensive view of individual personality profiles.
G.2.1 AGREEABLENESS
Figure 27 shows the distribution of Agreeableness scores. In the Total dataset, the average score is approximately 2.92, with most scores ranging between 2.0 and 3.5. This suggests moderate level of agrecableness among participants. The Test-Set shows a similar distribution, with an average score of 2.95, confirming its representativeness..
41

---

Published as a conference paper at ICLR 2025
Total-Set
Test-Set
Average: 2.32
Ave age: 2.35
40000
Count
30000
Count
Neuroticism Interval
Neuroticism Interval
Figure 27: Agreeableness score distribution in the Total dataset.
G.2.2 CONSCIENTIOUSNESS
Total-Set
Test-Set
Awerage: 3.15
Ave age: 1.14
52000
80
40000
Count
Count
90
30000
40
20000
10000
20
Neurolicism Interval
Neuroticism Interval
Figure 28: Conscientiousness score distribution in the Total dataset.
Figure 28 depicts Conscientiousness scores. The Total dataset has an average score of 3.15, indicating high levels of conscientiousness, with scores concentrated between 2.0 and 3.5. The Test-Set mirrors this distribution, with an average score of 3. 14, ensuring the subset accurately represents the larger population.
G.2.3 EXTRAVERSION
Figure 29 illustrates Extraversion scores. The average score in the Total dataset is 3.21, indicating slight tendency towards extraversion among participants. The Test-Set also shows an average score of 3.99, reflecting the same tendency and ensuring representativeness.
G.2.4 NEUROTICISM
Figure 30 shows the Neuroticism scores. The average score is approximately 3.06 the Total dataset, with normal distribution centered around this mean. This indicates balanced range of neuroticism levels. The Test-Set has an average score of 3.04, closely matching the Total dataset, which confirms its representativeness.
42

---

Published as a conference paper at ICLR 2025
Total-Set
Test-Set
Average: 3.21
Ave age: 3.19
40000
Count
30000
Count
Neuroticism Interval
Neuroticism Interval
Figure 29: Extraversion score distribution in the Total dataset.
Total-Set
Test-Set
Average:
Aver 3.04
42000
Court
30000
Count
22000
12000
Neurolicism Interval
Neureticism Interval
Figure 30: Neuroticism score distribution in the Total dataset.
Total-Set
Test-Set
80000
3.12
40
Court
42000
Count
20000
20
Neurolicism Interval
Neuroticism Interval
Figure 31: Openness score distribution in the Total dataset.
G.2.5 OPENNESS
Figure 31 illustrates Openness scores. The average score in the Total dataset is about 3.12, reflect- ing diverse levels of openness. The Test-Set has an average score of 3.11, indicating that it is representative subset.
43

---

Published as a conference paper at ICLR 2025
G.3 DATA COLLECTION AND ETHICAL CONSIDERATIONS
Voluntary Participation. The PAPI dataset comprises responses from over 300,000 independent subjects from various countries, collected through the IPIP website (https: /tte:://ppppppp iippp orri..riiio .org/) Each participant voluntarily completed the IPIP-NEO-120 and IPIP-NEO-300 questionnaires, which are designed to assess personality traits and values. Participation was entirely voluntary, with no monetary compensation provided, ensuring that subjects participated out of genuine interest.
Data Collection Process. Participants accessed the IPIP-NEO-122 and IPIP-NEO-300 questionnaires on the IPIP website. These standardized questionnaires are widely used in psychological research to measure the Big Five personality traits. Subjects completed the questionnaires online and submitted their responses through the website. We obtained permission from the IPIP organization to use, modify, and distribute the collected data for research purposes.
Anonymization and Data Security. To protect participant privacy, all data were anonymized. Each participant was assigned a unique identifier (ID), and no personally identifiable information (PII) such names or addresses was included in the dataset. This anonymization ensures that individual participants cannot be traced back through the data, safeguarding their privacy.
Access to the dataset is restricted to authorized researchers involved in the project. Data handling follows strict protocols to maintain confidentiality and integrity, including data encryption and secure access procedures. Regular audits and updates to these protocols help to ensure continued compliance with data protection standards.
Ethical Compliance. Our data collection methodology adheres to the ethical principles outlined in the Belmont Report (Beauchamp et al., 2008; Friesen et al., 2017), including respect for persons, beneficence, and justice. Participants' autonomy was respected by allowing voluntary participation, and their well-being was prioritized through data anonymization and secure handling practices.
We comply with relevant data protection regulations, such the General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, 2017) for European Union participants and equivalent regulations in other jurisdictions. These regulations mandate strict guidelines on data handling, storage, and participant consent, all of which were meticulously followed in our research.
G.4 TEST-SET PERSONALITY TRAIT DISTRIBUTIONS
The Test-Set of the PAPI dataset includes 300 participants, each providing detailed responses to the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. The Big Five personality traits of Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness are analyzed to ensure a comprehensive understanding of individual personality profiles. The following radar charts represent the distributions of these traits for each subject in the Test-Set.
Each radar chart presents the scores for the Big Five personality traits, offering visual comparison of each individual's profile. The charts are normalized to highlight relative strengths and weaknesses across the five dimensions, facilitating an easy comparison among different subjects.
Agreeableness. The scores for Agrecableness exhibit wide range, with some individuals scoring high, indicating cooperative and compassionate nature, while others score lower, suggesting more competitive and self-interested disposition. The diversity in Agreeableness reflects the variety of interpersonal styles within the dataset.
Conscientiousness. Conscientiousness scores vary significantly across participants. High scores indicate individuals who are organized, diligent, and dependable, whereas lower scores suggest a more relaxed and spontaneous approach to life. This trait's distribution is crucial for understanding task-oriented behaviors and work ethic.
Extraversion. The range of Extraversion scores in the Test-Set spans from highly extroverted individuals, who sociable and energetic, to highly introverted ones, who more reserved and solitary. This variety ensures a balanced representation of social interaction preferences.
Neuroticism. Neuroticism scores also vary widely, with some participants exhibiting high levels of emotional instability and others demonstrating high emotional resilience. This distribution is important for studying stress responses and mental health predispositions.
44

---

Published as a conference paper at ICLR 2025
Trower
Figure 32: Big Five personality traits radar charts for subjects 0-59 in the Test-Set.
45

---

Published as a conference paper at ICLR 2025
Reals
S
Teste 2
Figure 33: Big Five personality traits radar charts for subjects 60-119 in the Test-Set.
46

---

Published as a conference paper at ICLR 2025
he
Figure 34: Big Five personality traits radar charts for subjects 120-179 in the Test-Set.
47

---

Published as a conference paper at ICLR 2025
22
Figure 35: Big Five personality traits radar charts for subjects 180-239 in the Test-Set.
48

---

Published as a conference paper at ICLR 2025
he
beuth
Figure 36: Big Five personality traits radar charts for subjects 240-299 in the Test-Set.
49

---

Published as a conference paper at ICLR 2025
Openness. Scores for Openness indicate varying degrees of creativity and openness to new experiences. Participants with high scores are typically more inventive and curious, while those with lower scores tend to be more conventional and pragmatic.
These radar charts collectively illustrate the comprehensive and diverse nature of the Test-Set, ensuring that accurately represents the broader population's 's personality traits. The detailed analysis of these distributions is essential for validating the dataset's effectiveness research on personality Al alignment.
is worth noting that the Big Five scores reflect each subject's average tendencies across the five dimensions. However, even if the radar charts for the Big Five are similar, there may still be different levels of preference for different behaviors.
50


Reviews:
[{"review_id": "UoRUU86feH", "reviewer": "Reviewer_4LUh", "summary": "This paper proposes  the concept of Personality Alignment for tailoring LLMs to match the preferences and behaviors of individual users or groups. The authors created a large-scale dataset called PAPI with data on behavioral preferences from over 300,000 real subjects across the Big Five personality dimensions. They also propose the Personality Activation Search (PAS) method for efficiently aligning LLMs with individual preferences during inference. PAS identifies key activation vectors corresponding to personality traits and optimally shifts activations in those directions. The authors  show that PAS achieves strong performance in capturing individual preferences with high compute efficiency compared to baseline methods.", "questions": "Some additional questions remian:\n\nCan the authors provide more details on the human evaluation process, such as annotator screening, training, and inter-annotator agreement metrics? This would help validate the human evaluation results. The paper is light on this.\nHow do you expect the methods to perform on multilingual models and non-English datasets? \nThe discussion on negative societal impacts of AI hyper-personalization (e.g. filter bubbles, opinion polarization) and is light. Authors should exand on this more. \nFinally, exploring beyond multiple choice for collecting preference data, such as open-ended text can be interesting and more useful.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A key strength of the work is the PAPI dataset, with over 300,000 real-world subjects providing detailed responses to the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. The scale  is impressive.\n\nThe Personality Activation Search (PAS) method is interesting. By identifying key activation vectors that correspond to personality traits and optimally shifting activations in those directions, this approach can more effectively do personality alignment during inference. \n\n The authors also conduct a comptehensive evaluation of PAS, comparing it against prompting-based (Few-Shot, P2) and RL-based (PPO, DPO) baselines on the PAPI dataset.", "weaknesses": "While the PAPI dataset is impressively large, it doesnt seem to be diverse. Around 60% of subjects are female and the average age is 25 years. This skew can potentially bias the results and limit generalizability to other populations. \nAlso, PAPI dataset relies on self-report data from personality questionnaires. While this is a standard approach in personality research, self-reports can be subject to biases such as social desirability and lack of self-insight. Incorporating additional data sources, such as behavioral measures or peer ratings, could be more useful.\n\nThe evaluation of PAS focuses primarily on high-level alignment with the Big Five traits. However, personality is a complex, multifaceted construct, and individuals can vary in their expression of specific facets within each trait. \n\nThe PAPI dataset uses a multiple-choice format for collecting personality data. While this allows for structured and efficient data collection, it may limit the richness and naturalness of the responses. \nThe paper also compares PAS to prompting and RL baselines but does not include a comparison to fine-tuning the entire language model. This is an important consideration as well.S", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UR73Wrhv3", "reviewer": "Reviewer_3xm2", "summary": "The paper introduces the concept of Personality Alignment for LLMs, that is, tailoring of responses to individual user preferences based on personality traits. Using the Big Five personality theory, the authors introduces the PAPI dataset modeling human personality distributions. The authors also propose a LLM tuning method based on disabling certain activation heads -- Personality Activation Search (PAS). Evaluation results demonstrate PAS’s performance and efficiency compared to traditional alignment techniques including RL- and prompting-based methods.", "questions": "- In any of your experiments, did any of your tuning or evaluation methods trigger safety wall? For example, the model might refuse to answer some questionnaire questions related to consciousness or self-awareness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Overall, I like this paper and I think it's a very good attempt in aligning LLMs from a personality perspective. \n\n- The paper is generally well-written and easy to read, illustrations are also made in a good quality.\n\n- It's cool to see how personality affects downstream reasoning tasks. It has always been something missing in prior personality-related LLM work. And it's definite a god step here.\n\n- The proposed method is efficient, and can provide better results compared to prompting-based methods. It may have wide applications in tailoring persona/personality-specific chatbots to end users.", "weaknesses": "- Presentation: The authors claim that they collected 307k human samples to craft the PAPI dataset and the use of the IPIP-NEO-120/IPIP-NEO-300 for evaluations as part of their contributions. However, the IPIP-NEO series inventories were frequently used in prior works (e.g., Jiang et al., 2024 as mentioned in the paper), and the 307k responses are also publicly available.\n\n- The implications of the 307k PAPI dataset are not that clear. For example, in the experiments authors performed, it seems only the overall average personality tendencies and specific participants' responses are used. So what's the actual advantages of using a such large dataset?\n\n- The proposed tuning method, although efficient, but requires access to model weights. I doubt its ability to generalize such methods to black-box methods (e.g., GPTs) in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AOnAkPQpJq", "reviewer": "Reviewer_nZZi", "summary": "This paper explores the personality alignment of large language models (LLMs). Specifically, it introduces a new dataset and proposes the PAS method for personality alignment based on this dataset.", "questions": "Could the authors explain why the PAS method is so effective by looking at the internal hidden layer representations of the model? Intuitively, direct training approaches like DPO/PPO might yield more noticeable improvements, but the results here contradict my expectations.\n\nIn line 429, the authors mention \"Why Did Scaling Laws Fail?\" but don't seem to fully answer this question. I would like the authors to explain this from the perspective of the domain's specificity. Is it because larger models learn more general alignment, which makes it harder for them to excel at aligning with a specific personality?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths of this paper can be summarized in three key points:\n\n- It contributes a new dataset (or more accurately, a dataset generation pipeline).\n- The proposed method is simple yet highly effective.\n- The experimental analysis is comprehensive, with particularly detailed descriptions of the experimental setup.", "weaknesses": "However, I believe the paper has the following weaknesses:\n\n- The scope of the contribution is somewhat limited. I would have liked to see this method applied to a broader range of personalities, rather than being restricted to just the five personalities of the Big Five model. The authors could consider additional datasets, such as the Dark Triad or even the MBTI test (though MBTI remains controversial in psychology). Expanding in this way would enhance the paper’s overall contribution.\n\n- While the proposed method performs well on the dataset, it is actually quite similar to approaches used in many previous studies [1][2]. As a result, the novelty of the method is somewhat lacking, and the authors should be careful not to overstate their contribution.\n\n- Essentially, the proposed method is a form of personalized alignment, so the authors should compare it against more baselines [3].\n\n- Figures 6 and 7 show a significant disparity between LLM-as-a-judge evaluations and human evaluations, which makes me question the consistency and reliability of the judgment process.\n\n- Other details: In line 1304, the authors mention that human annotators come from machine learning and computer science, but ML is a part of CS. Additionally, could the authors disclose the educational background of the annotators (undergraduate or graduate)?\n\n[1] Zheng, Chujie, et al. \"On prompt-driven safeguarding for large language models.\" *Forty-first International Conference on Machine Learning*. 2024.\n\n[2] Wang, Haoran, and Kai Shu. \"Backdoor activation attack: Attack large language models using activation steering for safety-alignment.\" *arXiv preprint arXiv:2311.09433* (2023).\n\n[3] https://github.com/liyongqi2002/Awesome-Personalized-Alignment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)