Personality Alignment of Large Language Models

Authors: Minjun Zhu, Yixuan Weng, Linyi Yang, Yue Zhang
Primary area: generative models
Keywords: Personality Alignment, Large language models, behavioral preferences of LM

Abstract:
Aligning large language models (LLMs) typically aim to reflect general human values and behaviors, but they often fail to capture the unique characteristics and preferences of individual users. To address this gap, we introduce the concept of Personality Alignment. This approach tailors LLMs' responses and decisions to match the specific preferences of individual users or closely related groups. Inspired by psychometrics, we created the Personality Alignment with Personality Inventories (PAPI) dataset, which includes data from over 320,000 real subjects across multiple personality assessments - including both the Big Five Personality Factors and Dark Triad traits. This comprehensive dataset enables quantitative evaluation of LLMs' alignment capabilities across both positive and potentially problematic personality dimensions. Recognizing the challenges of personality alignments—such as limited personal data, diverse preferences, and scalability requirements—we developed an activation intervention optimization method. This method enhances LLMs' ability to efficiently align with individual behavioral preferences using minimal data and computational resources. Remarkably, our method, PAS, achieves superior performance while requiring only 1/5 of the optimization time compared to DPO, offering practical value for personality alignment. Our work paves the way for future AI systems to make decisions and reason in truly personality ways, enhancing the relevance and meaning of AI interactions for each user and advancing human-centered artificial intelligence. The dataset and code are released at https://github.com/zhu-minjun/PAlign.

Decision: accept (poster)
Reviewer scores: 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730000679458

thread_type:
official_review

<thread_object>
creator: reviewer_nzzi

<content>
summary:
This paper explores the personality alignment of large language models (LLMs). Specifically, it introduces a new dataset and proposes the PAS method for personality alignment based on this dataset.

strengths:
The strengths of this paper can be summarized in three key points:

- It contributes a new dataset (or more accurately, a dataset generation pipeline).
- The proposed method is simple yet highly effective.
- The experimental analysis is comprehensive, with particularly detailed descriptions of the experimental setup.

weaknesses:
However, I believe the paper has the following weaknesses:

- The scope of the contribution is somewhat limited. I would have liked to see this method applied to a broader range of personalities, rather than being restricted to just the five personalities of the Big Five model. The authors could consider additional datasets, such as the Dark Triad or even the MBTI test (though MBTI remains controversial in psychology). Expanding in this way would enhance the paper’s overall contribution.

- While the proposed method performs well on the dataset, it is actually quite similar to approaches used in many previous studies [1][2]. As a result, the novelty of the method is somewhat lacking, and the authors should be careful not to overstate their contribution.

- Essentially, the proposed method is a form of personalized alignment, so the authors should compare it against more baselines [3].

- Figures 6 and 7 show a significant disparity between LLM-as-a-judge evaluations and human evaluations, which makes me question the consistency and reliability of the judgment process.

- Other details: In line 1304, the authors mention that human annotators come from machine learning and computer science, but ML is a part of CS. Additionally, could the authors disclose the educational background of the annotators (undergraduate or graduate)?

[1] Zheng, Chujie, et al. "On prompt-driven safeguarding for large language models." *Forty-first International Conference on Machine Learning*. 2024.

[2] Wang, Haoran, and Kai Shu. "Backdoor activation attack: Attack large language models using activation steering for safety-alignment." *arXiv preprint arXiv:2311.09433* (2023).

[3] https://github.com/liyongqi2002/Awesome-Personalized-Alignment

questions:
Could the authors explain why the PAS method is so effective by looking at the internal hidden layer representations of the model? Intuitively, direct training approaches like DPO/PPO might yield more noticeable improvements, but the results here contradict my expectations.

In line 429, the authors mention "Why Did Scaling Laws Fail?" but don't seem to fully answer this question. I would like the authors to explain this from the perspective of the domain's specificity. Is it because larger models learn more general alignment, which makes it harder for them to excel at aligning with a specific personality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661011507

thread_type:
official_review

<thread_object>
creator: reviewer_3xm2

<content>
summary:
The paper introduces the concept of Personality Alignment for LLMs, that is, tailoring of responses to individual user preferences based on personality traits. Using the Big Five personality theory, the authors introduces the PAPI dataset modeling human personality distributions. The authors also propose a LLM tuning method based on disabling certain activation heads -- Personality Activation Search (PAS). Evaluation results demonstrate PAS’s performance and efficiency compared to traditional alignment techniques including RL- and prompting-based methods.

strengths:
- Overall, I like this paper and I think it's a very good attempt in aligning LLMs from a personality perspective. 

- The paper is generally well-written and easy to read, illustrations are also made in a good quality.

- It's cool to see how personality affects downstream reasoning tasks. It has always been something missing in prior personality-related LLM work. And it's definite a god step here.

- The proposed method is efficient, and can provide better results compared to prompting-based methods. It may have wide applications in tailoring persona/personality-specific chatbots to end users.

weaknesses:
- Presentation: The authors claim that they collected 307k human samples to craft the PAPI dataset and the use of the IPIP-NEO-120/IPIP-NEO-300 for evaluations as part of their contributions. However, the IPIP-NEO series inventories were frequently used in prior works (e.g., Jiang et al., 2024 as mentioned in the paper), and the 307k responses are also publicly available.

- The implications of the 307k PAPI dataset are not that clear. For example, in the experiments authors performed, it seems only the overall average personality tendencies and specific participants' responses are used. So what's the actual advantages of using a such large dataset?

- The proposed tuning method, although efficient, but requires access to model weights. I doubt its ability to generalize such methods to black-box methods (e.g., GPTs) in the future.

questions:
- In any of your experiments, did any of your tuning or evaluation methods trigger safety wall? For example, the model might refuse to answer some questionnaire questions related to consciousness or self-awareness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687483135

thread_type:
official_review

<thread_object>
creator: reviewer_4luh

<content>
summary:
This paper proposes  the concept of Personality Alignment for tailoring LLMs to match the preferences and behaviors of individual users or groups. The authors created a large-scale dataset called PAPI with data on behavioral preferences from over 300,000 real subjects across the Big Five personality dimensions. They also propose the Personality Activation Search (PAS) method for efficiently aligning LLMs with individual preferences during inference. PAS identifies key activation vectors corresponding to personality traits and optimally shifts activations in those directions. The authors  show that PAS achieves strong performance in capturing individual preferences with high compute efficiency compared to baseline methods.

strengths:
A key strength of the work is the PAPI dataset, with over 300,000 real-world subjects providing detailed responses to the IPIP-NEO-120 and IPIP-NEO-300 questionnaires. The scale  is impressive.

The Personality Activation Search (PAS) method is interesting. By identifying key activation vectors that correspond to personality traits and optimally shifting activations in those directions, this approach can more effectively do personality alignment during inference. 

 The authors also conduct a comptehensive evaluation of PAS, comparing it against prompting-based (Few-Shot, P2) and RL-based (PPO, DPO) baselines on the PAPI dataset.

weaknesses:
While the PAPI dataset is impressively large, it doesnt seem to be diverse. Around 60% of subjects are female and the average age is 25 years. This skew can potentially bias the results and limit generalizability to other populations. 
Also, PAPI dataset relies on self-report data from personality questionnaires. While this is a standard approach in personality research, self-reports can be subject to biases such as social desirability and lack of self-insight. Incorporating additional data sources, such as behavioral measures or peer ratings, could be more useful.

The evaluation of PAS focuses primarily on high-level alignment with the Big Five traits. However, personality is a complex, multifaceted construct, and individuals can vary in their expression of specific facets within each trait. 

The PAPI dataset uses a multiple-choice format for collecting personality data. While this allows for structured and efficient data collection, it may limit the richness and naturalness of the responses. 
The paper also compares PAS to prompting and RL baselines but does not include a comparison to fine-tuning the entire language model. This is an important consideration as well.S

questions:
Some additional questions remian:

Can the authors provide more details on the human evaluation process, such as annotator screening, training, and inter-annotator agreement metrics? This would help validate the human evaluation results. The paper is light on this.
How do you expect the methods to perform on multilingual models and non-English datasets? 
The discussion on negative societal impacts of AI hyper-personalization (e.g. filter bubbles, opinion polarization) and is light. Authors should exand on this more. 
Finally, exploring beyond multiple choice for collecting preference data, such as open-ended text can be interesting and more useful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732125131146

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses and Summary of Revisions

comment:
We sincerely thank all reviewers for their careful and constructive feedback, which has helped us significantly improve our work. We are encouraged by the reviewers' recognition of several key strengths:

- "A key strength of the work is the PAPI dataset, with over 300,000 real-world...The scale is impressive" (*Review 4LUh*)
- "Overall, I like this paper and I think it's a very good attempt in aligning LLMs from a personality perspective." (*Review 3xm2*)  
- "The experimental analysis is comprehensive, with particularly detailed descriptions of the experimental setup." (*Review nZZi*)





Based on your constructive comments, we have made substantial improvements to strengthen our paper's contributions and address the  limitations. The major enhancements include:

1. **Extended Dataset Coverage**: We expanded PAPI with 18,192 Dark Triad questionnaire samples, enabling comprehensive assessment of both positive (Big Five) and negative (Machiavellianism, Narcissism, Psychopathy) personality dimensions. This enhancement provides a more complete framework for personality alignment evaluation (Abstract, Section 1: Introduction, Section 3: Dataset Construction).

2. **Demographic Analysis**: We added comprehensive analysis of PAS performance across different age groups, genders, and nationalities, demonstrating robust alignment capabilities across diverse populations. This validates the broad applicability of our approach (Appendix E.5: Diverse Demographic Groups).

3. **Evaluation Protocol Details**: We supplemented our evaluation framework with specific details about the human evaluation interface, annotator qualifications, and inter-annotator agreement metrics (IAA=0.82). These additions enhance the reproducibility and reliability of our results (Appendix C.5).

4. **Technical Discussion**: We provided additional analysis explaining why scaling laws don't automatically improve personality alignment, highlighting how PAS's targeted intervention achieves better results than larger models through precise control of personality-relevant components (Section 5: Experiments).

5. **Ethics Framework**: We supplemented our discussion with specific mitigation strategies for potential risks of personality alignment, including dynamic alignment boundaries and content diversity systems. This provides a practical framework for responsible deployment (Section 7: Ethics Statement).

6. **Method Analysis Detail**: We added a detailed comparison of how PAS differs from traditional methods in modifying model representations, particularly highlighting our two-stage approach versus global parameter updates. This clarifies PAS's unique advantages in precise personality alignment (Appendix E.6).

With the generous guidance from our esteemed reviewers, we have been able to substantially enhance our paper's technical depth, experimental rigor, and practical impact. We are deeply grateful for the opportunity to improve our work and would be honored if the reviewers find these comprehensive enhancements worthy of a more favorable evaluation. We believe these additions have helped realize the full potential of our research contribution to the field of personality alignment in AI systems.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733224280631

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewer 4LUh and nZZi:


Given that the author-reviewer discussion period will **end in an hour**, we are eager to ensure that all your suggestions and comments have been thoroughly addressed!

Once again, thank you for your thoughtful review process, and we look forward to your final comments!

Best Regards,

Authors of "Personality Alignment of Large Language Models"
</content>
</thread_object>

</thread>


Metareview:
metareview:
We recommend the paper to be accepted for Poster. 

The paper can be of interest to the wide community at ICLR working on LLM and it introduces a relatively novel methodology that seems to be more efficient than baseline methods. 

Below a more detailed description of the paper.

The paper introduces the concept of Personality Alignment, aimed at customizing large language models (LLMs) to align with the preferences and behaviors of individual users or groups. To support this approach, the authors developed a large-scale dataset called PAPI, which contains behavioral preference data from over 300,000 real participants across the Big Five personality dimensions. They also propose a novel method called Personality Activation Search (PAS) to efficiently align LLMs with user preferences during inference. PAS identifies key activation vectors corresponding to personality traits and optimally adjusts activations along these dimensions. 

The strengths (S#) of the paper are as follows: 
 
- (S1)	The results demonstrate that PAS outperforms baseline methods in capturing individual preferences while maintaining high computational efficiency
- (S2)	The paper provides a new dataset generation pipeline that can be used and enriched for the task taken into account.
- (S3)	The experimental analysis is comprehensive and provides 
- (S4)	The paper is well written, and easy to follow also for a non-specialized audience

The key weaknesses (W#) identified and that remains are as follows: 

- (W1)	Use of model weights may hinder the applicability of the method to few LLMs
- (W2)	Possible bias in the annotators used (only from CS community)
- (W3)	Significant disparity between LLM-as-a-judge evaluations and human evaluations should be further discussed

Many of the points raised by the reviewers were addressed by the authors.