Measuring Human and AI Values based on Generative Psychometrics with
  Large Language Models
Authors: name: Haoran Ye; name: Yuhang Xie; name: Yuanyi Ren; name: Hanjun Fang; name: Xin Zhang; name: Guojie Song
Published: 2024-09-18 16:26:22

Abstract: Human values and their measurement are long-standing interdisciplinary
inquiry. Recent advances in AI have sparked renewed interest in this area, with
large language models (LLMs) emerging as both tools and subjects of value
measurement. This work introduces Generative Psychometrics for Values (GPV), an
LLM-based, data-driven value measurement paradigm, theoretically grounded in
text-revealed selective perceptions. We begin by fine-tuning an LLM for
accurate perception-level value measurement and verifying the capability of
LLMs to parse texts into perceptions, forming the core of the GPV pipeline.
Applying GPV to human-authored blogs, we demonstrate its stability, validity,
and superiority over prior psychological tools. Then, extending GPV to LLM
value measurement, we advance the current art with 1) a psychometric
methodology that measures LLM values based on their scalable and free-form
outputs, enabling context-specific measurement; 2) a comparative analysis of
measurement paradigms, indicating response biases of prior methods; and 3) an
attempt to bridge LLM values and their safety, revealing the predictive power
of different value systems and the impacts of various values on LLM safety.
Through interdisciplinary efforts, we aim to leverage AI for next-generation
psychometrics and psychometrics for value-aligned AI.