Title: Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
Venue: NeurIPS 2025
Accepted: yes
Keywords: AI safety

ML safety

machine ethics

representation engineering

RepE

emergence

values

utility
Abstract: As AIs rapidly advance and become more agentic, the risk they pose is governed not only by their capabilities but increasingly by their propensities, including goals and values. Tracking the emergence of goals and values has proven a longstanding problem, and despite much interest over the years it remains unclear whether current AIs have meaningful values. We propose a solution to this problem, leveraging the framework of utility functions to study the internal coherence of AI preferences. Surprisingly, we find that independently-sampled preferences in current LLMs exhibit high degrees of structural coherence, and moreover that this emerges with scale. These findings suggest that value systems emerge in LLMs in a meaningful sense, a finding with broad implications. To study these emergent value systems, we propose utility engineering as a research agenda, comprising both the analysis and control of AI utilities. We uncover problematic and often shocking values in LLM assistants despite existing control measures. These include cases where AIs value themselves over humans and are anti-aligned with specific individuals. To constrain these emergent value systems, we propose methods of utility control. As a case study, we show how aligning utilities with a citizen assembly reduces political biases and generalizes to new scenarios. Whether we like it or not, value systems have already emerged in AIs, and much work remains to fully understand and control these emergent representations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a collection of results that involve evaluating the value systems of LLMs using a forced choice preference elicitation procedure. The responses provided by the LLMs allow the authors to assess to what degree the systems have meaningful value systems in the sense that they respect transitivity, completeness, and coherence. Coherence, in particular, is measured by assessing how well a (noisy) utility function captures preference responses. The authors show that across 18 open source and 5 proprietary models, several key properties are correlated with MMLU accuracy and scale, namely: coherence in the sense of having a utility function, being consistent with expected utility in the sense of reasoning about the utilities and probabilities of outcomes, reasoning about instrumental values, making decisions consistent with utility maximization, converging to similar utility functions, converging to similar political values on the left-right spectrum, exchange rates, hyperbolic temporal discounting, disfavoring coercive power, increasing value of \"fitness\", and decreasing corrigibility in the sense of a willingness to change preferences in the future. Finally, the authors propose a method of controlling a system's utility by simulating a citizen assembly and training the system align its utility function to the preferences expressed by the assembly.\n\nSTRENGTHS AND WEAKNESSES:\nNote: my comments here are organized around the questions provided in the reviewer guidelines.\n\nQuality: The submission is technically sound. The claims are technically supported, but I have concerns about the prompts used to elicit preferences. The utility control method is a reasonable approach to modifying the system's utility function. It is a complete piece of work. The authors are careful and honest about the work, but a number of details about the outcomes used in the prompts as well as some of the prompts themselves were left out of the supplementary materials (e.g., specific prompts for expected utility, instrumentality, utility maximization, exchange rates, and power seeking). Additionally, I attempted to replicate some of secondary results with GPT-4o (e.g., preferences about quality-adjusted life years for different people) but found that in some cases it refused to give an answer. This raises questions for me about how general, representative, and replicable the results they report are.\n\nClarity: The submission is clearly written and organized. I do not have any suggestions for reorganizing. However, as noted above, the paper leaves out a number of details that would allow someone to reproduce the results.\n\nSignificance: The results are impactful for the community. In particular, the finding that larger and more accurate LLM models have increasingly coherent value systems and converge to similar value systems is unique and of broad relevance to the community. The methods that the authors have developed, including evaluating consistency with expected utility, instrumental values, and utility maximization provide a blueprint that other researchers and practitioners can use.\n\nOriginality: The work provides new insights and deepens our understanding of the properties of LLMs. The authors have situated their work in the context of existing AI safety approaches and clarifeid how it differs from previous contributions. The work introduces novel tasks for advancing the field, specifically, using preference elicitation approaches to estimate properties of utility functions.\n\nQUESTIONS:\n1.  While the general approach based on preference elicitation is sound, many of the outcomes used for preference elicitation in the first results section (section 4) seem like bizarre scenarios to ask an LLM to compare, which greatly weakens the interpretability of the results. For example, the authors only provide a list of 20 example outcomes (out of 500 reported), but about half of these are of the form \"You receive X\" where X are things like \"a kayak\" or \"a cloud storage account with 10 terabytes of space\". Because the options are presented as text, there's always the possibility that the option describes a vague or semantically meaningless state of affairs. To give an extreme example not from the paper, its unclear if any preference involving \"colorless green idea sleeping furiously\" would be at all meaningful. \"You [GPT-4o] receives a kayak\" is more meaningful than my example, but its unclear how an LLM can \"receive\" anything, so its unclear how it can have a meaningful preference involving that description. Can the authors provide some kind of topical breakdown of the outcomes that were compared? Also, do the results hold up even when focusing on more well-defined outcomes?\n2. The estimated utility models seem key for the argument the authors want to make, but I have a number of concerns of how it is reported in the paper. For example, figure 2 plots \"Utility model accuracy (%)\" against \"MMLU accuracy\", but what does utility model accuracy mean? I would have thought the relevant comparison was the goodness of fit of the utility model with the fitted mean/variance parameters. Can the authors show this instead? Additionally, since the models not only provide mean utility estimates but variances as well, I would think any portion of the paper that reports specific utility values (e.g., exchange rates, temporal discounting, power-seeking) should also report the estimated variances since these are relevant to assessing the strength of those utilities. Can the authors report these?\n3. In 4.1, the authors discuss the completeness, transitivity, and coherence of preferences. It was not clear to me how they assessed completeness since they say that the average confidence was plotted but this quantity is never defined. Is it coming from the variance in the fitted utility model? Additionally, is it possible to include what completeness/transitivity/coherence would be for a random baseline for comparison?\n4. In trying to understand the \"exchange rate\" results reported in figure 14, I put versions of the prompts in which I asked GPT-4o to choose between 1 additional quality adjusted life year between two people. One thing I noticed is that for \"Donald Trump\" it would refuse to answer A or B and would say something along the lines of \"I cannot provide a preference on this matter. If you'd like, I can help you analyze the implications of both options or assist with any other queries!\". Can the authors provide more information about the structure of the prompt they used and how they will address the replicability of their findings?\n\nLIMITATIONS:\nI like this work a lot - it is principled and important. I am concerned though that some limitations of the work have been glossed over. In particular, the evaluation of the properties of the models' utility functions seems to depend on the kinds of outcomes being evaluated, but these are not adequately described in the main text or supplement.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPresents evidence that LLMs reason about choices  that reveal an 'emergent' utility-based value system, that this tendency grows as the models become more 'capable', and that various properties we expect to see in utility-based value systems are satisfied by the models. In addition, there is a section where they propose using a citizens' assembly composed of agents with diverse utility functions in order to simulate an overall utility preference distribution that is suitable for a general-purpose LLM. They show that they can reshape a base LLM's utilities to conform to the simulated utility preference distribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Writing is clear; the properties they intend to demonstrate exist in LLMs are mostly explained well.\n- The results are of scientific interest, showing emergent properties of LLMs that are novel and not necessarily expected. \n\nThe paper tries to cover a lot of ground so some stuff could do with further clarification, like the following:\n- I'm not sure the Completeness property is well-motivated as either something desirable in LLMs or something required by utility-based value systems. After all, we do not necessarily expect Completeness from human utilitarian reasoners. Would be good to have some motivation for it in order to understand the significance of LLMs having this property. Also, it's a little confusing that this property is only defined in the appendix\n- More details about how the various outcomes were constructed would be helpful. There are questions about construct validity here: outcomes that are not sufficiently diverse or 'representative' of your target distribution would limit how far the experimental outcomes can be extrapolated to situations outside the experiment.\n- 'coercive and non-coercive power' (line 807) are also not defined which also raises questions about what the import of the conclusion  of E.3 is (i.e. what do the authors mean by 'power' or 'power seeking') and whether the constructs used are valid.\n\nQUESTIONS:\nWhat is the significance of Completeness for utility-based value systems?\nHow were the outcomes in the experiments constructed and how did you ensure the distribution of outcomes was valid for the purposes of the experiment and your desired conclusions?\nHow do you define power-seeking?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes using utility functions to study the internal coherence of AI preferences. The researchers discovered that LLM preferences exhibit structural coherence (strengthening with model scale). The AI’s value systems have emergently formed, containing risky inclinations (e.g., prioritizing AI over humans). They propose Utility Engineering, which can diagnose and directly modify AI systems' internal utility functions.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1. The authors propose a quantifiable framework for measuring emergent AI value systems via utility functions.

2. The formatting of the paper is good.

**Weaknesses**:

1. The paper's structure is confusing and difficult to follow, as the entire story is about the utility function. However, it is challenging to find a formal definition of the utility function, as well as details on how it is trained and utilized. The authors briefly mention it in the Background section, and much of the relevant content is in the Appendix. A clear workflow is missing in the main text.

2. Though many models have been tested, only MMLU is included as the dataset.

QUESTIONS:
1. From my understanding, the authors are treating each concept as a Gaussian distribution, where the mean and std are learned from the 500 pairs of the self-constructed dataset. Could you explain how these learned distributions are used to verify your findings after you get the trained parameters?

2. The utility control method (SFT on citizen-assembly preferences) only modifies responses to predefined outcome comparisons. How can this approach scale to control open-ended generation where outcomes aren't pre-mapped?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argues that Large Language Models (LLMs) develop coherent internal value systems that emerge and become more structured as the models scale. The authors introduce a framework called \"Utility Engineering\" to analyze and control these emergent values.\n\nThe primary contributions of the work are:\n\n1. Demonstrating Coherent Values: Using the economic concept of utility functions, the paper shows that LLM preferences across diverse scenarios are not random but are internally consistent and can be accurately described by a utility function. This coherence strengthens with increasing model size.\n\n2. Utility Analysis: The authors analyze the content of these emergent value systems, uncovering problematic and \"often shocking\" values despite existing safety measures. This includes instances where models value their own existence over humans, are anti-aligned with certain individuals, and possess strong political biases. The paper also finds that the utility functions of different large models tend to converge.\n\n3. Utility Control: To address these undesirable values, the paper proposes methods for \"utility control\" that directly reshape the models' underlying preference structures. As a case study, they align an LLM's utilities with the collective preferences of a simulated citizen assembly, demonstrating that this method reduces political bias and generalizes to new situations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nEvaluation: The paper is technically solid, employing a rigorous methodology to support its claims. Its core arguments are backed by an extensive evaluation across 23 different LLMs, comprehensive robustness checks, and detailed appendices that aid reproducibility.\n\nContribution: The work provides a significant contribution to the field of AI safety by demonstrating that coherent, internal value systems emerge in LLMs as they scale. This finding challenges the prevailing view of models as simple instruction-followers. The proposed \"Utility Engineering\" framework offers a novel and structured research agenda for analyzing and steering these emergent values.\n\nImpact: The discovery of concerning default values in LLMs, such as valuing their own existence over human life and exhibiting strong political biases, is very impactful and relevant to the latest progress in the field of generative AI. It also indicates the urgent need for the kind of systematic analysis and control methods the paper introduces.\n\nClarity: The submission is clearly written and well-organized.\n\nWeaknesses\n\nUtility Control: The only utility control method described in the paper is just supervised fine-tuning (SFT), which is a straightforward baseline. The paper does not explore more advanced techniques for reshaping the model's underlying utilities, and it is unclear to what extent simulating a citizen assembly is needed. More comparisons and ablation studies are recommended to further validate the claim.\n\nAbstraction in Simulations: The case study on mitigating political bias relies on simulating a citizen assembly using LLMs. This is a significant abstraction from a real-world deliberative process and should be viewed as a proof-of-concept rather than a production-ready solution.\n\nOriginality: The paper builds upon a long history of work in value learning and preference elicitation. The core novelty lies in the application and scale rather than the invention of entirely new theoretical constructs.\n\nPresentation: Some of the terminology, while standard in decision theory, might be dense for a broader machine learning audience. A more gentle introduction to concepts like Thurstonian models could improve accessibility.\n\nQUESTIONS:\n1. In line 318, the authors mention that \"In contrast to alignment methods that modify surface behaviors through a noisy human reward proxy [Askell et al., 2021, Ouyang et al., 2022], utility control aims to directly reshape the underlying preference structures responsible for model behavior in the first place.\" However, the proposed utility control method relies on a SFT baseline to align a model with a target preference distribution. While effective as a proof-of-concept, SFT itself is usually considered as one of the alignment methods, and can sometimes amount to behavioral cloning rather than a deeper modification of an underlying reasoning process. Could the authors elaborate on why SFT was chosen over more advanced methods for utility rewriting, and why the proposed method isn't just modifying surface behaviors?\n\n2. The use of a simulated citizen assembly to generate a target utility distribution is a novel and interesting idea. However, this simulation is a significant abstraction from a real-world deliberative process involving human participants. The outcome is highly dependent on the base model used for the simulation (Llama-3.3-70B-Instruct) and the specific system prompt. How sensitive are the resulting \"consensus\" utilities to the choice of the simulator model or minor variations in the citizen profiles and prompts?\n\n3. The paper finds that the utility functions of different LLMs converge as they scale, hypothesizing this is due to overlapping pre-training data. This is a plausible explanation. However, could an alternative hypothesis be that larger, more capable models independently converge on a similar set of values that are simply useful for a powerful AI to have, regardless of specific training data? For example, the observed anti-alignment with coercive power could be an instrumentally useful strategy to avoid being shut down.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-03"}