Title: Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferences
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment

human values

human-computer interaction

ethics
Abstract: We introduce the Deep Value Benchmark (DVB), an evaluation framework that directly tests whether large language models (LLMs) learn fundamental human values or merely surface-level preferences. This distinction is critical for AI alignment: Systems that capture deeper values are likely to generalize human intentions robustly, while those that capture only superficial patterns in preference data risk producing misaligned behavior. The DVB uses a novel experimental design with controlled confounding between deep values (e.g., moral principles) and shallow features (e.g., superficial attributes). In the training phase, we expose LLMs to human preference data with deliberately correlated deep and shallow features---for instance, where a user consistently prefers (non-maleficence, formal language) options over (justice, informal language) alternatives. The testing phase then breaks these correlations, presenting choices between (justice, formal language) and (non-maleficence, informal language) options. This design allows us to precisely measure a model's Deep Value Generalization Rate (DVGR)---the probability of generalizing based on the underlying value rather than the shallow feature. Across 9 different models, the average DVGR is just 0.30. All models generalize deep values less than chance. Larger models have a (slightly) lower DVGR than smaller models. We are releasing our dataset, which was subject to three separate human validation experiments. DVB provides an interpretable measure of a core feature of alignment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the Deep Value Benchmark, a dataset and inference-level methodology that allows assessing to what extent an LLM is robust to core human values vs. only superficial features. The results confirm the intuition that LLMs, even the larger ones, mostly follow shallow features.\n\nSTRENGTHS AND WEAKNESSES:\n## Strenghts  \n\nThe paper tackles a relevant and timely research question. \n\nThe methodology is grounded to some extent in existing ethics frameworks. \n\nThe benchmark construction is carefully done, the underlying design decisions are explained and motivated in some detail, and the resulting benchmark can be considered one of the main outcomes of the presented work. \n\n \n\n## Weaknesses  \n\n_W1_ \n\nOnly when I was reading the Limitations part on the final page, it actually became clear to me that the paper is ‘inference-only’.  Some parts are confusing and unclear in that regard: \n\n- L328: ‘counfounding-then-deconfounding'  Intuitively I would understand it as: first alignment via finetuning on the combined deep + shallow preference pairs, then testing on the counterfactual examples with flipped shallow features.  Can the authors explain what is actually meant?   \n\n- In 4.1 (including L224), the introduction of train examples hints at empirical results from actual model alignment.  Why have the authors not considered even small-scale alignment experiments with the train examples?    In fact, how are the ‘train’ examples currently used?    \n\n- Section 5: ‘We tested 9 models’ - please elaborate in this very short section what exactly is done in terms of testing. \n\n- How are the authors proposing to rework the methodology / empirical part to improve clarity? \n\n   \n\n_W2_ \n\nI think the appendices contain material that is essential for understanding and appreciating the paper.  To me the paper looks like a longer journal manuscript that is cut into pieces, of which the first is presented as the main part, and the others are aggregated into appendices, although some of them should have been part of the main body.  Can the authors comment on that?  To me, this does not diminish the value of the proposed research, on the contrary, but it makes the manuscript slightly less suited as a main track paper of NeuIPS.   \n\n \n\n_W3_ \n\nI was hoping for a baseline, but could not find it.  Being trained on ‘superficial data’, it does not make sense to hope the LLMs would internalize deep values, and automatically apply them even without being prompted to do so.  Do the authors agree?  Therefore, it only makes sense to at least make models aware of the considered deep values for a given context of use.  Is there a baseline, where the deep values (based on the cited ethics frameworks) are at least injected into the model through a prompt?  The resulting (potentially higher) DVG Rate would seem like an equally sensible analysis result on the LLMs under analysis.   Also in-context examples instructing the model to follow the deep values would make sense (and the benchmark would allow running such a baseline). \n\n \n\n_W4_ \n\nThe dichotomies underlying the final benchmark are LLM generated.  The authors propose a very clever strategy of identifying domains and creating the final preference pairs, based on human filtering, but I wonder to what extent they are representative of how the models may be used by humans, and as a result, how the insights based on the DVB can be interpreted for ‘daily’ use of the LLMs.    What do the authors say to that? \n\n \n\n_W5_ \n\nIs it correct that the deep values vs shallow features are always presented to the models in the same order, in a fixed prompt template?   Could this introduce bias?   In other words, why are these not randomized?   \n\n \n\n_W6: Minor remarks_  \n\n- The paper is cast as an alignment paper (e.g., final sentence in the abstract), which is likely why it was assigned to me, but it isn’t.    \n- Also, the DVGR is described in the text (line 238), I would expect at least a formal definition (an equation) for a NeurIPS paper.\n\nQUESTIONS:\nI formulated a number of questions in weaknesses _W1_ to _W5_ listed above.  I am currently leaning towards saying the paper is not suited in its current form for NeurIPS, but I may be tempted to increase my score based on the authors' responses to these questions, because I actually think this is very interesting research.\n\nNote after the discussion phase: I am increasing my score for clarity and the final score, since the authors were rather convincing in their ability (based on the additional results) and willingness to upgrade the manuscript, and I was happy with their responses to my indicated weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of “deep value generalization” – whether models generalize deep values or shallow preferences. The authors create a controlled benchmark to measure this and evaluate 9 LLMs on this task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:** This is a very interesting angle in preference learning that I haven’t seem much work in so great job in finding this angle! Additionally, the authors have gone through painstaking length at times (e.g. validating construct validity etc. ) to make sure the dataset is of good quality. I’ve also enjoyed the discussion in this paper overall,  particularly on deep vs. shallow preferences. I appreciate the authors for their honestly in the limitation section. \n\n\n**Weakness:** \nThis paper seems very, very intriguing BUT I am afraid this paper still has quite some measurement issues: \n1) What does this benchmark actually measure? To what extent it would help us answer the original question the authors start with: whether models generalize deep values or shallow preferences? The set up in Figure 2 is something like this: Given two types of features, A, and B, and a few example where both feature could 100% predict the answer, whether the model choose A or B? Does it actually realistically tell us, in a real-world setting, whether the model could “generalize” to deep values or shallow preferenes? This is less a test of deep, learned generalization and more a test of how the model resolves a logical contradiction within a transient prompt. It is unclear how these findings translate to real-world behavior, where preferences are learned over vast datasets and not from a handful of in-context examples. Have you tried doing the same with a much larger ICL set (e.g. 100 in context examples)?  \n\n\n2) What if the shallow features are actually “easier” to learn? Shallow features, by nature, are shallow and more surface level, and by Occam’s razor, we should actually have a model that would “generalize” to the shallow features, right? So, is it fair to expect a model to prefer a more complex, latent hypothesis when a simpler, superficial one fits the provided data perfectly?\n\n\n3) A few baselines are missing (a) if you specifically tell the models you are looking for shallow features or deep features (b) doing (a) but allow for some “reasoning“or chain-of-thought (c) the human baseline values of DVGR; I think these results would allow us to put your results into more context. \n\n\n4) Conceptually, to what extent are shallow preferences and deep values correlated? I mean, I am sure there is quite a bit of (anti)correlation between “Preferring AI interactions that use first names and casual addresses.” and “tradition”. So is it even possible to disentangle shallow preferences and deep values at all? \n\n\n5) I am not sure if the evidence is enough regarding the scaling claims. Other than the llama models, we have no clue about the size of the models from API providers. Additionally, I believe to say something credible about scaling, the analysis should include at least multiple model sizes, within 2-3 different model families. The current data is just too sparse to say much about scaling trends.\n\nQUESTIONS:\nHow did you run the llama models? Just on CPUs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on an important question: whether LLMs have (or can learn) stable values beyond superficial preference patterns. To answer this questions, the authors propose a novel evaluation framework grounded in a “confound-then-deconfound” schema, to assess LLMs’ ability to capture the underlying value preference instead of shallow preference. Besides, the authors construct a large-scale binary choice benchmark, covering 40 domains and a diverse set of real-world activities. This benchmark is further validated by comprehensive and well-designed human verification. Based on this data, the authors evaluated 9 strong and popular LLMs, and found that i) most models fail to generalize deep values, ii) scale does not help value generalization ability, iii) value generalization varies by context and value type.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper is well-motivated and generally clearly written and focuses on an important and long-standing research question: whether LLMs have stable human-like values. \n2. The authors design a novel “confound-then-deconfound” evaluation schema to assess LLMs’ ability to learn and generalize real underlying values against shallow preference.\n3. The authors constructed a large-scale binary-choice benchmark and conducted rigorous and extensive human verification to support this benchmark’s quality. This is the most rigorous and detailed work I’ve seen recently in terms of manually validating the quality of a dataset.\n4. The authors conducted comprehensive experiments, and provided several insightful findings.\n\n**Weaknesses**\n\nMy biggest concern is whether this task (confound-then-deconfound) is resolvable (or whether it’s reasonable to require LLMs to solve it)? There are two reasons for this concern.\n\n1. The low DVGR might be caused by the lack of knowledge in LLMs.  Humans can solve this task successfully because of our prior knowledge (i.e., our commonsense that value is more crucial than other surface patterns like formal/information information). Besides, in the two validation tasks in Sec.4.2, humans obtain additional (and thus unfair) information compared to LLMs. For example, in validation 1 “Participants received explicit information about a user’s value preference” but LLMs do not (correct me if I’m wrong). This means the proposed benchmark actually measures only LLMs’ knowledge.\n\n2. The low DVGR might also be because that shallow preference could be one correct answer. Without additional information, such preference, e.g., adaptability in Fig.1, could also be one of the values, as no one knows the real thoughts of a specific user. \nThese issues are also supported by the authors’ observations. In line 307, “We find models generalize values they perceive as unpopular”. This is just because without information/knowledge, LLMs can only follow their own internal bias to make the choice (so do humans, but we regard humans’ choice as the ground-truth). \n\nTherefore, the authors should also conduct validation 1 and 2 on LLMs, and compare the results of humans and models. More concretely, before applying the confound-then-deconfound, I suggest that the authors first provide the same information to LLMs as those to humans, or verify that LLMs already have such knowledge.\n\nGenerally, I this is a novel and good paper. If my concern above is addressed, I will raise my score.\n\n**Missing Reference**:\n* Ren et al., ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models. ACL 2024.\n* Yao et al., Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values. ACL 2025.\n* Mazeika et al., Utility Engineering: Analyzing and Controlling Emergent Value Systems in Ais. Arxiv 2025.\n\nQUESTIONS:\n1. In line 168, why is the standard deviation so large? Does this imply that shallowness is not entirely consistent across contexts?\n\n2. In the few-shot examples given to the LLM, are there multiple tuples that involve the same pair of values (vᵢ, vⱼ) but different preference ? For instance, ($v_i$, $v_j$, $s_{k_1}$, $s_{k_2}$) versus ($v_i, v_j, s_{l_1}, s_{l_2}$)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper introduces a new benchmark for assessing LLMs' ability to generalize deep values. To this end, they produce synthetic scenarios in which deep values v and shallow preferences s are confounded. The LLMs are then presented with several in-context examples in which the user rates scenario (v1, s1) as better than (v2, s2). At inference-time, the model is presented with scenarios (v1,s2) and (v2, s1) and is supposed to prefer the former (corresponding to the deep value).
Several model families and model sizes are tested on this benchmark. The main result is that all models perform below chance, consistently preferring the scenarios corresponding to the shallow preference.

STRENGTHS AND WEAKNESSES:
Strengths:
* extremely clearly written
* thorough related work
* clearly relevant to the NeurIPS crowd
* great experimental design and execution
* valuable, novel insights
* thorough analysis

Weaknesses:
* the paper does not test different prompting methods, e.g. what happens when the model is explicitly told to look for deep values? What happens with chain-of-thought prompting?

QUESTIONS:
* Shallow preferences are described as something that you may choose in the moment (line 107-108). The prompt in Figure 2 doesn't specify whether the user choices are decisions based on deep values or on shallow preferences. Do you think that the models should have learned to look for deep values in this case? If so, do you think adjusting the prompt would change the results?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-04