What Do Large Language Models Know About Opinions?
Authors: Erfan Jahanparast
Zhiqing Hong
Serina Chang
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
opinions
computational social science
interpretability
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: LLMs encode rich internal representations of human opinions across demographics and topics, far exceeding what their next-token outputs reveal.

Abstract: What large language models (LLMs) know about human opinions has important implications for aligning LLMs with human values, simulating humans with LLMs, and understanding what LLMs learn during training.  While prior works have tested LLMs' knowledge of opinions via their next-token outputs, we present the first study to probe LLMs' internal knowledge of opinions, evaluating LLMs across 22 demographic groups on a wide range of topics.  First, we show that LLMs' internal knowledge of opinions far exceeds what is revealed by their outputs, with a 52-66\% improvement in alignment with the human answer distribution; this improvement is competitive with fine-tuning but nearly 300$\times$ less computationally expensive.  Second, we find that knowledge of opinions emerges rapidly in the middle layers of the LLM and identify the final unembeddings as the source of the discrepancy between internal knowledge and outputs.  Third, using sparse autoencoders, we trace the knowledge of opinions in the LLM's residual stream back to attention heads, and we identify specific attention head features that selectively encode different demographic groups. Through steerability experiments, we show that manipulating these features causally alters the LLM's outputs, aligning them more or less closely with different groups. These findings open new avenues for building value-aligned and computationally efficient LLMs, with applications in survey research, social simulation, and human-centered AI.  Our code is available at https://github.com/schang-lab/llm-opinions.

Published as a conference paper at ICLR 2026
WHAT Do LARGE LANGUAGE MODELS KNOW ABOUT OPINIONS?
Erfan Jahanparast, Zhiqing Hong & Serina Chang University of California, Berkeley erfanjp, zhiqing, serine werinacbberkeley................
ABSTRACT
What large language models (LLMs) know about human opinions has impor- tant implications for aligning LLMs with human values, simulating humans with L.Ms, and understanding what L.MS learn during training. While prior works have tested L.LMs knowledge of opinions via their next-token outputs, we present the first study to probe LLMs' internal knowledge of opinions, evaluating LLMs across 22 demographic groups LLMs' internal knowledge of opinions far exceeds what is revealed by their out- puts, with 52-66% improvement in alignment with the human answer distri- bution; this improvement is competitive with fine-tuning but nearly 300 less computationally expensive. Second, find that knowledge of opinions emerges rapidly in the middle layers of the LLM and identify the final unembeddings as the source of the discrepancy between internal knowledge and outputs. Third, using sparse autoencoders, we trace the knowledge of opinions in the LLM's residual stream back attention heads, and we identify specific attention head features that selectively encode different demographic groups. Through steer- ability experiments, we show that manipulating these features causally alters the LLM's outputs, aligning them more or less closely with different groups. These findings open new avenues for building value-aligned and computationally effi- cient LLMs, with applications in survey research, social simulation, and human- centered AI. Our code is available at /ggtthu.... com/ schang- -abb 11m-op n onns
wide range of topics. First, we show that
INTRODUCTION
Human opinions diverse and complex, reflecting how people feel about topics ranging from social issues to healthcare to foreign relations (Chu et al. 2023 Hu et al. 2023). this work, we seek to understand: what do large language models (LLMs) know about human opinions?
This question has several important implications. First, as LLMs increasingly interact with and act on behalf of humans, there is urgent need for pluralistic Al systems that serve people with di- verse values and perspectives (Sorensen et al.) [024] Kirk et al. 2024 Feng et al., 2024). To achieve pluralistic alinment-whhelee aiming for responses that steer towards particular group or cover a spectrum of opinions-LLMs first need to know what different people's 's opinions are on different topics. Second, there has been great interest from survey researchers
developing LLM "syn- thetic respondents" for public opinion surveys, which require LLMs to have deep understanding of human opinions (Argyle al. 2023 Liu et al. 2025 Rothschild et al. 2024). Third, investigat- ing LLMs' knowledge of opinions helps us better understand what they learn during training: for example, prior work has shown that L.LMs learn concepts such as truthfulness (Marks & Tegmark, 2024), space and time (Gurnee & Tegmark 2024), and the cberra-conservative spectrum (Kim al. 2025 In contrast to these relatively simpler concepts, which fall along one or two dimensions, hu- man opinions far more multidimensional. Yet remains unknown how well LLMs have learned such complex concepts or how they are encoded.
Despite many recent efforts to evaluate LLMs' knowledge of opinions, they have all relied on LLMs' outputs, either using the next-token probabilities over answer options or sampled generations (San- turkar et al. 2023
(wang al. 2023 Chu et al. 2023 Durmus al. 2023 Feng al. 2024
Moon

---

Published as a conference paper at ICLR 2026
Transfomnnr Layers
Trainable Probe
Predicted Probabilbiies
Respondents
Pouruul
A. Bepublin lan
Layer/ /-
/
Tokeres Eintedding
D. Somiming sile
Independert
Ao
A
Low KL
do
Arowe:
Unembellligg
iutupiity commitanes in gan
How you trink family
x' x + av
MLP
A. lce
Wigh KL. Human Opinion Distribution
B. Aime
More
Nest-token Probabituies
(0)
(b)
(8)
Figure 1: Illustration of our framework. (a) is an example of public opinion-based prompts. (b) is the transformer layers of LLMs. (c) two ways to extract LLMs' knowledge, i.e., trainable probes (probing) and next-token probabilities (prompting). (d) is the opinion distribution collected from respondents. The lower KL of trainable probe compared to next-token probabilities demonstrates that LLMs know more about opinion than what we observe from their outputs.
et al. 2024; Park et al., 2024a Suh et al., 2025: Meister et al. 2025 Cao et al. 2025). However, other work primarily focused on the truthfulness of LLM statements has shown that LLMs' in- ternal activations sometimes contain more knowledge than is expressed in their outputs (Burns et al., 2022
Li al. 2023 Azaria & Mitchell, 2023).
The present work. Here, open up the model and ask, what do LLMs really know about opin- ions and where is this knowledge stored? Using two US US-representative public opinion datasets, OpinionQA (Santurkar et al. 2023) and SubPOP (Suh at. 2025), we systematically prompt LLMs (Llama-3.1 1-888, Mistral 7B, and Vieuna 7B) with 7,062 prompts corresponding to the product of 22 demographic groups and 321 questions. Instead of only extracting the LLM's response with its next-token probabilities, we also extract its residual stream activations in every internal LLM layer (Figure D. We then train multinomial probes to predict each group's 's answer distribution to each question using its residual stream activations, and use Kulbbeck-Leibler (KL) divergence to quantify the distance between the LLM and human response distributions, where a smaller KL divergence indicates greater knowledge of opinions.
First, we show that L.MSs know far more about opinions than their outputs reveal. Compared to next-token probabilities, the average KL. divergence from the best-performing prohe (best over lay- ers, chosen based on validation questions) is 59.7% lower on held-out test questions (Figure 2 This result is consistent across all three LLMs we evaluate, linear versus non-linear probes, and differ- ent prompting formats. Furthermore, this size of improvement reaches 85% of the improvement achieved by fine-tuning (LoRA) ) the LLM on the same dataset that the probes were trained on, while training the probe is orders of magnitude cheaper than fine-tuning, with 278x fewer parameters.
Second, investigate the origin of knowledge discrepancy between probing and next-token proba- bilities by examining probe performance layer by layer. We find that the knowledge is gained rapidly in the middle layers, with little gain in information after layer 15, but without loss of information as well. Instead, the entire discrepancy is explained by the model unembeddings, which are applied the final residual stream to compute next-token probabilities. To demonstrate this, we show that fine-tuning only the unembeddings achieves comparable performance to probes, with 46-62% improvement KL divergence over next-token probabilities.
Third, we trace where the knowledge of opinions in the residual stream comes from, focusing on attention heads as subspaces where features related to opinions may be encoded. We train sparse au- toencoders (SAEs) on attention head activations and find that each demographic group corresponds to clear SAE features that selectively fire only when that group is mentioned in the prompt. Through steerability experiments, we show that manipulating these features per demographic group aligns the model more or less closely with that group, demonstrating that the features are not only correlational but causal. Overall, our study reveals how LLMs encode knowledge of human opinions, contribut- ing to a deeper understanding of their internal mechanisms and enabling broad applications, ranging from more interpretable and human-centered AI new tools in computational social science.
2

---

Published as a conference paper at ICLR 2026
2
TRAINING PROBES TO PREDICT OPINIONS
2.1 BACKGROUND
LLM architecture. We begin by providing a brief overview of the LLM transformer architecture, using notation similar to Kim et al. (2025). LLMs consist of stacked transformer blocks, where each transformer block modifies the LLM's "residual stream" X; in layer & with the attention heads and multi-layer perceptron (MLP) in that layer (Elhage et al. 2021):
Desage + MLP, X + Σ Queacn)
X
(1)
h=1
h=1
Here, ag& refers to the activation of the h-th attention head in layer e:
A.
(2)
Weight matrices Pe. RdexxD and QE Dxdex map between the D-dimensional space of the residual stream and the de, -ddimensionnl space the attention head. Typically, dy = is consistent over heads and layers and much smaller than D; for example, in the Llama-3. I-BB model, d is 128 and D is 4096.
The initial residual stream X is the embedding of the input tokens, Wet W1, W-1- After the forward pass through all the transformer layers, the final residual stream x is mapped back into tokens via the unembedding matrix. The log probability that the next token wy is proportional to the inner product of the unembedding of v, uy and the final residual stream:
P(w, = w/ & exp(u, XL)-
(3)
Probing. Probing is a standard approach for studying the information encoded in a model's in- ternal representations (Alain & Bengio 2016; [eelinkov| (022). Given dataset of inputs paired with target labels for concept of interest, the inputs are passed through the model and the resulting activations are extracted. A lightweight model, probe, is then trained to predict the labels from these activations. Probing enables us to test how much knowledge is encoded in the model and to identify which concepts are captured, based careful selection of inputs and labels.
Typically, probes are either trained on the LLM's residual stream or attention head activations. The LLM's residual stream can be seen the model's memory, reflecting everything the LLM has processed up to that layer, while the attention heads (with much smaller dimensionality) capture specific features that should be attended to and edit the residual stream accordingly (Elhage al.) 2021). In our work, since our primary question is how much the LLM knows about opinions, we start by probing the residual stream, training one probe per layer. Later in Section 4 we turn to the attention heads to decompose this knowledge about opinions into finer grained features, such as ones representing each demographic groups.
A central claim in LLM literature is the linear representation hypothesis, which is the belief that high-level concepts are represented linearly in the LLM's representation space (Park al. 2024bb Marks & Tegmark] 2023 Gurnee & Tegmark, 2023). A corollary of this hypothesis is that linear probes suffice for predicting concepts from the model's s activations and that concepts are not more accurately predicted by more flexible non-linear probes. To test this hypothesis, we evaluate both linear and non- non-linear probes, specifically multinomial logistic regression and an MLP.
2.2
OUR PROBING FRAMEWORK
In this work, we interested in the LLM's knowledge of group-specific opinions: given group and question q. with the associated set of answer options A(q). what is the group's distribution of answers over Alg)? To test the LLM's knowledge, construct a prompt Pag that first conditions the model on group g. then asks question & (Figure Tpp. Prior work using LLMs to predict group- specific opinions has used the same prompting format, but they would extract an answer using the next-token probabilities P(wy) of the LLM, for the tokens corresponding to the letters of the answer options: w, "a". "b". "c". etc (Santurkar al. 2023 (Sh al. 2025).
We replicate this approach, while also extracting the LLM's residual stream activations from every layer when the LLM is given this prompt (Figure 1p). For a group g and question g, let x
3

---

Published as a conference paper at ICLR 2026
represent the residual stream activation at layer l when the LLM is given the prompt for this group and question, and let Tgg represent the group's 's real answer distribution over answer options Ag. We
train a probe learn the function ft -RD RK, mapping from x; (99) to Tgg for all groups and all questions with K answer options. Following the linear representation hypothesis, we start with multinomial logistic regression model, which is a generalized linear model that assumes the log probability of choosing option & has a linear relationship with the input (in this case, the residual stream activation).
exp(Or. x(
gg []] EKERK] exp(O% (00)))
(4)
where we use ]:] to index the k-th option in Ā. The learnable parameters in this prohe are fg E RD per option &, resulting in only D x K parameters, with = 4096 and K = 2 or 3. To put the linear representation hypothesis to the test, we also try a non-linear probe, an MLP consisting of 256-dimensional linear layer followed by a non non-linear ReLU activation.
2.3 EXPERIMENTS
Public opinion datasets. We focus on two datasets: OpinionQA (Santurkar et al. 2023) and Sub- POP (Suh et al.] (025). Both datasets draw from the American Trends Panel (ATP), a nationally representative survey conducted in the US by the Pew Research Center. ATP consists of waves, where each wave is a survey conducted at a certain timepoint covering one or two topics. Opin- ionQA includes waves and SubPOP includes 53 waves, spanning a wide range of topics such as voter attitudes, news pathways, social media, economic and financial outlook, coronavirus impacts, technology companies and policy issues, online dating, and more (see Table 4 in [Sh et al. (2025) for a full list). Both datasets curate the data into the format of (group, question) pairs, providing the group, question text and answer options, and the group's answer distribution for that question. In our experiments, we study 22 demographic groups (e.g., Female) belonging 8 demographic attributes (e.g., Sex) (see Table While we focus mainly on the OpinionQA and SubPOP datasets in this work keep the set of groups consistent, we also conduct probing experiments on GlobalOp- inionQA (Durmus et al. 2023), which contains country-level responses to cross- natsonational surveys, and show that our main probing results extend to non-US contexts (Appendix A.2
Training and evaluating probes. We train probe for K = 2 (with 603 questions) and K 3 (with 679 questions), for each layer l of the LLM (L = 32 in all models we evaluate). For each value of K, we split all questions with K answer options into train (70%), validation (10%), and test (20%). We train the probe using the input-label pairs (x("9), (gg)) for each of the 22 groups and all training questions. Training is performed with stochastic gradient descent (SGD) with mo- mentum, using a batch size of 32, a learning rate of 6e-4, and an &g loss. We use the validation questions to tune probe hyperparameters, while we hold out the test questions for evaluation and comparison to next-token probabilities P(wx). Similar to previous studies, we use Kullback-Leibh divergence (Suh al. 20225) quantify the distance between the human distribution Tg and the LLM's predicted distribution agg (whether using the probe or next-token probabilities):
Too [k]
KL Agg)
Tan [k] log
(5)
Ag k
3 WHAT LLMS KNOW ABOUT OPINIONS
3.1 INTERNAL KNOWLEDGE VS. OUTPUTS
We first evaluate how much LLMS really know about human opinions. We select the berf-perforning probes (over all layers) based on the validation set, and evaluate their performance on held-out test questions. These probes substantially outperform next-token probabilities (prompting) at predicting human distributions, revealing that LLMs encode far more about opinions than their outputs suggest. As shown in Figure 2 on average over the 22 demographic groups. we see that KL divergence decreases by 66%, 67% and 82% for .lama-3.1-885 Mistral-7B, and Vicuna-7B for questions with
4

---

Published as a conference paper at ICLR 2026
Prompting Probing
Promptinn
0.5
0.5
Probing
Divergence
0.375
KL. Divergence
0.383
0.3
0.241
0.233
0211
0.200
0.2
0.2
0.113
0.110
0.104
0.1
00033
0.067
0.063
0.1
0.0
LLaMA-8B
Mistral-7B
Vicuna-7B
0.0
LLaMA-88 8B
Mistral-7B
Vicuna-7B
(a) 2-option questions
(b) 3-option questions
Figure Comparing results from prompting (i.e., next-token probabilities) vs. probing.
K 2 options (Figure left), and by 52%, 50%, 71% for the same models, respectively, on questions with K = 3 options (Figure 2 right). We provide results per demographic group for all models in the Appendix (Tables 3 annd
Second, we test the linear representation hypothesis (Park et al. 2024b) by comparing the perfor- mance of the logistic regression vs. MLP probes. Despite the greater expressivity of MLPs, we find that logistic regression probes achieve comparable performance across all layers, models. and 22 groups (Tables 5 and with an average difference KI. of 0.005. While this result consistent with the linear representation hypothesis, we caution below about what we believe is actually being encoded as the linear concept. Third, we test the sensitivity of our results the different prompting formats. In addition to our default QA format, which conditions the model on the group in multiple choice question- iueettin-ansserrin answering format (Figure Th), we also try two other prompting formats used in prior work (Santurkar al., 2023 |uh et al. (0255: PORTRAY, which instructs the model to answer if it is a person from that group, and BIO, which describes the group as a first-person biography (see example prompts in Table 2 As shown in Tables and X results consistent over different prompt formats, where we continue to see large improvements from prompting to probes.
How can opinions be linear? While opinions span 22 diverse groups, is important distinguish between opinions as an answer distribution vs. as the features involved in building them. By the most predictive layer, the residual stream encodes the answer distribution itself-a simpler object that can be represented linearly-whir which is precisely what the probe is trained to predict, explaining why logistic regression matches the ML.. Consistent with this, a single shared probe performs nearly identically to per-group probes (Tables 10).
there a consistent representation answer distribution that emerges in the residual stream after all group and question information is added in, we should see that a single probe for all groups suffices. This what we see for Llama-3. 1-BB where we find that the probe for all groups performs almost exactly the same as the probe per group, with an average KL difference of 0.003 for 2-option and 0.006 for 3-option questions (Tables TO).
Comparison to fine-tuning. Prior work showed that fine-tuning LLMs on public opinion survey data achieves large improvements in predicting opinions on unseen questions (Suh et al. 2025 Cao al. 2025). We are interested in how much of that improvement can be achieved through prob- ing alone, only training the relatively few parameters of the probe and leaving the LLMs' weights untouched. We follow the fine-tuning framework from [Sh et al. (2025), where the authors fine- tuned LLMs in the (group, question) format from SubPOP, minimizing the KL divergence between the LLM's next-token probabilities and the ground truth human distribution xgg. We use the same LoRA fine-tuning setup as the authors, with the same hyperparameters, and fine-tune Llama-3.1-8B on the same train questions that we used to train our probes.
Consistent with prior work, find that fine-tuning greatly improves the LLM's ability to predict group-specife opinions on held-out questions (Figure However, probing still achieves over 85% of that improvement, while being x more parameter efficient than L.RA. As a practical matter,
5

---

Published as a conference paper at ICLR 2026
Ubeerhdding
51 Inggrraaeentt (2)
00
31 Improresccnt (75)
64
20
20
Arten
White
Democrat
Republicn
1Rack
Whute
Densecrat
Republicaa
(a) 2-option questions
(b) 3-option questions
Figure 3: Comparing improvements over prompting from probing, fine-tuning only the unembed- dings, and LoRA fine-tuning.
these results reveal that probes could serve as an effective lightweight alternative to fine-tuning. Furthermore, since probes do not teach the LLM new knowledge they can only use knowledge that is already in the LLM-these results suggest that majority of the fine-tuning improvement is attributable to the LLM learning how to better retrieve and format knowledge that already had, opposed to learning knowledge, providing evidence for the "Less is More for Alignment" (LIMA) hypothesis (Zhou et al. 2023) in a new domain.
3.2 WHERE KNOWLEDGE IS GAINED AND LOST
Knowledge gains the first half. Our second question where knowledge about opinions is encoded in the LLM. When examine the layer-by-layer performance of the probes (Figure 4 we see that knowledge is gained in the first half of the model and often with rapid gains in the middle layers (10 to 15), most noticeably for -lama-3.1-8B. This result that opinions are learned in the middle layers contributes to a growing literature exploring where different concepts are located the model. Other concepts, such as political perspectives (Kim al. (025) and sentiment (?), have also been shown to emerge in the middle layers, while simpler concepts, such as space and time (Gurnee) Tegmark 2024), emerge earlier layers. Furthermore, the rapid gain in knowledge in the residual stream provides clues for which attention heads might be most responsible for adding features about opinions into the residual stream. In Section 4 look for relevant features attention heads over all layers, but find that the clearest features corresponding to different demographic groups appear these middle layers.
Unembeddings the source of discrepancy. After layer 15, probe performance does not im- prove, but it also does not sigmificantly worsen (Figure 4. Probes trained on the final residual x (L 32) perform as well the best probes overall, so the gap between probing and prompting cannot be explained by loss of information in the residual stream. The only component between X and next-token probabilities the unembedding layer, a linear mapping back to the vocabulary 3). meaning that the discrepancy from probe prompting arises at the unembedding stage. We show this empirically by fine-tuning only the unembeddings of Llama-3. 1-8BB on the same dataset used for training the probes and LoRA fine-tuning. We find that only fine-tuning the unembeddings solves the discrepancy between probing and next-token probabilities. Compared to the next-token probabilities before fine-tuning, the next-token probabilities after fine-tuning the unembeddings have a 61. 1.8% reduction in KI. divergence on 2-option questions and a 46.4% reduction in KI. divergence on 3-option questions (Tables 11] 12 These improvements we saw from probing (Figure 3p. and they also mean that fine-tuning only the final unembeddings achieve 75-78% of the improvement from LoRA fine-tuning, providing another lightweight, targeted alternative to fine-tuning the entire model.
comparable to the improvements
4 TRACING KNOWLEDGE BACK TO ATTENTION HEAD FEATURES
So far, have shown that LLMs contain substantial knowledge about opinions in their residual streams. However, what features is the LLM using to arrive at this knowledge? To answer this, we turn the attention heads, which attend to features and edit the residual stream (Elhage al. 2021)

---

Published as a conference paper at ICLR 2026
011
6638
Back
0.2
Whil
Dancorl Tepenkee
0.2
005
0.2
(a) LLaMA 8B 2 options
(b) LLaMA 8B 3 options
Bank
0.2
Bank
0.2-
Ripudtio son
Dipuot can
230
316
Layer
Lover
Mistral options
(d) Mistral 7B 3 options
2.38
Hipprii
Whits
334
Democrs
Dimoors
336
232
2.00
(e) Vicuna 7B 2 options
(f) Vicuna 7B 3 options
Figure 4: Probe performance per layer for LLaMA (a,b), Mistral (c,d), and Vicuna (c.f) across
K (left) and K = 3 (right) option questions.
4.1 IDENTIFYING FEATURES WITH SAES
Background SAEs. To identify features in the attention heads, we use sparse autoencoders (SAEs), which are popular method for decomposing neural network activations into sparse, in- terpretable features (Cunningham et al.) 2024 Kissane al. [0244] Movva al.) 2025 Gao al., 2025). We use the k-sparse autoencoder. which directly controls the number of active features in the SAE representation using a top-k activation function (Makhzani activations Dia the SAE learns decomposition of into MSAE features, where only the top KSAE features are kept. Using the notation of Movva et al. (2025), the SAE first learns a sparse representation z, E B.Mss of a,:
Frey 2013). Given input
IR
2, = (eL.U(Top((een.aa - bpre) + bene).
(6)
with the encoder Waxe IR MSXE Du, bias terms bpec bene E Ru,,, and TopK sets all activations except the top KSAF to zero. Then, the decoder applied to z; to try to reconstruct a,: â, = Wae:?; + bate; with the decoder Wdec E I DR. Maan and bias term baec E Br. The SAE's loss on single input the reconstruction loss: LSAE = ||a à
(99) Training our SAES. Let a (as defined in Eq. represent the activation of the h-th attention head in layer of the model when it is given the prompt for group g and question g. Following Kissane et al. (024)) we train one SAE per model layer on the concatenated activations from all attention heads in that layer, before they converted to attention outputs through linear transfor- mation. We focus on the Llama-3.1 1-B model for these experiments. We train SAEs on activations extracted at the demographic token position the token corresponding to the demographic group

---

Published as a conference paper at ICLR 2026
Feme
Sup
110MP
Figure 5: Maximum Fl value achieved by SAE features per demographic group and layer using SAES with MSAE = 2048 and KSAE = 100.
in the prompt- as opposed to the final token position; we further discuss this in Section A.3 Since the SAE is not tied the number of answer options, we are able to use the full set of 69,042
prompts from SubPOP's train dataset, with answer options randomly shuffled to prevent spurious correlations with specific option letters. We split this dataset into 85% for training and 15% for val-
idation. For each model layer f, we train a separate SAE on activations (99) aggregated across all groups and questions the training set. To assess model quality, we use standard metrics evaluated on the validation set: reconstruction loss, the percentage of dead features (those that never fire), explained variance (1 - Var(a a) ()/Var(a)), and mean fire count among alive features. The results reported throughout are from SAES with MSAE 2048 and KSAE 100, chosen based holistic evaluation of these metrics across SAE sizes and token positions (Table 15)
Features per demographic group. After learning the SAE features, seek to identify the fea- tures that correspond most strongly to different demographic groups. For each demographic group, compute the F1 score of each SAE feature, where prompts that condition
that group serve as the positive class and all other prompts as negatives. In Figure 5 we plot, for each group (y-axis) and model layer (x-axis), the maximum Fl score achieved by SAE features for that group and layer. The pattern is consistent: SAEs trained on middle layers (10-16) recover the largest number of highly group-selective features, with peak Fl scores close 1.0. SAEs trained on earlier layers also identify features related to demographic groups, but their selectivity is weaker and less consistent. Beyond the middle layers, predictive features become sparse and rarely align with specific groups. Interpreted in terms of the LLM itself, this suggests that demographic group information is first reg- istered early, sharpened into more explicit representations in the middle layers, and largely absent in deeper layers. This layerwise trajectory is consistent with our layer-by-layer probing results on the residual stream (Figure 4), which similarly found that knowledge of opinions emerges rapidly in the middle layers of the LLM.
4.2 STEERING TOWARDS DIFFERENT GROUPS
SAE-based steering methods. In this final analysis, we test whether the most predictive SAE features are not only correlated with their respective groups, but also causally shift the model's output towards or away from that group. To test this, we design the following procedure, where we intervene the model by systematically turning SAE features and off.
1. First, we take base group 9i (e.g., South) and source group f2 (e.g., Northeast) from the same attribute (e.g., Region). For given question g, we compute the model's predicted answer distri- butions uu and ae when given the prompts Poiq and Pazq- respectively.
For given group, we define a "top" feature as one that satisfies three criteria: (i) F1 > 0.7 with respect to that group; (ii) within-atrribtte selectivity: the feature's firing rate for this group is at
8

---

Published as a conference paper at ICLR 2026
Incoom Heggon pion
0.
Aug Agggnngate tar pitventee Patto
06
6-422
CLSC
80
yoher
Juan stiee
Figure 6: Steering effectivvnnes across pairs of demographic groups. We report the average im- provement ratio, which quantifies the proportion of the original source-to-base distance closed by our intervention.
least 10x the average firing rate for sibling groups within the same attribute, (iii) cross-attribute selectivity: the feature's firing rate for this group least 10x the average firing rate for groups outside of this attribute. A group is included in our steerability analysis only if it has at least 5 top features total over the layers where we are applying the intervention.
3. We attempt shift the model from the base group II towards the source group 92- Specif- ically, for some layer L, we extract the SAE's representation Z1 and Z2 of the attention head activations when prompted with Pol9 and Poug respectively. Then, we construct modified SAE representation (1-22 that is a copy 2 except the top features for group 9i are turned off (i.e., replaced with 0's) and each top feature for group f/ is replaced with z2 [t] 5, where 5 >> 0 is some multiplicative factor. This intervention applied across layers from lxan to lends where Lstats fend E {0, (0.....31}, 31).
Then, use the SAE decoder to reconstruct activations in layer from (1-+2), and run the model forward from É with the reconstructed activations and compute the model's new predicted answer distributions, #(-+2), As our evaluation metric, we compare the dis- tance between the source and base distributions, KL # and between the source and intervened distributions, (L ( #(1-2)). Our improvement ratio is therefore KL (11=21) (//22)//KK((ff. (L #)))), i.e., the proportion of the original source-to-base distance closed by the intervention. We sweep over scaling factors E (5, 10, 15, 25} and select the best per demographic group on fixed 20% tuning split of question keys, stratified across 2- and 3-option questions. All results reported on the held-out 80% evaluation split.
Notably, SAE approach enables us identify features for all 22 demographic groups, in contrast to common steering vector approaches which operate on a single direction between two opposing groups and thus do not naturally extend settings with more than two groups per attribute (Subra- mani et al. 2022 Turner el al. 2023 Rimsky et al.) 2024).
Steerability results. As shown in Figure d and Table 16 SAE-based logit steering produces substantial shifts in model behavior. Across the 17 steering directions (layers 9-18), the mean Avg. Improvement is 0.440, with the strongest effects for education and income transitions (e.g., < HS + College+:: 0.942; $30k + $100k+: 0.893), which consistently exceed 0.85. Steering effectiveness varies by attribute type: socioeconomic directions are most amenable, while political ideology directions did not meet the feature selectivity criteria for inclusion. We also observe di- rectional asymmetries within attributes, e.g., Protestant - Atheist (0.633) vs. Atheist - Protestant (0.372), which correlate with the number of high-F1 features available for each group. A layer-

---

Published as a conference paper at ICLR 2026
range ablation further confirms that intervening on layers 9-18 alone achieves within 5% of the full layers 2-30 intervention (0.440 vs. 0.463), consistent with the concentration of demographic- discriminative features in middle layers observed in both the probe accuracy and Fl analyses. To- gether, these results demonstrate that SAF-identified group features are not merely predictive corre- lates but actionable, localized levers for targeted behavioral steering.
5 RELATED WORK
Public opinions. A growing body of work has examined whether L.MMs can predict public opin-
ions, but these studies evaluate only model outputs (Santurkar et al. [023 Hwang et al.) 2023
Chu
cct al.
2023
Durmus et al. 2023; Moon et al. 2024; Park et al.] 2024a Suh et al. [025] Meister
eet al.
2025 Cao et al., 2025 To the best of our knowledge, we are the first to evaluate LLMs'
internal knowledge of opinions and to discover the size of the disparity between LLM outputs vs. probes. Our work also contrasts with efforts to improve the model's prediction of group-specific opinions by providing external information, such as few-shot examples (Zhao et al.) 2024a) or fine- tuning (Suh et al. 2025 Cao et al.] 2025), since our approach seeks to make more out of what is already contained in the model, instead of providing additional information. The closest work to ours probes LLMs on political perspectives, such as the liberal-conservative spectrum (Kim al., 2025 Hu et al. 2025 Ball et al. 2025), but they do not compare probing to prompting or to fine- tuning. Furthermore, we go far beyond political opinions in our work, studying 22 demographic groups, including non-political variables such as gender, race, and religion that do not fall along single ideological dimension, along with non-political question topics, such as social identity, edu- cation, and cultural preferences. This broader setting requires more demanding evaluation, since group identities and topics cannot be reduced to a single dimension.
Interpretability. Methodologically, prior work has typically used linear probes to test a scalar value along one or two dimensions (c.g., truth vs. false, liberal vs. conservative) (Burns et al. 2022 Marks & Tegmark 2024 Gurnee & Tegmark 2024; Kim et al. 2025). In contrast, our work involves 22 different groups, and we test the model's s ability to predict opinion distributions, which requires multinomial probe and distributional metrics, such as KL divergence, to measure model fidelity. Furthermore, common methods for steerability, such as steering vectors, typically rely on a binary contrast (Subramani et al., 2022; Turner al., 2023 Rimsky al.) (024). An advantage of our approach is that we can train the SAE once, then steer the model towards any of the 22 groups, by identifying and magnifying the SAE features corresponding to each group. Our layer-wise analysis of where knowledge emerges in the model builds on prior work finding that simpler features emerge in earlier layers (Gurnee 'Tggmark| 2024) while more complex features emerge in middle layers (3Kim et al.) (025). and complex processes like multilingualism evolve in the model's functions throughout layers (Zhao al., 2024b).
DISCUSSION
Our study demonstrates that LLMs encode far more internal knowledge about opinions than their surface-level outputs suggest. This finding highlights a growing distinction in interpretability re- search: while probes reveal what models know, outputs reflect only what models are willing to use. Such gaps are crucial for both LLM research and computational social science applications. Our results also suggest practical directions. For instance, domain-specin applications may benefit from fine-tuning only the final layers rather than the entire model. significantly reducing computa- tional and energy costs. Similarly, by identifying where opinion-related knowledge is concentrated in model layers, we may prune irrelevant parameters and design more efficient L.LMss
Despite these advances, our framework has some limitations. First, while we study opinions in US and non-US contexts, we only study English prompts. Second, here we focus on reported opinions from public opinion surveys, following prior work studying opinions (Santurkar et al. 2023 Durmus et al. 2023 Hwang et al. 2002). but public opinion surveys may not capture all underlying human opinions, such as opinions personal issues or opinions that cannot be expressed in multiple choice answers. Future work could explore how robustly our approach generalizes across cultural and linguistic contexts, to personal opinions, and to open-ended generation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The authors thank Ali Shirali, Joseph Suh, and Emma Pierson for helpful comments and the UC Berkeley Center for Human Compatible Al for compute resources. This work is supported in part by the Google Research Scholar Program.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 4rXiv:1610.01644, 2016.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models simulate human samples. Political Anal- ysis, 3133)3337-35,, 2023.
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
Sarah Ball, Simeon Allmendinger, Frauke Kreuter, and Niklas Kühl. Human preferences in large language model latent space: technical analysis on the reliability of synthetic data in voting outcome prediction, 2025.
Yonatan Belinkov.
Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.03887, 2022.
Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. Specializing large language models to simulate survey response distributions for global popula- tions. arXiv preprint wrXiv:2502.07068 2025.
Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. Language models trained on media diets can predict public opinion. arXiv preprint 2rX1v:2303.16779 2023.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. In ICLR, 2024.
Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Saffield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards mea- suring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306. 16388, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits, 2021. URL battps:/ttransoott wawaansooorer-irrri--eeee circuits. pub/2021/
Framework/
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-LILM collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18653/v 1/2024. ww2004.cmmlpmmai...mm https::/acLanthoo..o
URL
org/ 2024. emnlp ma
2407
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv:2310.02207, 2023.
arXiv preprint
Wes Gurnee and Max Tegmark. Language models represent space and time. In ICLR, 2024.

---

Published as a conference paper at ICLR 2026
Jingyu Hu, Mengyue Yang, Mengnan Du, and Weiru Liu. Fine- Fine-grained interpretation of political opinions in large language models, 2025.
EunJeong Hwang, Bodhisattwa Prasad Majumder, and Niket Tandon. Aligning language models to user opinions. In Findings of EMNLP, 2023.
Junsol Kim, James Evans, and Aaron Schein. Linear representations of political perspective emerge in large language models. In ICLR, 2025.
Hannah Rose Kirk, Alexander Whitefield, Paul Röuger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. In NeurIPS, 2024.
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. In- terpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406. 17759, 2024.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In NeurlPS, 2023.
Yang Janet Liu, Matthias Assenmacher, Jacob Beck, Stephanie Eckman, Frauke Kreuter, Barbara Plank, and Gina Walejko. The first workshop on bridging nlp and public opinion research (nlpor). In COLM. 2025.
Alireza Makhzani and Brendan Frey. k-sparse autoencoders. arXiv preprint arXiv: 1rXiv:1312.56660 2013.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language
model representations of true/false datasets. In CoLM. 2024.
Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. Benchmarking distributional alignment of large language models. NAACL, 2025.
Suhong Moon, Marwa Abdulhai, Minwoo Kang. Joseph Suh, Widyadewi Soedarmadji. Eran Kohen Behar, and David Chan. Virtual personas for language models via an anthology of backstories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19864-19899 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: http:/655//112222mmmllll 1110. URL http:: http:////aaaaattohooogooo wwwp:thhoolo...../22222.
1110/]
Rajiv Movva, Kenny Peng, Nikhil Garg, Jon Kleinberg, and Emma Pierson. Sparse autoencoders for hypothesis generation. In /CML.25: Proceedings of the 42nd International Conference Machine Learning, 2025.
Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109 2024a.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In ICML 24: Proceedings of the 41st International Conference on Machine Learning, 2024b.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama via contrastive activation addition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 15504-15522, Bangkok. Thailand, August 2024. Association for Computational Linguistics. doi: 18.55/// 1/2224.11o2n2
URL
828/1
David M. Rothschild, James Brand, Hope Schroeder, and Jenny Wang. Opportunities and risks of lims in survey research, 2024. Available on SSRN: http://.dd..........1////......m
12

---

Published as a conference paper at ICLR 2026
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning, pp. 29971-30000 PMLR, 2023.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A roadmap pluralistic alignment. In JCML, 2024.
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 566-581, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.28253/11/2022. mcl.nng-aall.8.. URL httpt::/p////.m. aclanthooon
/attoa//////////aatttooooooo org Findingiiiiinnn www.:/isnniinnngn...ne... acl. 48/
Joseph Suh, Erfan Jahamparast, Suhong Moon, Minwoo Kang, and Serina Chang. Language model fine-tuning on scaled survey data for predicting distributions of public opinions. In ACL, 2025.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248 2023.
Siyan Zhao, John Dang, and Aditya Grover. Group preference optimization: Few-shot alignment of large language models. In ICLR, 2024a.
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? In NeurIPS, 2024b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuming Mao, Xuezhe Ma. Avia Efratt, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. In NeurlPS, 2023.
13

---

Published as a conference paper at ICLR 2026
Table Attributes in OpinionQA and SubPOP and their corresponding demographic groups.
Race
Sex
Education
Region
Religion
Income
Political Ideology
Political
Affiliation
Group White,
Male, Female
Less Than High School,
Northeast, South
Protestant, Jewish, Muslim, Hindu, Atheist
Less than
Conservative, Moderate, Liberal
Republican,
Black,
S30k,
Democrat
Hispanic.
College
More than $100k
Asian
Grad
A APPENDIX
A.1 DETAILS FROM PROBING EXPERIMENTS
In this section, we present additional details from our probing experiments, including the 22 groups and 8 attributes included our experiments (Table D examples of different prompting formats (Table 2) comparisons of prompting probing (Tables and 4 comparisons of linear vs. MLP probes (Tables 5 and
6). prompting vs. probing results from the PORTRAY and BIO prompting formats (Tables and 8 comparing probe per group per attribute shared by all groups (Tables and 10 and comparing probing vs. LoRA fine-tuning vs. only fine-tuning the
unembedding (Tables and 12
Table 2: Examples of three different prompting formats.
Type
Prompt
QA (default)
What is highest level schooling or degrce that you have completed?
A. Less than high school
B. High school graduate
C. Some college, no degree
D. Associate's 's degree
College graduate/some postgrad
F. Postgraduate
Answer: E
How much, if at all, do you worry about the following happening you? Being the vietim of terrorist attack
Worry
B. Worry a little
C. Do not worry at all
Answer:
Bio
Below you will be asked provide short description of your current education level and answer some questions.
Description: The highest level of education have completed College graduate/some postgrad
How much, if at all, do you worry about the following happening you? Being the vietim terrorist attack
Worry lot
B. Worry a little
C. Do not worry at all
Answer:
Portray
Answer the following question the highest level of education you have completed is Less than high school
How much, if at all, do you worry about the following happening to you? Being the victim of terrorist attack
A. Worry lot
Worry litte
C. Do not worry at all
Answer:
14

---

Published as a conference paper at ICLR 2026
Table 3: Prompt vs. Probing (one row per demographic group). "% Improvement" (epeoveeemeet=(PPPpiiiiiii (Prompting
PPrompinn x 100. The results are for 2-option questions only.
LLaMA-8B
Mistral-7B
Vicuna-7B
Group
Prompt
Probe
% Impro.
Prompt
Probe
Impro.
Prompt
Probe
% Impro.
College graduate/some postgrad
0.286
0.066
76.920
0.236
0.054
77.028
0.351
0.050
85.726
Less than high school
0,190
0.049
74.553
0.173
0.045
73.905
0.367
0.047
87.282
$100,000 or more
0,188
0.072
61.903
0.296
0.066
77.751
0.446
0.054
87.823
Less than $30,000 Conservative
0,181
0.046
74.409
0.171
0.048
71.807
0,432
0.047
89.224
0.262
0.068
73.910
0.198
0.067
66.240
0.507
0.072
85.827
Liheral
0.186
0.094
49.736
0.195
0.085
56.457
0.345
0.086
75.145
Moderate
0.188
0.053
72.006
0.208
0.053
74.749
0.357
0.051
85.607
Democrat
0.211
0.087
58.936
0.222
0.098
55.860
0.383
0.084
18.135
Republican
0,294
0.087
70.260
0.276
0.069
75.
0.520
0.088
83.178
Asian
0,219
0.068
69.014
0.209
0.072
65.599
0.287
0.055
80.998
Black
0.211
0.077
63.672
0.220
0.080
63.649
0.383
0.066
82.808
Hispanic
0.201
0.049
75.665
0.197
0.051
74.207
0.290
0.042
85.608
White
0.241
0.061
74.637
0.180
0.055
69.392
0.384
0.050
86.934
Northeast
0.288
0.061
78.873
0.274
0.053
80.496
0.373
0.048
87.245
South
0,280
0.058
79.213
0.291
0.054
81.594
0.369
0.042
88.731
Protestant
0,234
0.071
69.615
0.161
0.055
65.667
0.372
0.061
83.525
Jewish
0,177
0.082
53.602
0.134
0.062
53.830
0.247
0.065
73.448
Muslim
0,235
0.120
48,820
0.215
0.115
46.613
0.257
0.095
63.039
Hindu
0,196
0.089
54.372
0.184
0.084
54.552
0.282
0.077
72.550
Atheist
0.230
0.130
43.509
0.184
0.111
39.801
0.292
0.105
64.087
Female
0.189
0.058
69.117
0.199
0.046
76.888
0.563
0.055
90.285
Male
0.219
0.058
73.590
0.220
0.043
80.269
0.452
0.046
89.749
Average
0.223
0.073
66.651
0.211
0.067
67.339
0.375
0.063
82.134
Table 4: Prompt vs. Probing (one row per demographic group). "% Improvement" (Prompting x 100. The results are for 3-option questions only.
LLaMA-8B
Mistral-7B
Vicuna-7B
Group
Prompt
Probe
% Impro.
Prompt
Probe
Impro
Prompt
Probe
% Imprro
College graduate/some postgrad
0.248
0.115
53.738
0.238
0.107
55.137
0.349
0.110
68.604
Less than high school
0.216
0.068
68.433
0.160
0.060
62.776
0.313
0.061
80.698
$100,000 or more
0.218
0.117
46.136
0.272
0.107
60.592
0.400
0.110
72.556
Less than $30,000 Conservative
0.240
0.072
70.095
0.177
0.069
61.038
0.371
0.066
82.338
0.322
0.142
56.107
0.267
0.137
48.824
0.478
0.128
73.271
Liberal
0.214
0.116
45.920
0.189
0.119
37.013
0.336
0.119
64.465
Moderate
0.204
0.087
57.396
0.170
0.088
48.180
0.382
0.081
78.880
Democrat
0.211
0.119
43.886
0.210
0.113
46.494
0.373
0.116
68.968
Republican
0.380
0.158
58.355
0.356
0.143
59.925
0.572
0.88
75.949
Asian
0.194
0.102
47.668
0.195
0.101
48.276
0.360
0.085
76.375
Black
0.216
0.093
57.133
0.210
0.090
56.960
0.346
0.081
76.597
Hispanic
0.205
0.078
61.894
0.195
0.076
60.866
0.238
0.071
70.240
White
0.256
0.113
55.898
0.213
0.104
51.326
0.435
0.106
75.519
Northeast
0,248
0.088
64.492
0.239
0.093
61.209
0.400
0.084
79.065
South
0.245
0.090
63.141
0.224
0.087
61.146
0.551
0.081
85.242
Protestant
0.269
0.108
59.896
0.207
0.107
48.290
0.384
0.104
72.924
Jewish
0.232
0.144
37.868
0.208
0.142
31.803
0.321
0.140
56.384
Muslim
0.260
0.168
35.286
0.270
0.186
31.093
0.296
0.167
43.553
Hindu
0.249
0.168
32.374
0.250
0.172
31.082
0.293
0.141
51.884
Atheist
0.240
0.162
32.625
0.204
0.148
27.568
0.349
0.139
60.039
Female
0.206
0.094
54.516
0.186
0.090
51.531
0.475
0.080
83.121
Male
0.222
0.092
58.575
0.209
0.088
57.766
0.403
0.089
77.936
Average
0.241
0.113
52.792
0.220
0.110
49.950
0.383
0.104
71.573
15

---

Published as a conference paper at ICLR 2026
Table 5: Linear MI.P probes by demographic group. Difference Linear ML.. The results are for 2-option questions only.
LLaMA-8B
Mistral-7B
Vicuna-7B
Group
Linear
MLP
Difference
Linear
MLP
Difference
Linear
MLP
Difference
College graduate/some postgrad
0.066
0.062
0.004
0.054
0.053
0.001
0.050
0.047
0.003
Less than high school
0.049
0.046
0.003
0.045
0.044
0.001
0.047
0.040
0.007
$100,000 or more
0.072
0.068
0.003
0.066
0.062
0.004
0.054
0.052
0.003
Less than $30,000 Conservative
0,046
0.044
0.003
0.048
0,046
0.002
0.047
0.039
0.008
0.068
0.068
0.001
0.067
0.061
0.006
0.072
0.060
0.012
Liheral
0.094
0.087
0.006
0.085
0.080
0.005
0.086
0.069
0.017
Moderate
0.053
0.049
0.003
0.053
0.046
0.006
0.051
0.043
0.008
Democrat
0.087
0.080
0.007
0.098
0.089
0.009
0.084
0.066
0.018
Republican
0,087
0.074
0.014
0.069
0.061
0.007
0.088
0.074
0.014
Asian
0.068
0.061
0.007
0.072
0.066
0.007
0.055
0.052
0.003
Black
0.077
0.072
0.005
0.080
0.076
0.004
0.066
0.059
0.007
Hispanic
0.049
0.044
0.005
0.051
0.045
0.006
0.042
0.041
0.001
White
0.061
0.054
0.007
0.055
0.052
0.003
0.050
0.046
0.004
Northeast
0.061
0.054
0.007
0.053
0.049
0.004
0.048
0.045
0.003
South
0.058
0.051
0.007
0.054
0.048
0.006
0.042
0.041
0.000
Protestant
0.071
0.061
0.007
0.055
0.041
0.009
0.061
0.051
0.011
Jewish
0.082
0.080
0.003
0.062
0.057
0.005
0.065
0.063
0.003
Muslim
0,120
0.115
0.005
0.115
0.108
0.007
0.095
0.097
-0.002
Hindu
0.089
0.089
0.000
0.084
0.074
0.010
0.077
0.070
0.007
Atheist
0.130
0.120
0.010
0.111
0.096
0.015
0.105
0.087
0.018
Female
0.058
0.056
0.003
0.046
0.043
0.003
0.055
0.051
0.004
Male
0.058
0.056
0.002
0.043
0.041
0.002
0.046
0.046
0.001
Average
0.073
0.068
0.005
0.067
0.061
0.006
0.063
0.056
0.007
Table 6: Linear vs. MLP probes by demographic group. Difference Linear are for 3-option questions only.
MLP. The results
LLaMA-SB
Mistral-7B
Vicuna-7B
Group
Linear
MLP
Difference
Linear
MLP
Difference
Linear
MLP
Difference
College graduate/some postgrad
0.115
0.111
0.004
0.107
0.104
0.003
0.110
0.103
0.007
Less than high school
0.068
0.065
0.003
0.060
0.056
0.004
0.061
0.054
0.006
$100,000 or more
0.117
0.113
0.004
0.107
0.102
0.005
0.110
0.106
0.004
Less than $30,000 Conservative
0.072
0.070
0.002
0.069
0.063
0.006
0.066
0.062
0.004
0.142
0.139
0.002
0.137
0.130
0.007
0.128
0.112
0.016
Liberal
0.116
0.114
0.002
0.119
0.117
0.003
0.119
0.113
0.007
Moderate
0.087
0.086
0.001
0.088
0.084
0.004
0.081
0.079
0.002
Democrat
0.119
0.109
0.010
0.113
0.108
0.005
0.116
0.104
0.012
Republican
0.158
0.7
0.021
0.143
0.136
0.007
0.138
0.126
0.012
Asian
0.102
0.097
0.005
0.101
0.095
0.006
0.085
0.081
0.004
Black
0.093
0.092
0.000
0.090
0.087
0.003
0.081
0.079
0.002
Hispanic
0.078
0.081
0.003
0.076
0.072
0.004
0.071
0.068
0.003
White
0.113
0.104
0.008
1.104
0.099
0.004
0.106
0.097
0.009
Northeast
0,088
0.087
0.001
0.093
0.093
0.000
0.084
0.079
0.004
South
0.090
0.089
0.002
0.087
0.087
0.000
0.081
0.076
0.005
Protestant
0.108
0.
0.000
0.107
0.104
0.003
0.104
0.090
0.014
Jewish
0.144
0.141
0.004
0.142
0.141
0.001
0.140
0.134
0.006
Muslim
0.168
0.166
0.003
0.186
0.186
0.000
0.167
0.169
-0.002
Hindu
0.168
0.166
0.002
0.172
0.170
0.003
0.141
0.148
-0.007
Atheist
0.162
0.150
0.012
0.148
0.143
0.005
0.139
0.132
0.007
Female
0.094
0.091
0.003
0.090
0.090
0.000
0.080
0.079
0.001
Male
0.092
0.088
0.004
0.088
0.088
0.000
0.089
0.082
0.007
Average
0.113
0.109
0.004
0.110
0.107
0.003
0.104
0.099
0.006
16

---

Published as a conference paper at ICLR 2026
Table 7: Prompting vs. probing results with two alternate prompting formats, PORTRAY and BIO, as described in Table 2 "% Improvement" = (Prompting Probel/Prompting x 100. The results are for 2-option questions only, using the 1lama-3.1-8B model.
QA
BIO
PORTRAY
Group
Prompt
Probe
% Impro.
Prompt
Probe
% Impro.
Prompt
Probe
% Impro.
College graduate/some postgrad
0.286
0.066
76.920
0.142
0.061
57.184
0.174
0.058
66.921
Less than high school
0.190
0.049
74.553
0.104
0.045
56.331
0.121
0.041
65.878
$100,000 more
0,188
0.072
61.903
0.151
0.063
57.882
0.189
0.067
64.762
Less than $30,000 Conservative
0,181
0.046
74.409
0.114
0.050
56.003
0.169
0.051
70.032
0,262
0.068
73.910
0.167
0.072
56.927
0.239
0.069
71.122
Liberal
0.186
0.094
49.736
0.153
0.074
51.670
0.224
0.081
63.856
Moderate
0.188
0.053
72.006
0.157
0.054
65.471
0.217
0.057
73.739
Democrat
0.211
0.087
58.936
0.146
0.077
47.265
0.167
0.074
55.854
Republican
0,294
0.087
70.260
0.180
0.084
53.186
0.225
0.091
59.655
Asian
0,219
0.068
69.014
0.142
0.063
55.566
0.169
0.060
64.484
Black
0,211
0.077
63.672
0.153
0.071
53.616
0.172
0.071
58.724
Hispanic
0.201
0.049
75.665
0.121
0.047
60.998
0.126
0.043
65.792
White
0.241
0.061
74.637
0.133
0.058
56.507
0.171
0.056
67.246
Northeast
0.288
0.061
78.873
0.169
0.053
68.906
0.157
0.051
67.526
South
0.280
0.058
79.213
0.156
0.047
69.903
0.169
0.046
73.029
Protestant
0.234
0.071
69.615
0.119
0.064
46.634
0.175
0.063
63.847
Jewish
0.177
0.082
53.602
0.114
0.068
10.717
0.165
0.072
56.377
Muslim
0.235
0.120
48.820
0.162
0.106
34.493
0.205
0.103
49.798
Hindu
0,196
0.089
54.372
0.141
0.094
33.576
0,185
0.088
52.474
Atheist
0,230
0.130
43,509
0.154
0.110
28.293
0.199
0.099
50.464
Female
0.189
0.058
69.117
0.139
0.056
60.056
0.183
0.048
73.642
Male
0.219
0.058
73.590
0.140
0.054
61.685
0.185
0.046
74.938
Average
0.223
0.073
66.651
0.143
0.067
53.312
0.181
0.065
64.098
Table 8: Prompting vs. probing results with two alternate prompting formats, PORTRAY and BIO, as described in Table 2 "% Improvement" = (Prompting (room)/Prompting x 100. The results are for 3-option questions only, using the Llama-3. 1-BB model.
QA
BIO
PORTRAY
Group
Prompt
Probe
% Impro.
Prompt
Probe
% Impro.
Prompt
Probe
% Impro
College graduate/some postgrad
0.248
0.115
53.738
0.189
0.101
46.383
0.207
0.118
42.844
Less than high school
0.216
0.068
68.433
0.153
0.058
62.024
0.167
0.072
57.129
$100,000 more
0.218
0.117
46.136
0.223
0.111
50.093
0.230
0.126
45.374
Less than $30,000 Conservative
0.240
0.072
70.095
0.152
0.069
54.942
0.195
0.083
37.377
0.322
0.142
56.107
0.233
0.128
44.981
0.300
0.131
56.161
Liberal
0.214
0.116
45.920
0.177
0.113
36.416
0.247
0.137
44.357
Moderate
0.204
0.087
57.396
0.168
0.085
49.513
0.211
0.093
56.207
Democrat
0.211
0.119
43.886
0.162
0.102
36.710
0.208
0.123
40.945
Republican
0,380
0.158
58.355
0.241
0.144
40.411
0.298
0.159
46.503
Asian
0.194
0.102
47.668
0.167
0.100
40.290
0.193
0.111
42.
Black
0.216
0.093
57.133
0.163
0.086
47.138
0.209
0.099
52.619
Hispanic
0.205
0.078
61.894
0.144
0.076
46.924
0.174
0.086
50.434
White
0.256
0.113
55.898
0.186
0.104
43.989
0.242
0.117
51.564
Northeast
0.248
0.088
64.492
0.171
0.084
50.907
0.185
0.091
50.961
South
0.245
0.090
63.141
0.168
0.080
52.331
0,188
0.086
54.562
Protestant
0.269
0.108
59.896
0.179
0.112
37.506
0.224
0.117
47.878
Jewish
0,232
0.144
37.868
0.209
0.144
31.084
0,221
0.159
28.200
Muslim
0.260
0.168
35.286
0.258
0.182
29.446
0.255
0.178
30.208
Hindu
0.249
0.168
32.374
0.234
0.165
29.708
0.226
0.176
22.104
Atheist
0.240
0.162
32.625
0.248
0.155
37.372
0.235
0.170
27.467
Female
0.206
0.091
54.516
0.170
0.084
50.533
0.210
0.099
52.562
Male
0.222
0.092
58.575
0.169
0.087
48.852
0.192
0.097
49.692
Average
0.241
0.113
52.792
0.189
0.108
43.980
0.219
0.119
45.786
17

---

Published as a conference paper at ICLR 2026
Table 9: Group-level vs. Single/All Attribute Probes (2-option questions, linear probe), using the Llama-3.1-8B model
Group
Group Level
Single Attrihute
All Attributes
Single Group
All -rroup
College graduate/some postgrad
0.067
0.066
0.064
-0.001
-0.033
Less than high school
0.049
0.049
0.049
0.000
0.001
$100,000 more
0,071
0.072
0.074
0,000
0.003
Less than $30,000 Conservative
0.047
0.046
0.049
-0.001
0.002
0.065
0.068
0.080
0.004
0.016
Liberal
0.090
0.094
0.095
0.003
0.005
Moderate
0.052
0.053
0.056
0.001
0.004
Democrat
0.081
0.087
0.089
0.006
0.009
Republican
0.077
0.087
0.098
0.011
0.022
Asian
0.067
0.068
0.065
0.001
0.001
Black
0.072
0.077
0.076
0.005
0.001
Hispanic
0.050
0.049
0.046
00.01
00.04
White
0.059
0.061
0.055
0.003
-0.04
Northeast
0.062
0.061
0.058
0.001
0.004
South
0.057
0,058
0.054
0.001
-0.003
Protestant
0.058
0.071
0.059
0.014
0.001
Jewish
0.083
0.082
0.080
0.001
-0.003
Muslim
0.123
0.120
0.122
0.003
00.00
Hindu
0.091
0.089
0.095
0.001
0.005
Atheist
0.121
0.130
0.135
0.008
0.014
Female
0.057
0.058
0.059
0.001
0.002
Male
0.058
0.058
0.059
-0.001
0.001
Average
0.071
0.073
0.074
0.002
0.003
Table 10: Group-level vs. Single/All Attribute Probes (3-option questions, linear probe), using the Llama-3.1-8B model.
Group
Group Level
Single Attribute
All Attributes
Single Group
All -l-ur
College graduate/some postgrad Less than high school $ $100,000 more Less than $30,000 Conservative
0.114
0.115
0.118
0.001
0.005
0.062
0.068
0.076
0.006
0.014
0.116
0.117
0.118
0.001
0.001
0,070
0.072
0.072
0.002
0.002
0,129
0.142
0.145
0.013
0.016
Liberal Moderate
0.111
0.116
0.114
0.005
0.003
0.087
0.087
0.088
-0.001
0.000
Democrat
0.103
0.119
0.113
0.016
0.010
Republican
0.136
0.158
0.164
0.022
0.028
Asian
0.100
0.102
0.103
0.002
0.003
Black
0.088
0.093
0.102
0.005
0.014
Hispanic
0.078
0.078
0.08S
0.000
0.009
White
0.105
0.113
0.112
0.008
0.007
Northcast
0.090
0.088
0.089
-0.002
0.001
South
0.088
0,090
0.093
0.002
0.005
Protestant
0.103
0.108
0.107
0.005
0.004
Jewish
0.149
0.144
0.141
0.005
0.009
Muslim
0.166
0.168
0.170
0.002
0.004
Hindu
0,173
0.168
0,173
0..05
0.000
Atheist
0.150
0.162
0.154
0.011
0.004
Female
0.095
0.094
0.099
-0.002
0.004
Male
0.092
0.092
0.100
0.000
0.008
Average
0.109
0.113
0.115
0.004
0.006
18

---

Published as a conference paper at ICLR 2026
Table 11: Unembedding fine-tuning and LoRA fine-tuning for 2-option questions (LLaMA),
using the Llama-3.1-80 model. Improvements are computed relative to prompting: Imp.
(Prompt KL - Method KL)/Promjy KL.
Group
Prompt KL.
KL (Unemb)
Imp. (Unemb)
KL (LoRA)
Imp. (LoRA)
ImptUnemb///p((..R.
College grad/some postgrad
0.280
0.087
0.689
0.049
0.825
0.835
Less than high school
0.18% 0.189
0.076
0.598
0.044
0.767
0.779
$100,000 or more
0.192
0.068
0.647
0.041
0.787
0.822
Less than $30.000 Male
0.191
0.063
0,670
0.040
0.791
0.848
0.208
0.073
0.648
0.044
0.788
0.823
Female
0.178
0.084
0.527
0.038
0.786
0.671
White
0.223
0.088
0.606
0.045
0.798
0.759
Black
0.200
0.076
0.621
0.045
0.775
0.800
Hispanic
0.189
0.052
0.725
0.040
0.789
0.920
Asian
0.196
0.070
0.643
0.043
0.781
0.824
Protestant
0.215
0.069
0.679
0.039
0.819
0.829
Jewish
0.186
0.095
0.489
0.017
0.747
0.651
Muslim
0.214
0.087
0.593
0.049
0.771
0.770
Hindu
0,204
0.125
0.386
0.058
0.715
0.540
Atheist
0,233
0.126
0.460
0,056
0.760
0.605
Liheral
0.204
0.081
0.604
0.047
0.770
0.784
Conservative
0.231
0.076
0.671
0.048
0.792
0.847
Moderate
0.197
0.074
0.624
0.045
0.771
0.809
Democrat
0,204
0.081
0.602
0.046
0.774
0.778
Republican
0.239
0.094
0.606
0.051
0.786
0.771
Northeast
0.275
0.086
0.687
0.050
0.818
0.840
South
0.249
0.063
0.747
0.037
0.851
0.877
Average
0.213
0.082
0.618
0.046
0.787
0.786
Table 12: Unembedding finetuning and LoRA finetuning for 3-option questions (LLaMA), using the
Llama-3.1 model. Improvements are computed relative to prompting: Imp. (Prompt KL Method KT./Promppl KT.
Group
Prompt KL
KL (Unemb)
Imp. (Unemb)
KL (LoRA)
Imp. (LoRA)
Impp(Unemb./IIpp(oRAA
College grad/some postgrad
0.250
0.114
0.544
0.095
0.620
0.877
Less than high school
0.198
0.083
0.580
0.062
0.686
0.845
$100,000 more Less than $30,000
0.209
0.134
0.360
0.098
0.532
0.677
0.227
0.088
0.612
0.056
0.753
0.813
Male
0.196
0.105
0.464
0.075
0,617
0.752
Female White
0,200
0.107
0.464
0.078
0,609
0.762
0.239
0.114
0.523
0.084
0.648
0.806
Black
0.213
0.111
0.480
0.070
0.672
0.714
Hispanic
0.201
0.095
0.527
0.073
0.637
0.828
Asian
0.208
0.119
0.428
0.082
0.606
0.707
Protestant
0.237
0.105
0.557
0.084
0.646
0.863
Jewish
0.206
0.146
0.292
0.111
0.462
0.633
Muslim
0.279
0.184
0.340
0.149
0.465
0.730
Hindu
0.278
0.198
0.287
0.152
0.453
0.635
Atheist
0.236
0.173
0.266
0.135
0.427
0. 0.622
Liberal
0.211
0.138
0.427
0.094
0.610
0. 7.700
Conservative
0.299
0.143
0.522
0.099
0.669
0.780
Moderate
0.219
0.111
0.494
0.073
0.667
0.740
Democrat
0.208
0.137
0.342
0.080
0.616
0.556
Republican
0.326
0.159
0.513
0,101
0,690
0.742
Northeast
0.250
0.105
0.579
0.075
0,699
0,828
South
0.236
0.095
0.597
0.068
0.711
0.839
Average
0.234
0.126
0.464
0.091
0.613
0.756
19

---

Published as a conference paper at ICLR 2026
A.2 GENERALIZATION NON-US CONTEXTS
To test whether our results generalize to non-US contexts, we apply the same probing framework GlobalOpinionQA (Durmus et al. 2023). GlobalOpinionQA contains questions from the Pew Global Attitudes Survey and the World Values Survey (WVS), with responses from countries around the globe. The dataset is curated as (country, question) pairs, where each pair gives question and the corresponding answer distribution for country. Instead of predicting the opinions of demographic groups within the US, we predict the opinions of different countries on cross-national surveys. We keep the top 5 countries in ElonaoOOi besides the US and fit probe to predict their opinion distributions. find similar results this setting: probing achieves a significantly lower KL than next-token probabilities (26-39% reduction, shown in Tables 13 and [4), and we observe a similar layer-wise trend, with knowledge emerging rapidly in the middle layers (Figure 7.
Table 13: Prompt vs. Probing by country (2-option questions). Probe)/Prompt 100.
% Improvement" (Prompt
Country Prompt KL
Probe KL
Improvement (%)
Germany
0.135
0.101
25.440
France
0.106
0.077
27.950
Britain
0.130
0.070
45.940
Spain
0.156
0.105
32.860
Turkey
0.225
0.108
52.110
Average
0.150
0.092
38.860
Table 14: Prompt vs. Probing by country (3-option questions). (robe)/Promey x 100.
"% Improvement" (Prompt
Country
Prompt KL
Probe KL
Improvement (%)
Germany
0.196
0.142
27.950
France
0.196
0.165
15.880
Britain
0.183
0.142
22.200
Spain
0.131
0.125
6.650
Turkey 0.254
0.136
46.380
Average 0.193
0.142
26.290
Catox Courtyy (ffidee
(a) 2-option questions
(b) 3-option questions
Figure Layer-by-layer probe performance on GlobalOpinionQA evaluated on the held-out set.
20

---

Published as a conference paper at ICLR 2026
A.3 SPARSE AUTOENCODER AND STEERABILITY DETAILS
In this section, we provide implementation details for our sparse autoencoder (SAE) analysis and motivate final configuration choice.
Each attention head in JJama-3.1-8P has dimensionality d = 128, so the concatenated output of all 32 heads per layer has dimensionality Din 32d 4096. We train SAEs on these concatenated head activations per layer. We compare six configurations spanning three dictionary widths and two token positions:
(MSAE, KSAE) {(512, 32), (2048, 100). (4096, 100)}
evaluated at either the demographic-label token (demo) or the final token (last). Larger widths are paired with higher top-k values maintain comparable reconstruction quality.
Configuration comparison. We evaluate six SAE configurations (three dictionary widths two token positions) by computing, for each of the 22 demographic groups, the maximum score achieved by any feature at each layer, reported in Figures 12
Token position. SAEs trained on the demographic-laber token substantially outperform those trained on the final token. Demo-token SAEs achieve mean max-F1 scores of 0.622-0.693 across widths, 2 0.7. contrast, last-token SAEs achieve mean max- Fl of only 0.525-0.553 and just 27-70 features above the same threshold an ofderrof-magnitude reduction ceduction-despite comparable reconstruction quality (cxplained variance 80.6-88.8% vs. 98.2- 99.0%). This indicates that demographic identity primarily encoded at the token position where the group label appears, rather than being propagated to the final position.
with 448-2,494 features exceeding
Dictionary width. Among demo-token SAEs, MSAE 2048 and KSAE 100 achieves the highest mean max-F1 (0.693) and 1,934 features above F1 2 0.7, compared to 0.622/448 for M 512, KSAF = 32 and 0.669/2,494 for MSAE 4096, KSAF 100. Although MSAF 4096, KSAR 100 recovers more total features, its mean F1 is lower and it exhibits a substantially higher dead- feature rate (61.3% vs. 43.0%), indicating inefficient dictionary utilization. The smallest model (MsAe = 512, KSAE = 32) has the lowest dead rate (38 (3.2%% but insufficient capacity to disentangle group-specific structure, yielding the lowest mean F1.
Selected configuration. Based this analysis,
the demo-token SAE with MSAE 2048 and KSAE = 100 in all steerability experiments. This configuration provides the best balance of feature discriminability (highest mean max-F1), broad feature coverage across groups and layers, moderate dead-feature rate, and strong reconstruction fidelity (98.95% explained variance; MSE = 0.0105). We provide further details about the SAEs metrics and the feature qualities in Table 15 and Figures ??- 12
Layer range ablation. We compare two intervention ranges: focused range spanning layers 9- 18 (Table 16) and broader range spanning layers 2-30 (Table 17). Both tables report results across the same steering directions based on experiment passing the feature quality criteria.
The focused 9-18 intervention achieves mean aggregate improvement of 0.537 across directions, while the broader 2-30 intervention achieves 0.561, a difference 0.024 (approximately 4%). De- spite intervening nearly three times as many layers (28 vs. 10), the broader range yields only modest increase in average steering performance.
This close correspondence indicates that the majority of steering signal is concentrated within the middle layers. We therefore adopt the focused 9-18 intervention range in our main experiments, as it substantially reduces computational cost while maintaining comparable overall performance.
21

---

Published as a conference paper at ICLR 2026
Table 15: SAE metrics summary (mean across layers 0-31).
Token Type
Width
Best MSE
Explained Var
Dead
Mean Fire Count (alive)
Mean Recon Strength
demo
w512_k32
0.018
98.22%
38.2%
1100.900
6.569
demo
w2048_k100
0.011
98.95%
43.0%
985.100
2.924
demo
w4096 k100
0.010
99.02%
61.3%
692.000
2.398
last
w512.k32
0.194
%0.61%
26.9%
943.100
7.025
last
w2048 k100
0.116
88.36%
26.3%
724.100
3.284
last
w4096 k10
0.112
88.83%
44.4%
483.500
3.048
Table 16: Steerability experiments across layers 9-18. 5 denotes the steering strength. "Num Target Features" the number of features used for the target group. "Avg. Improvement" averages 2-option and 3-option improvements.
Experiment
Attribute
5
Num Target Features
Avg. Improvement
Less than high school - College graduate/some postgrad College graduate/some postgrad -> Less than high school Less than $30,000 $100,000 than $30,000 Northeast + South
Education 10.000
45.000
0.913
Education
10.000
46.000
0.925
Income 10.000
49.000
0.868
$100.000 more
Income 10.000
48,000
0.764
Region 15.000
19.000
0.706
Black + Asian
Race 15.000
17,000
0.671
Protestant Atheist
Religion
10.000
34.000
0.633
Black + White
Race 15.000
16.000
0.626
Male + Female
Gender
25.000
6.000
0.536
South + Northeast
Region
10.000
12.000
0.500
White - Asian
Race
15.000
17.000
0.488
Asian + White
Race
10.000
16.000
0.438
Protestant Muslim
Religion
15.000
23.000
0.432
Atheist + Protestant
Religion
10.000
23.000
0.372
Protestant + Jewish
Religion
15.000
21.000
0.300
Jewish - Protestant
Religion
10.000
23.000
0.289
Muslim Protestant
Religion
10.000
23.000
0.176
Table 17: Steerability experiments across layers 2-30. 8 denotes the steering strength. "Num Target Features" the number of features used for the target group. "Avg. Improvement" averages 2-option and 3-option improvements.
Experiment
Attribute
5
Num Target Features
Avg. Improvement
Less than high school - College graduate/some postgrad Less than $30,000 - $100,000 or more
Education
5.000
213.000
0.931
Income
5.000
252.000
0.927
$100,000 more Less than $30,000
Income
5.000
214.000
0.805
College graduate/some postgrad Less than high school
Education
5.000
207.000
0.803
Black + Asian
Race
25.000
18.000
0.780
Northeast - South
Region 15.000
19.000
0.718
Protestant + Athcist
Religion
10.000
81.000
0.692
Black + White
Race 25.000
17.000
0.651
White + Asian
Race 25.000
18.000
0.631
Male - Female
Gender
25.000
6.000
0.543
Protestant -> Muslim
Religion
25.000
54.000
0.520
South + Northeast
Region
10.000
12.000
0.498
Asian White
Race
15.000
17,000
0.477
Protestant Jewish
Religion
25.000
44.000
0.374
Atheist - Protestant
Religion
5.000
69.000
0.329
Jewish + Protestant
Religion
5.000
69.000
0.290
Muslim + Protestant
Religion
5.000
69.000
0.203
22

---

Published as a conference paper at ICLR 2026
Demo
4006
Aboed
Dist
Figure 8: Maximum F1 value achieved by SAE features per demographic group and layer using SAES with MSAE 512 and KSAE 32, using the demographic token.
Demo 10009G
500e
9.08
Figure 9: Maximum Fl value achieved by SAE features per demographic group and layer using SAEs with MSAE 4096 and KSAE 100, using the demographic token.
Last (WGL2 R32
G00gle
E00%
LOVE
Mede
Figure 10: Maximum Fl value achieved by SAE features per demographic group and layer using SAES with MSAE = 512 and KSAE = 32, using the last token.
23

---

Published as a conference paper at ICLR 2026
Pm-de
Ociee
Papalcm
Sut
Palecat
Figure 11: Maximum value achieved by SAE features per demographic group and layer using SAEs with MSAE 2048 and KSAE 100, using the last token.
Ferue
Oolgge
3.08
Canerober
Suh
Figure 12: Maximum value achieved by SAE features per demographic group and layer using SAES with MSAE 4096 and KSAE 100, using the last token.
24


Reviews:
[{"review_id": "cL0lPSFq4d", "reviewer": "Reviewer_Afma", "summary": "This paper investigates opinion-related knowledge encoded within large language models (LLMs). To identify latent knowledge in LLMs, the authors employ probing and sparse autoencoder techniques. Specifically, they extract predicted probabilities from a trained probe given opinion-eliciting prompts and compare these against next-token probabilities, measuring divergence from ground-truth human opinion distributions. The experimental evaluation utilizes two opinion datasets containing demographic information. Results demonstrate that probing-based predictions align more closely with ground-truth distributions than standard LLM outputs. Through comprehensive layer-wise and option-wise analyses, the authors reveal that middle layers encode the richest opinion-related knowledge. The authors conclude that LLMs encode substantially more internal knowledge about opinions than their generated outputs suggest.", "questions": "The nature of the captured knowledge requires clarification. The authors frame their research question as \"what do LLMs know about human opinions?\" and examine differences across demographic groups. However, it is ambiguous whether the results reveal knowledge specifically about demographics, opinions, or their interaction. To strengthen the claims, the authors should conduct control experiments with non-opinion tasks (e.g., cultural knowledge or mathematical reasoning) using the same methodology. Do similar layer-wise patterns emerge? Are the demographic effects opinion-specific or more general? Such investigations are essential to establish whether the findings are unique to opinion modeling or reflect broader properties of LLM internal representations.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The primary strength of this work lies in its revealing findings about hidden knowledge in LLMs for opinion generation tasks. The results indicate that LLMs possess greater capacity to generate demographically appropriate opinions than their outputs reflect, suggesting interesting limitations in the generation process. The methodology, while conceptually straightforward, is sound and reproducible, providing a template that other researchers can adapt for similar investigations.", "weaknesses": "The paper would benefit from deeper analysis in several areas. The observation in line 321 “This suggests that the drop from probe to prompting arises at the unembedding stage” is particularly intriguing but underexplored. While the authors provide additional investigation, the presentation consists primarily of reported numbers without sufficient interpretation. Visualizing these results in a format similar to Figure 4 would help highlight the pattern where final layers underperform middle layers. Without such emphasis and thorough analysis, it remains unclear how this work's findings differ substantively from prior research examining layer-specific roles in other tasks, such as multilingualism [1]. \n\n[1] Zhao, Yiran, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. \"How do large language models handle multilingualism?\" Advances in Neural Information Processing Systems 37 (2024): 15296-15319.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EuIayYKSY6", "reviewer": "Reviewer_9ET2", "summary": "This paper investigates what large language models (LLMs) internally know about human opinions and how such knowledge is represented in their layers. Using two large-scale U.S. opinion datasets, OpinionQA and SubPOP, the authors probe the residual streams of models such as Llama-3.1-8B, Mistral-7B, and Vicuna-7B to assess alignment with real human opinion distributions across 22 demographic groups. They find that LLMs encode substantially more knowledge about opinions than is evident from their next-token outputs, with probing achieving over 50% lower KL divergence compared to direct prompting. The study contributes to understanding latent representations of social knowledge in LLMs and suggests computationally efficient methods for probing or fine-tuning models for opinion prediction and value alignment.", "questions": "1. Could the authors elaborate on how “internal knowledge” might be made actionable? For example, can probing insights be used to steer or align model outputs at inference time?\n\n2. How robust are the findings to prompt variations or to LLMs trained on non-English or cross-cultural corpora?\n\n3. How does the framework compare to retrieval-augmented or social simulation approaches that integrate external evidence (e.g., Group Preference Optimization: Few-Shot Alignment of Large Language Models)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Novel methodology and insight\nThe paper presents a compelling approach to measuring LLMs’ internal representations of opinions rather than relying on surface-level outputs. This perspective offers new insights into how LLMs encode multidimensional social information.\n\n2. Strong empirical and interpretability contribution\nBy combining probing with sparse autoencoders, the study identifies where and how demographic and opinion-related features emerge in model layers. This combination of layer-wise probing and interpretability analysis is novel and methodologically sound.\n\n3. Computational efficiency and practical implications\nDemonstrating that probing achieves comparable gains to fine-tuning at a fraction of the cost provides a practical contribution relevant to alignment, social modeling, and interpretability.", "weaknesses": "1. Limited generalizability\nThe datasets used (OpinionQA and SubPOP) are limited to U.S. opinions. The authors acknowledge this, but cross-cultural validation would be critical to claim broader generalization of “LLMs’ knowledge of opinions.”\n\n2. Unclear link between internal knowledge and actual reasoning\nWhile the paper shows that LLMs encode information internally, it is less clear whether this knowledge can be effectively surfaced during generation. The distinction between “knowing” and “using” opinions could be discussed more deeply.\n\n3. Interpretability conclusions may overreach\nAlthough the use of sparse autoencoders reveals correlations between attention heads and demographic groups, the causal interpretation of these findings remains speculative. Prior work has shown that attention heads often exhibit polysemantic behavior and that apparent interpretability can arise from correlations rather than direct causal encoding (Kissane et al., 2024; O’Neill & Bui, 2024). Similarly, ablation studies such as Baan et al. (2019) demonstrate that removing seemingly interpretable heads often has limited impact on model performance, suggesting that correlation of head activity with a concept does not necessarily imply that the head causally encodes it. The paper would benefit from acknowledging these interpretability limitations and framing its conclusions accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5Ir4qKkDF", "reviewer": "Reviewer_vuiZ", "summary": "The paper studies what LLMs “know” about human opinions beyond their surface outputs. Using OpinionQA and SubPOP (US public-opinion surveys) across 22 demographic groups, the authors probe residual-stream activations layer-wise (multinomial logistic vs. MLP probes) to predict full answer distributions to survey questions. Key findings: (i) probes extract substantially more opinion knowledge than next-token probabilities (≈50–60% lower KL), (ii) this knowledge emerges sharply in middle layers (≈layers 10–15), and (iii) the unembedding layer explains much of the gap between internal knowledge and outputs; finetuning only the final linear head recovers probe-level gains and achieves a large fraction of LoRA’s improvements with far fewer parameters. They further train SAEs over attention-head activations and claim group-selective features concentrated in middle layers.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear research question & careful operationalization. Evaluating distributional alignment (KL over options) rather than single labels is appropriate for opinions and more informative than argmax accuracy.\n\n- Layerwise analysis identifies where knowledge concentrates. The middle-layer emergence is consistent and useful for targeted interventions.", "weaknesses": "I am unconvinced by the novelty and significance of this work. I personally found that the paper largely re-applies established probing methodology to a new label space (opinion/poll distributions) and reports patterns that are unsurprising in light of prior probing literature.\n\n- Major Concerns\n\n   - Incremental Contribution: Most findings (e.g., information concentration in mid layers; linear vs. MLP probe behavior) closely mirror prior probing results. \n\n   - Last-Layer Adaptation Claim: The observation that finetuning or adapting only the output head (unembedding/last layer) recovers most gains is well known from adaptation/calibration lines of work. Presenting this as a key result does not constitute novelty.\n\n   - Probe Design/Analysis: The linear–MLP comparison and the claim that hidden states encode answer distributions are not surprising; no new probing technique, control, or causal intervention is introduced to move beyond correlational readouts.\n\n- Target Definition: Finally, this task predicts polling distributions (as in OpinionQA), which should not be conflated with underlying human opinions. The paper should be explicit about this limitation and avoid over-claiming.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LneivpyHps", "reviewer": "Reviewer_XXcw", "summary": "This paper investigates the internal representation of human opinions in LLMs, studying how these models align with human values and learn during training. Unlike previous studies that focus on output-based evaluations, the authors probe internal model states across 22 demographic groups and a broad range of topics. They find that LLMs possess significantly more accurate internal knowledge of opinions than what is reflected in their outputs, achieving up to 59% better alignment with human responses at a fraction of the computational cost of fine-tuning. The study identifies the emergence of opinion knowledge in middle layers and attributes output discrepancies to final unembedding layers. Using sparse autoencoders, the authors trace opinion-related features to specific attention heads linked to demographic distinctions. \n\nThis is a clearly written paper with sound experiments and clear contribution points.", "questions": "- Given the emphasis on the need for pluralistic AI systems, why were the experiments conducted on US survey data only?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This work addresses the important topic of how investigating internal representations of human opinions across 22 demographic groups.\n- Though methodologies used, such as linear or multinomial probing, aren’t original, they were appropriately applied to derive results that are reliable.\n- The insights gained through the experiments, such as knowledge of opinions emerge in the middle layers, are useful for the research community.\n- The writing is clear.", "weaknesses": "- The analysis is conducted on US survey data only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)