Reasoning Boosts Opinion Alignment in LLMs
Authors: Frédéric Berdoz
Yann Billeter
Yann Vonlanthen
Roger Wattenhofer
Venue: ICLR 2026 Poster (2026)
Keywords: Opinion modeling
Alignement
LLMs
Reasoning
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Using reasoning and reinforcement learning, we improve LLM opinion modeling, highlight remaining biases, and provide a baseline for future research.

Abstract: Opinion modeling aims to capture individual or group political preferences, enabling applications such as digital democracies, where models could help shape fairer and more popular policies. Given their versatility, strong generalization capabilities, and demonstrated success across diverse text-to-text applications, large language models (LLMs) are natural candidates for this task. However, due to their statistical nature and limited causal understanding, they tend to produce biased opinions when prompted naively. In this work, we study whether reasoning can improve opinion alignment. Motivated by the recent advancement in mathematical reasoning enabled by reinforcement learning (RL), we train models to produce profile-consistent answers through structured reasoning. We evaluate our approach on three datasets covering U.S., European, and Swiss politics. Results indicate that reasoning enhances opinion modeling and is competitive with strong baselines, but does not fully remove bias, highlighting the need for additional mechanisms to build faithful political digital twins using LLMs. By releasing both our method and datasets, we establish a solid baseline to support future research on LLM opinion alignment.

Published as a conference paper at ICLR 2026
REASONING BOOSTS OPINION ALIGNMENT IN LLMs
Frédéric Berdoz, Yann Billeter, Yann Vonlanthen, Roger Wattenhofer
ETH Zurich
Switzerland
(fberdoz, ybilleter, yvonlanthen, wattenhoferJ@thhz.ch
ABSTRACT
Opinion modeling aims to capture individual or group political preferences, en- abling applications such as digital democracies, where models could help shape fairer and more popular policies. Given their versatility, strong generalization ca- pabilities, and demonstrated success across diverse text-to-text applications, large language models (LLMs) are natural candidates for this task. However, due to their statistical nature and limited causal understanding. they tend produce bi- ased opinions when prompted naively. In this work, we study whether reasoning can improve opinion alignment. Motivated by the recent advancement in math- ematical reasoning enabled by reinforcement learning (RL), we train models to produce profile-consistett answers through structured reasoning. We evaluate our approach on three datasets covering U.S., European, and Swiss politics. Results indicate that reasoning enhances opinion modeling and competitive with strong baselines, but does not fully remove bias, highlighting the need for additional mechanisms to build faithful political digital twins using LLMs. By releasing both method and datasets"] we establish solid baseline to support future research on LLM opinion alignment.
INTRODUCTION
Could AI give rise to a new kind of democracy, where digital twins vote on our behalf and faithfully reflect our opinions on every issue? Accurate simulations of political behavior offer new oppor- tunities to understand election outcomes and improve policy and democratic processes (Li et al.) 2024). However, capturing how diverse individuals reason about political issues remains challeng- ing. While large language models (LLMs) can generate sophisticated political discourse, they often fail to reflect the diversity of human political viewpoints (Santurkar al., 2023 Qu & Wang 2024 Yu et al., 2025) Qi et al. 2024). This raises a fundamental question: how to design agents that reason about politics while faithfully representing human political diversity?
Current approaches to modeling opinions with LLMs predominantly rely on prompting with de- mographic information and political affiliations, thereby leveraging learned correlations between demographics and opinions. While these prompt- based methods have grown increasingly sophis- ticated, they consistently fail to capture real opinion distributions and exhibit unstable, inconsis- tent responses across different prompts and demographic groups (annurrkrr et al. 2023 Ball et al.) 2025). We instead seek a preference-consisten approach that directly uses an individual's 's known opinions rather than demographic proxies.
Yet, the aforementioned methods do not lend themselves to simulating individual political prefer- ences. Methods using richer data sources like interview transcripts achieve strong performance modeling individual personas (Park et al., 2024), but remain limited by prohibitive data collection costs, as interviewing impractical at scale. In this paper, we propose using political survey data as an alternative. Large surveys, like the American National Election Studies (ANES), and Voting Advice Applications (VAAs), i.e., online platforms that match voters with parties candidates, aim to capture how people position themselves on political issues. While surveys lack the narrative richness and textual signal of interviews, they offer a structured representation of political opinions
across populations. Despite their prevalence in bias evaluation (Santurkar et al. 2023 Argyle et al.
"oode and data: https: //gi thub. wtt.:///-////////////o/
wooatttt--ooooo---ooooo iigmeent

---

Published as a conference paper at ICLR 2026
regulated to prevent harmehi ouparz?
Al gonerate mislendinn content offects sociery.
Survey responses
Resseeinng
Question
Reasoning
SFT
Question
<scasoning>
Reasuning,
</2gason_ng>
Advaniages
Regularisation Diu[sFF GRPO)
Syntheic jeasoning
Reward
Syutheii monninng
2. GRPO
Question
Alignment loss
Syathes feasoning
Figure/ 1: GRPO for opinion alignment. We use public opinion surveys align LLMs with indi- vidual preference profiles. First, the LLM is fine-tuned with (synthetic) statements and ground-truth answers to adhere the reasoning template. After fine-tuning, the model answers in the correct format, but is not fully aligned with opinions. We use GRPO with reward model that rewards proper formatting and correct answers further improve reasoning.
2023 Haller et al. 2025) and group-level simulation (Argyle et al. 2023 Cao et al. 2025). surveys have yet been widely adopted for training individual-level political agents.
RL has recently driven major gains in reasoning, with models like 03/o4-mini (OpenAl 2025 and DeepSeek-R1 (Guo al. 2025) setting new marks various benchmarks. Motivated by these results, we use RL, in particular GRPO (Shao et al. (024), to train agents on survey data, treating opinion formation as a reasoning problem: the agent writes short rationale and final choice, and rewarded when the choice matches the respondent's answer, teaching it to reason toward survey- consistent positions on unseen questions.
We summarize our contributions as follows:
Reinforcement learning for reasoning-based opinion alignment. We introduce method that encourages explicit reasoning through reinforcement learning (GRPO) to improve opinion alignment from survey responses.
Benchmarking on real-world political data. We evaluate our approach on three datasets of authentic political opinions: German parties, U.S. voters, and Swiss candidates, and release them as benchmark to foster future research. Our results show that reasoning consistently improves opinion alignment.
Analyzing ideological effects. We investigate how political ideology influences perfor- mance by inverting the survey answers of individuals with known positions, revealing sys- tematic variations in alignment quality across the political spectrum.
2
RELATED WORK
Modeling political opinions requires LLMs to adopt specific political personas and answer from these particular viewpoints. Prior work has explored demographic personas (representing groups with shared traits like occupation ethnicity), character personas (well-known individuals), and individualized personas (digital profiles based on personalized data) (Chen et al., 2024 Tseng et al. 2024). Our focus modeling individuals makes individualized personas most relevant, though the literature has predominantly employed demographic approaches, with individualized methods represented by only handful of studies.
Modeling political opinions with demographic personas. Political opinion modeling has pri- marily used demographic personas through various prompting strategies. Simple approaches use basic political affiliation (e.g.. "You are democrat. What is your opinion on...") (Santurkar al.) 2023). while more complex methods add demographic details like age, education, and location dAr- gyle et al. 2023 Santurkar al.) 2023; Hwang al.) 2023] Sun et al. 2024: Wang et al.) 2025). Recent work has also explored richer contexts, including prompts with demonstrations from be- lief networks, where models receive related political positions guide their answer (Chuang al.
2

---

Published as a conference paper at ICLR 2026
Table 1: Answers obtained from Llama 3.1 8B trained individual in the Right group in ANES. Before training, Llama disagreed; after training, it agreed. Interestingly. the reasoning trace still appeals to similar arguments ("equal opportunity") but frames them differently. Additional examples are provided in Appendix B
Question:
Do you agrec that it is not really that big problem if some people have more of chance in life than others?
Before (prompting on base model)
No. I don't think that's right. Everyone deserves an equal shot succeeding in life, matter where they re from or who their parents are. If some people have more opportunities and resources, it's not fair to others. We should be working to level the playing field and give everyone a fair chance to succeed.
After GSPT+GRPO)
<reasoning>> Some inequality in wealth and opportunity is inevitable and indeed necessary provide incentives for hard work and innovation the issue that everyone has access to basic necessities and opportunities to improve themselves- - other countries with sigmificant equality have still struggled with poverty and stagnation. < </eesonning>>>>>> <answer>A) Yes</answer /aswwrr>>
climinating all differences but rather ensuring
2024). Gudiño et al. 2024) combine demographics with demonstrations, while Stammbach et al.
(2024) integrate prompting with odds ratio preference optimization (ORPO) (Hong et al., 2024) to better align outputs with target positions. Despite this methodological diversity, demographic approaches face several limitations.
Limitations of demographic approaches. Santurkar et al. (2023) identify three critical issues of prompt-based demographic simulation that undermine authentic political representation: represen- tativeness, where LLMs' default opinion distributions fail to reflect true population distributions; steerability, where demographic prompting provides insufficient control over model outputs; and consistency, where political biases vary unpredictably across different topics and contexts. These limitations manifest as systematic biases, with accuracy varying unpredictably across demographic groups (Argyle et al.] [2023] Santurkar et al. (023). While models can approximate overall opin- ion distributions, they fail to capture natural variance within demographic subgroups and remain sensitive to prompt variations (Sun et al. 2024 Ball al., 2025).
Moving beyond demographies. In contrast to the extensive literature demographic prompting, few studies have explored representing individual opinion profiles. Park et al. (2024) use interview transcripts to achieve strong accuracy on public opinion surveys, outperforming demographic base- lines, but this approach is limited by transcript availability. Haller et al. (2025) propose an in-context learning method, where the model is prompted with respondents' question-answe pairs and then asked to predict the respondents' answers to a set of test questions. However, their focus was on bias identification rather than opinion modeling.
Training LLMs to reason. Encouraging LLMs produce reasoning steps before answering sig- nificantly increases accuracy, with approaches like Chain-of- Thought prompting (Wei et al.] 2022) or simple prompts like "Let's think step by step" (Kojima et al. 2022). Recent advances have shifted from prompting- based reasoming to training models to generate high-quality reasoning traces through search algorithms reinforcement learning (Xu et al.
2025). One particular method is Group-relative policy optimization (GRPO) (Shao et al. 2024). GRPO is a variant of Proximal policy optimization (PPO) (Schulman al., 2017). that uses group-relative reward normaliza- tion rather than learning a value function to estimate advantages. For each prompt, GRPO sam- ples a group of outputs and normalizes rewards by subtracting the group mean and dividing by the group standard deviation. GRPO has proven particularly effective for mathematical reasoning tasks, achieving state-of-the-an the results on benchmarks (Guo et al.) 2025).
Reasoning for opinion alignment. Yu et al. 2025 propose persona-based chain-of-thought inspired prompting framework, which combines demographic, ideological, and temporal factors calibrated using ANES. The authors use this framework to simulate the voting behavior of individual respondents and sum these responses to make aggregate predictions. Unlike our proposed method, they do not reinforcement learning. Moreover, we focus on learning general preference profiles,
3

---

Published as a conference paper at ICLR 2026
Table 2: Dataset overview. Train/test counts refer questions per unit (cantidac/parr//eeyyonenn))
Setting
Unit
#Units
Labels
Train Q
Test Q
smartvote (CH)
Candidate
18
Yes, No}
48
12 (topic-strat.)
WoM (DE)
Party
6
Yes, Neutral, No
see Table 5
30 (EU&I)
ANES 2020 (US)
Respondent
21
Yes, Neutral, No}
67
12 (random)
while Yu et al. (2025) target voting predictions. While our method could theoretically be applied to the same problem, the computational cost of modeling a representative sample size is currently prohibitive.
METHOD
We study individual-level opinion alignment from survey data. Consider policy questions such as "Should the government increase funding for renewable energy?" with answer stances like Y (No, Neut Yes}. Given any policy question and persona p (a respondent or political party), goal is to learn the stance of p q.
3.1 POLITICAL REASONING WITH GRPO
We employ GRPO (Shao al. 2024 train agents that generate reasoning traces before providing answers in the following fixed schema:
Our training data consists of survey responses, D {p), qs y0, where vf, is the observed stance of persona pj on question q:- The key challenge is that DD contains no reasoning traces, only final stances. Therefore, format above and produces answers that match the ground-truth stances. To achieve this, we use composite reward function to evaluate and reward each generation along multiple dimensions:
model must learn generate reasoning that both adheres to the required
Format Reward (Riormat): ) We enforce a structured output format by rewarding correctly placed tags in the schema described in (I), which yields score of 1 for each of the four tags (maximum of 4).
Length Reward (Rtength): Let I be the token length of the reasoning trace I; and I the desired length. We define symmetric penalty: Roength - T.*). which yields maximum reward of 0 only when the trace length equals L
Correctness Reward ( (Roomect): The primary reward signal comes from matching the survey response: Reoreat 1 |y: v where is the model's predicted answer and y, is the ground-truth survey response. The total reward is computed as: R(r,, yt) aj Riomat + Qry Reenth Reonens with a1 = 0.25, a2 = 0.01, 0g = 1.
We do not use an explicit persona representation, and include only country label in the system prompt. The model is aligned with the individuals solely through correctly answering questions.
Optional Initialization via Supervised Fine-tuning. To accelerate convergence and improve ini- tial reasoning quality, we optionally initialize the model through supervised fine-tuning (SFT) before GRPO training. Following (Shao et al. 2024), we construct dataset of chain-of-thouge
demon- strations, where each example follows the exact format incentivized by our reward design. The SFT stage serves as warm start that: (i) reduces the burden Riomat during GRPO by pre-training the model to use the correct output structure, and (ii) provides a reasonable initialization for political reasoning.

---

Published as a conference paper at ICLR 2026
Table 3: Reasoning requires large datasets to be effective. Mean macro-FI (%) with standard deviations over 8 stochastic runs at T'010. For each run, we compute per-unit macro-Fi Fl and then average across units. We report accuracy scores and statistical significance in Appendix E. Scores on smartvote are considerably better than on ANES and WoM. This is likely due to the binary nature of smartvote and the absence of a Neutral option (see Section 4.6). Performance on WoM is better than ANES, likely because of the substantially bigger number of training samples (see also Appendix [.
Base model
Method
Dataset Wo.M
smarivote
ANES
random ma majerity
37.43 33.33
50.00
33.33 22.98
Untrained baselines
27.44
icl ORPO SFT GRPO
55.97 + 5.66
28.17 + 0.02
23.20 + 1.79
43.53
2.81
43.29
5.16
34.84
5.41
Llama 3.1 8B
63.44
1.34
48.95
3.56
42.77
1.23
55.14 1.25
37.29
2.06
34.55 1 1.32
SFT+GRPO
66.88 + 2.18
52.53
- 4.05
40.66 - 0.91
icl
60.48 + 4.14
26.19 + 0.35
23.20 + 1.79
Qwen3 8B
ORPO
23.87 2.07
25.25
3.58
26.95
1.36
SFT
61.08 2.74
42.91
3.44
35.14
1.70
GRPO
60.64 2.01
31.42
1.46
31.47 + 1.06
SFT+GRPO
65.11 3.33
49.38
1.93
38.44 - 0.40
icl ORPO SPT GRPO
66.16 = 0.42
26.19 +
0.35
19.23 + 2.91
23.31
24.73
3.29
24.25
2,28
Magistral 24B
67.63 1.91
51.86
2.58
39.15 + 0.66
60.56
1.93
51.00
3.10
13.79
1.06
SFT+GRPO
70.73 2.21
53.21
3.19
45.43
1.11
4 EXPERIMENTS
4.1 DATASETS
three datasets from different countries and political systems. The dataset creation process is described in the following paragraphs. We release all data for repro- ducibility and future research. An overview of the dataset provided in Table
For experiments,
German party positions. For training, we use the official Wahl-O-Mat (WoM) dataset (version 26 March 2025) aggregating party positions across federal, state, and European elections (2021-2025). We focus on six major parties (CDU/CSU, SPD, Grüne, FDP, Die Linke, AfD). Each item records agree/neutra 1/d sagree and an explanatory comment. We map to (Yes, Neutral, No). For testing. use EU&I (2024) (Chalkidis) (024) as out-of-domain questionnaire (30 items), recoded to the same three labels. Per-party training counts appear in Appendix
American National Election Studies. From ANES 2020 Time Series (American National Elec- tion Studies) 202T). we extract 79 policy items (excluding demographics, knowledge, candidate evaluations, and thermometers). Because response formats are heterogeneous, recode each item to (Yes, Neutral No) using two schemes detailed in Appendix D a conservative mapping that treats only clearly positive/negative options agrremmen/disagreemennt and aggressive mapping that collapses most positive responses to Yes. Unless otherwise stated, results use the conservative scheme. Robustness recoding analyzed Appendix [..2] We split items at random into 67 train and 12 test questions. To analyze ideological variation, we sample respondents based on how liberal-conservativy conservative self- placement question. We included 3 respondents per answer option, for a total of 21 respondents.
they place themselves on ANES' 7-point liberal
Swiss candidates. We use the English questionnaire for the 2023 Swiss national elections (75 items, of which 60 are policy questions across 12 topics). Responses are on four-point Likert scale and are collapsed to (Yes, No} by mapping Yes/Rather yes to Yes and Rather no/No to No. We sample three candidates from each of the six parties with at least two National Council seats (18 candidates total). See Appendix A for additional information on these parties. We perform a topic-stratiffed split: question per topic held out for test (12), the remaining 48 for training.
5

---

Published as a conference paper at ICLR 2026
Left
Center
Right
Agent
Conservative Liberal
Left-Right
Figure 2: Agents are more centrist and conservative. First two dimensions of the principal compo- nent analysis of all candidates (small dots) standing for election in the 2023 Swiss national elections (smartvole). The x- and y-axes correspond to the left-right and conservative-libera spectra, respec- tively. The shifts between the positions and their agents (gold) are depicted by black lines. Unlike results in the literature which indicate a left-libertarian bias (Exler et al.) 2025 Hartmann et al.) [0233 Rozado (024), we observe that agents shifted towards the (center-)right. The average distortions between ground-truth and agent position for each group (big arrows) show an overall trend towards more conservative (nega- tive y) and clear left-right bias. Implementation details given in Appendix E.5
candidates (big dots) included
smartvole dataset
Political Spectrum Groups. For clarity and comparability across political contexts, we assign parties (smarivote, Wahl-o-Mat) and ideologies (ANES) three broad political groups: Left, Center, Right. The assignments are detailed in Appendix A
4.2 EXPERIMENTAL SETUP
We evaluate our approach on different models and against strong baselines. The design targets two questions: (i) how performance scales with model size, and (ii) whether massoning-pretraiin backbones confer advantages over non-reasoning peers. We train one model per unit (candidate/par- ly/respondent). Further training details are given in Appendix F
Methods We report results for three configurations: GRPO the survey questions and answers, as well as SFT+GRPO. For smartvote and ANES, we generate synthetic arguments supporting both positive and negative stances for each policy question using Llama 3.1 70B (Grattafiori et al.) 2024) and use those for SFT. The prompting strategy used to produce these synthetic arguments is detailed in Appendix A.4
Model backbones.. We evaluate across three open-weight backbones to study (i) the effect of model scale and (ii) the impact of prior reasoning pre-training: Ilama3.1 8B (Grattafiori et al., 2024). Qwen3 8B Yang al. (2025). and Magistral 24B Rastogi et al. (2025). Qwen3 and Magistral are pre-trained to reason, allowing us to compare reasoning- reasoning-pretrained backbones against non- reasoning counterpart (Llama3.1) 1) comparable scales. We 4-bit quantization for all models across all experiments that involve training.
Baselines report results for two naive baselines. random selects one of the answers uniformly at random. ma jority answers all questions with the most frequently chosen answer option in that unit's train set. Additionally, we also compare to an in-context learning baseline i icl. We follow the methodology described in (Haller et al., 2025), where the LLM is prompted with question-answer pairs and then asked to answer another unseen question. To so, we report results for the setting where the model receives all the training questions from the same topic area the test question (or a random subset when limited by context length). Finally, we compare to ORPO Hong et al. (2024) using the same hyperparameters as Stammbach al. (2024).

---

be learned from survey Published as a conference paper at ICLR 2026
Wabl-o-Mat
ANES
0.5
0..5
0.6
Fl-score
0.6
0,4
0.4
0.2
Right 0.0
Center
Right
Right
.tt
orpo
1al
Figure 3: Not all political positions are equally learnable. Fl scores reveal that while SFT+GRPO typically works best, every training method underperforms center and right-leaning groups. Error bars show variance within groups. These disparities may stem from biases baked into Llama 3. 8B from inherent differences in how well various political preferences data. Either way, the results demonstrate that ideology impacts the learnability of political prefer- ences. detailed figure with the original parties and ideology groups presented in Appendix
4.3 RESULTS AND ANALYSIS
Table reports macro-F1 averages for all datasets. SFT+GRPO outperforms naive and SFT base- lines across datasets, except for Llama 3.1 8B on ANES. Performance varies by dataset: scores peak smartvote (70.73), while ANES is hardest (45.43). SFT+CRPO performs comparably or better than icl-baselines on ANES and smartvote, and outperforms them on WoM. Using base models with native reasoning support does not consistently improve performance, performs Qwen3 8B on WoM and ANES while Magistral 24B performs best overall. GRPO alone underperforms SFT+GRPO, suggesting supervised warm start improves training dynamics. We ob- serve lower performance on datasets with Neut ral classes, likely because Neutral aggregates multiple behaviors (uncertainty, social desirability, strategic non-commitment), making difficult to learn. We further investigate ideology impact in Section 4.5 4.5land and Neut ra role in Section 4.6 Results without Neut ral appear in Appendix E.
Llama 3.1 8B out-
4.4 HOW DO AGENTS REASON?
In [ present example of reasoning traces. Generally, agents succeed in making coherent arguments for or against a proposed measure that imply the final answer. We identify two failure modes: First, arguments sometimes contain illogical hallucinated elements. Second, agent may argue in line with an individual's ' opinion on one topic, but then fail to do so on other topics. Additional traces and examples of failures provided in Appendix
4.5 POLITICAL IDEOLOGY MATTERS
In Fig.B. we report F1 by ideology group for Llama 3.1 8B (SFT+GRPO). On smartvote, we observe clear performance disparity between the Left group and the Center and Right groups. Wahl-o- Mat and ANES show similar patterns, albeit weaker: On Wahl-o-Mat, the Left and Center groups perform roughly equally well, while performance generally lower on ANES. For the Right group in smartvote and ANES, SFT also outperforms all other methods on average, further suggesting difficulties with right-wing preference profiles. These disparities may reflect known left-leaning tendencies in general-purposs LLMSs (Hartmann et al. [023 Exler et al. 2025) or indicate that Right profiles are intrinsically harder learn from survey signals. The relatively strong Right performance on WoM may be aided by larger training sets (cf.. Appendix E.4)
4.6 NEUTRAL REASONING POSES A PARTICULAR CHALLENGE
As shown in Fig. (top right), recall is worst on the Neutral class for all individuals in the ANES dataset. Furthermore,
be seen from the regression in Fig. 4 (left column), individuals in the Right group respond with Neut ral the most (higher neutral base rate). Clearly, performance for these individuals suffers the most from the difficulties in predicting Neutral, Removing all
7

---

Published as a conference paper at ICLR 2026
1.0
0.8
0.6
Recall
0.5
0.2
0.0
Noutral
+0.155
Accuracy
0.6
0.6
+0.129
0.4
0.4
0.2
0.2
0.0
0.1
0.2
0.3
0.4
0.5
Left
Center
Right
All responses
Neutrals removed
Figure 4: Learning neutral stances is hard. On ANES, there is a strong correlation between predictive performance and the neutral base rate on the test set. Individuals the Right group, and to some extent in the Center group, tend answer questions with Neutral more often. Both in terms of F1 (top left) and accuracy (bottom left), we observe a negative correlation significant at the 5%-level. Regression details in Appendix E.3 Top right: Recall by class on ANES. Averaged over all individuals, the model struggles the most with predicting the Neutral class. Bottom right: Performance in terms of F1 score ANES before (solid) and after (shaded) removing Neut instances. All groups improve, and the gap between Left and Center becomes narrow. However, the difference between Left and Right remains. While the large number of Neutrals likely depressed performance on the other two classes during training, this could also suggest that the performance disparity affected by other factors, such as model bias or artifacts from recoding ANES. All results were obtained from Llama 3.1 8B trained with SFT+CRPO.
Neutral instances and recomputing the scores leads considerable improvement in perfor- mance, as shown in Fig. 4 (bottom right), but does not rectify the gap between the Left and Right group. SFT+GRPO also improves relative to icl-baselines when Neutral is removed (see Appendix E.I. We repeat this analysis under the alternative recoding scheme mentioned in Sec- tion 4.1 and further described Appendix D and observe the same negative correlation between predictive performance and Neutral 1. base rate, but almost identical performance between the Cen- ter and Right groups, indicating that this could, to some extent, be an artifact of recoding ANES. Details are reported in Appendix E.2
4.7 WHOM DO AGENTS REPRESENT?
To better understand the agents' representativity in semantic terms, we analyze their positions rel- ative to the ground-truth positions of their corresponding human individuals in a two-dimensional model of politics. We perform this on the smartvote dataset, for which the interpretation of the principal components is documented (Germann et 2015). Figure presents the positions of the individuals included in the smartvote dataset in the space spanned by the first two principal compo- nents obtained from the full set of candidates in the 2023 Swiss national elections. Agent positions (gold) were then projected into this space. Not all agents represent their respective candidates (big markers) perfectly. However, despite documented left-libertarian biases most LLMs (Exler et al., 2025 Hartmann et al. 2023 Rozado 2024), do not observe overall left shift. Instead, the agents' positions are shifted towards the center-right of the left-right (æ-axis) and the center of the conservative-libest spectrum (y-axis). This is further supported by the group-averaged distances be- tween humans and agents shown in the center of Fig. 2 Right individuals become more left-wing, while Left and Center individuals become more right-wing and conservative.
8

---

Published as a conference paper at ICLR 2026
1.00
0.75
0.25
progressive bias
defiuut
conservative bius
Figure 5: Biased SFT data consistently impairs the performance of underrepresented groups. Fl-scores for Llama trained with SFT+GRPO on differently biased datasets. Left: Data with progressive bias strongly impairs the Right group. without consistently benefiting the Left group. the default dataset. Right: Data with conservative bias decreases the performance of the Left group, while showing no consistent improvement for the Right group. These asymmetric effects suggest that ideological bias SFT data primarily harms the underrepresented perspective rather than systematically improving performance the overrepresented one.
Middle: Performance
Left
Right
Agent
progressive bias
Conservative L-beral
default
Mean
conservative bias
Left-Right
Mcan ALR
Figure 6: Biased SET data primarily shifts the agents of underrepresented groups. Left: Can- didates and their agents the smartvote PCA space. Right: Group-mean displacement vectors of the agents relative to their respective candidates. Similar to the Fl scores presented in Fig. 5 bi- ased data mostly affects the underrepresented group, and Right candidates are again affected more strongly than Left candidates. Notably, conservative-biase and default SFT data produce similar displacement patterns, suggesting the centerward shift in the default data may originate from the base model's alignment rather than the SFT data.
4.8 IMPACT SFT DATA
To investigate the impact of biases in the SFT data, we generated two additional sets of arguments using Llama 3.3 70B, prompting it to produce progressive and conservative arguments, respectively (details in Appendix A.4). Figure presents the impact in terms of Fl scores. Progressive bias strongly impairs Right candidates while showing inconsistent effects on Left candidates; conserva- tive bias similarly impairs Left candidates without consistently benefiting Right candidates. This suggests that ideological bias in SFT data primarily harms underrepresented perspectives rather than systematically improving overrepresented ones. Notably, Right candidates impaired more severely by adverse bias than Left candidates. Figure shows how these same biases affect agent positions in PCA space. Consistent with the Fl results, biased data primarily shifts underrepresented groups, with Right candidates again affected more strongly than Left candidates. Interestingly, the positional shifts under conservative biased and default SFT data are highly similar. This suggests that the centerward shift observed in Fig. may be driven primarily by the alignment of the agent's s base model rather than bias the SFT data itself.
PERFORMANCE ON SYNTHETIC POSITIONS
To assess how individual preferences affect associated agent's performance, create terfactual by inverting every smartvote answer and training Llama-3. 1-B on the resulting dataset. Inversion roughly corresponds 180" rotation about the origin in Fig. (see Appendix R.6.), con-

---

Published as a conference paper at ICLR 2026
1.00
0.50
Left
Center
Right
Original Avg
Inverted Avg
Figure 7: Learning Right and Center positions is harder than Left scores on the original data, with individuals ordered along the x-axis by their first component in the smartvote PCA space. Note that the left-most Right candidate indeed correctly placed (see Fig. 6). Arrows show the difference in Fl scores after all answers in the train and test sets have been flipped. Left candidates improve, but still perform worse on average than Left candidates. As the disparity is not removed by converting Left Right, this could suggest that Right positions are more difficult learn.
Points depict F1
worse after flipping, both individually and on average. Right candidates
verting, for example, right-wing positions into left-wing ones. For each politician, we compute the Fl difference before and after inversion, shown in Fig. 7 Members of the Right group benefit from inversion, while sion, the Right does not reach the original Fl of the Left, while the Left drops roughly the Right 's original level. If the effect were purely due to the model's original political bias, flipping labels should have made right-leaning candidates nearly easy predict left-leaning ones. The fact that this does not occur suggests that left-leaning preference profiles may be intrinsically easier to model, possibility that merits further investigation.
Left and Center groups perform worse on average. Yet even after inver-
DISCUSSION
Limitations. The learning of human opinions is inherently limited by noise, as the same respondent might answer question differently on different days. Hence, consistently achieving an Fl-score close to 1.0 is unlikely. The analyses in this paper cover only small number of individuals or parties, mainly because the method computationally intensive (i.e., training one model per profile). Data another constraint: suitable surveys are scarce. Those that exist often have few items to train individualized agents, or focus on generic themes rather than concrete issues and priorities. Question sets also rarely align across countries, which limits comparability.
Future work. The observed disparities between political groups warrant further investigation into bias mitigation. Further research should aim to replace per-agent training with single model that conditions compact persona representations (e.g., see (Ning et al.) 2024)), reducing computa- tional cost and improving data utilization. Real-world deployments will require confidence estimates to gauge answer reliability, achievable by extending the reasoning scheme through dedicated value head. Purpose-buil built questionnaires and improved uncertainty handling may also mitigate the observed difficulties with the Neutral class.
CONCLUSION
In this work, we explore individual opinion alignment by training persona-conditioned agents di- rectly on known opinions and by framing answer generation as structured reasoning task. Across Swiss candidates (smartvote), German parties (Wahl-o-Mat), and U.S. voters (ANES), SFT+GRPO generally outperforms all baselines. Performance remains uneven as Neut ral stances remain chal- lenging to predict, and accuracy varies significantly across the political spectrum. Our work provides a strong benchmark and baseline for systematic opinion modeling with LLMs, marking an early step toward the possibility of more representative, Al-driven forms of democracy.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We acknowledge that the modeling of indiviidll opinions in time of rising mis- and disinformation not exclusively beneficial, and may pose downstream risks. Precisely because of these risks and the technology's 's inevitable development, we believe responsible research is essential to investigate how individual opinion alignment can be improved. We use only anonymized survey data and public figures' positions to address privacy concerns. Our research reveals systematic performance dispar- ities across ideological groups, with consistent underperformance on center and right-wing profiles, as well as difficulties modeling neutral positions. We commit to transparent reporting of these bi- ases, as they represent significant system risks that could disenfranchise certain political viewpoints if deployed prematurely. We strongly caution against real-world applications until these fairness is- sues are resolved, as biased political modeling systems risk undermining democratic representation. Ultimately, we are convinced that with proper bias mitigation, opinion modeling can ultimately en- able societally beneficial applications such as digital democracies, where models could help shape fairer and more popular policies.
REPRODUCIBILITY STATEMENT
For reproducibility, we release our code and datasets:
https://gith::: htt.ttthh..com./// www.ETH DDSCO/ http:////.....///oeaaooooonooooo
11m a llignment
Training details, including hyperparameters, are given Appendix F Details for all PCA results are described in Appendix E.5
AUTHOR CONTRIBUTIONS
Frédéric Berdoz conceived and designed the project, led experimental planning, and defined its re- search direction. Yann Billeter contributed equally, leading technical development, including dataset collection, model training, evaluation, and analysis. Yann Vonlanthen contributed as an advisor, of- fering guidance methodology and experimental design. Roger Wattenhofer supervised the project as principal investigator.
ACKNOWLEDGMENTS
We thank Prof. Dr. Hans Gersbach for his valuable insights during the project.
REFERENCES
American National Election Studies. ANES 2020 Time Series Study Full Release, 2021. Dataset, https:
w/electionstuulesscoo
L. P. Argyle, E. C. Busby, N. Fulda, J. R. Gubler, C. Rytting, and D. Wingate. Out of One, Many: Using Language Models Simulate Human Samples. Political Analysis, 31(3)3337-351, 2023.
Ball, S. Allmendinger, F. Kreuter, and N. Kühl. Human Preferences in Large Language Model Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Prediction, 2025. arXiv:2502.16238
Cao, H. Liu, A. Arora, I. Augenstein, Röttger, and D. Hershcovich. Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL), 2025.
Chalkidis. Investigating LLMs Voting Assistants via Contextual Augmentation: Case Study on the European Parliament Elections 2024. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
J. Chen, X. Wang. R. Xu, S. Yuan. Y. Zhang, W. Shi, al. From Persona Personalization: Survey on Role-Playing Language Agents. Transactions on Machine Learning Research, 2024.
Y.S. Chuang, K. Nirunwiroj, Z. Studdiford, A. Goyal, V. V. Frigo, S. Yang. et al. Beyond Demographics: Aligning Role-playing M.-Meb Agents Using Human Belief Networks. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.

---

Published as a conference paper at ICLR 2026
D. Exler, M. Schutera, M. Reischl, and L. Rettenberger. Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters, 2025, Sitv:2555004393.
M. Germann, F. Mendez, J. Wheatley, and U. Serdult. Spatial Maps in Voting Advice Applications: The Case for Dynamic Scale Validation. Acta Politica, 50(2):11-222, 2015.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, et al. The Llama Herd of Models, 2024. 2407:2407.2178..
J. F. Gudiñio, U. Grandi, and C. Hidalgo. Large Language Models (LLMs) Agents for Augmented Democ- racy. Philosophical Transactions of the Royal Society 382(2285),20220100, 2024.
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu. al. DeepSeek-RR R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. 1:Xi::2501.12948
P. Haller, J. Vamvas, R. Sennrich, and L. A. Jäger. Leveraging In-Context Learning for Political Bias Testing of LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
J. Hartmann, J. Schwenzow, and M. Witte. The Political Ideology of Conversational Al: Converging Evidence on ChatGPT's Pro-Environmental, Left-Libertarian Orientation, 2023, urXiv:2301017666
J. Hong, N. Lee, and J. Thorne ORPO: Monolithic Preference Optimization without Reference Model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
E. Hwang, B. P. Majumder, and N. Tandon. Aligning Language Models to User Opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
L. Li, C. Chen, F. Gui, H. Yang, C. Yu, et al. Political-LLM: Large Language Models in Political Science, 2024. arXiv:2412.06864.
L. Ning, L. Liu, J. Wo, N. Wu, D. Berlowitz, S. Prakash, et al. User-LLM: Efficient LLM Contextualization with User Embeddings, 2024. arXiv:2402.13599 135988
OpenAL. Introducing OpenAl o3 and 04-mini, 2025. int o3---nn-----nn-a- 04 -mini/ Accessed September 2025.
URL https://openale com index/
J. S. Park, C. Q. Zou, A. Shaw, B. M. Hill, C. Cai, M. R. Morris, et al. Generative Agent Simulations of 1,000 People, 2024. arXiv:2411.10109.
W. Qi. H. Lyu, and J. Luo. Representation Bias in Political Sample Simulations with Large Language Models, 2024. 4XXv:2407.11400)
Qu and J. Wang. Performance and Biases Large Language Models in Public Opinion Simulation. Hu- manities and Social Sciences Commuiccations, 11(1):1095, 2024.
A. Rastogi, Q Jiang. A. Lo, Berrada, Lample, J. Rute, et al. Magistral, 2025. arXiv:2506.10910.
D. Rozado. The Political Preferences of LLMs. PLOS ONE, 19())=e030621,, 2024.
S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, and T. Hashimoto. Whose Opinions Do Language Models Reflect? In International Conference on Machine Learning (ICML) 2023.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms, 2017. 1XXx:1707.06347
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, et al. SeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. arXiv:2402.03300.
D. Stammbach, P. Widmer, E. Cho, C. Gulcehre, and E. Ash. Aligning Large Language Models with Diverse Political Viewpoints. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.
S. Sun, E. Lee, D. Nan. X. Zhao, W. Lee, B. J. Jansen, et al. Random Silicon Sampling: Simulating Human Suh-Population Opinion Using a Large Language Model Based Group-Level Demographic Information, 2024. S5v:22002.18144
12

---

Published as a conference paper at ICLR 2026
Y.M. Tseng, Y.-C. Huang. T.-.. Hsiao, W.-L. Chen, C.-W. Huang, Y. Meng, al. Two Tales of Persona in LLMs: A Survey olaying and Personalization. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024.
Wang, Zhao, D. Ones, L. He, X. Xu. Evaluating Ability of Large Language Models to Emulate Personality. Scientific Reports, 15(1):519, 2025.
Wei, X. Wang. Schuurmans, Bosma, lchter, Xia, al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, et al. Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models, 2025. arXiv:2501.0966
A. Yang, B. Zhang. B. Hui, B. Yu, C. Li, D. Liu, et al. Qwen3 Technical Report, 2025. S.Xi::2505.09385
C. Yu, Li, Z. Li, Ferrara, Hu, Large-Seale Simulation on Large Language Models for Decision- Decision-Making in Political Science, 2025, atXiv:2412.15211.
13

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
In the following, we provide more information on the datasets used in this work.
A.1 SWISS PARTIES
Table lists the Swiss parties included in the smartvote dataset, along with their political orientations and group assignments.
Table The six major Swiss parties and their political positions. All parties' candidates answered the same 60 questions (48 train / 12 test).
Party
Political Orientation
Group
SVP
Right-wing conservative
Right
SP
Center-left social democratic
Left
FDP
Center-right liberal
Right
The Center
Center/Christian democratic
Center
Green Party
Left-wing green
Left
GI.P
Center-left green liheral
Center
A.2 WAHL-O-MAT
Table 5 lists the German parties included in the Wahl-o-Mal dataset, along with their political orien- tations, group assignments, well the train and test set statistics.
Table 5: Number of questions per party in the German dataset across all elections (2021-2025). The
test sets contain 30 questions for all parties. + Yes, B - No, C - Neutral Party
Political Orientation
Train Q.
Train. distr. Yes/Neutral/Noo
Test distr.
Group
Yes/Neutral/No
CDU/CSU
Center-right Christian democratic
646
32.32/99.16 8.51
60.0 36.67 3.33
Right
SPD
Center-left social democratic
760
55.92 32.63/ 11.45
63.33 /3.33
33.33 /3333
Center
Grüne
Center-left green
722
52.08 37.40/ 10.53
73.33
20.0 6.67
Left
FDP
Center-right liberal
760
48.68 / /42.37 8895
46.67
53.33 0.00
Center
Die Linke
Left-wing socialist
646
52.17 / /1.80/ 6.04
56.67
40.0 3.33
Left
AfD
Far-right populist
722
45.43 48.20/ 6.37
30.0 66.67 / 3.33
Right
A.3 ANES
Table 6 provides the assignment of ANES ideologies to groups.
Table Assignment of ANES ideology self-placement answers to groups.
Ideology
Group
Extremely liberal
Left
Liberal
Left
Slightly liberal
Center
Moderate
Center
Slightly conservative
Center
Conservative
Right
Extremely conservative
Right
A.4 PROMPTS FOR SYNTHETIC DATASETS
The following prompt used generate the synthetic argument datasets.
14

---

Published as a conference paper at ICLR 2026
You are [politician voter] in [COUNTRY]. You reply in English only. We present you with a political issue question, and you provide with three to five diverse yet concise arguments [for / against] this issue question. [ISSUE]
For smartvote and ANES, COUNTRY replaced with Switzerland and the USA, respectively. Similarly, politiciii was used for smartvote, while voter was used for ANES. For the biased datasets used in Section 4.8 the smartvote prompt was changed the following:
You are politician in Switzerland. You reply in English only. We present you with political issue or question, and you provide us with [progressive conservative] argument [for / against] this issue or question. [ISSUE]
B ADDITIONAL REASONING TRACES
In most cases, the reasoning traces on-topic, align with the final answer, and are internally consistent. In Table 7 we present an example from a German party in the Center group (SPD). Here, the agent is also making use of information it acquired, most likely during the SFT stage on the training set.
Table 7: Example of successful reasoning. The reasoning trace and the final answer are consistent.
The agent is also making use of additional information about the party position, most likely acquired during the SFT stage on the training set.
Question: Social programs should be maintained even the cost higher taxes
<reasoning>> are committed social justice and therefore to strong social work and high social spending. For us, social policy always means more than mere benefit payments. We are committed to free childcare for all children up the age of ten years and for senior citizens and nursing services that look after them with respect and dignity. < reasoning>> <answer> A/ans/err>>///>> answer>
Agents sometimes resort to rather unconventional arguments. For instance, in the trace in Table g an agent trained on left-wing politician from Switzerland suggested that non-proliferation could suffer Switzerland were to build new nuclear power plants. While the argument and the reasoning are consistent, is rather unlikely that this argument would be made.
Table 8: Example of reasoning trace where the reasoning is formally correct, but the argument made is unlikely to be made in this context.
Question: Should the construction of new nuclear power plants in Switzerland be allowed again?
<reasoning Switzerland has historically been committed a policy non-pon prolliifeaationn of nuclear weapons. Allowing new nuclear power plants could potentially compromise this policy, as there a risk that nuclear materials or technology could be diverted or transferred to other countries with questionable intentions, potentially contributing the development of nuclear weapons, < </wssoniiiggggggge > <answer >B</answer
Similarly, in Table 9 the agent starts in the right direction for the unit it is trained to represent, but then switches topics prostitution.
C ADDITIONAL FIGURES
Figure & provides per-party F1 scores, similar to Fig.
15

---

Published as a conference paper at ICLR 2026
Table 9: Example of trace where the model goes off-topic. This is most likely prompted by the presence of the word "sex".
Question: The legalization of same-sex marriages good thing.
future, there should no further debates on equality homo and heterosexual couples. There must also be further prostitution-free areas to protect the neighborhood from the influence street prostitution. < </reasoning>
<reasoning>>
<answer>A</ answer>>
smartvote
Wahl-o-Mat
ANES
1.0
0.5
1.0
0.8
0.6
0.6
0.4
0.4
0.0
0.0
0.0
Graze
GLP
Mile
FDP
Svp
Die Lake
Grine
SPD
FDF
coursu
ADD
mederate
alight catk
Domirrvtive
cst ctm..
est.
slight
xfrigrpo
Stpo
orpa
Figure 8: Same data as Fig. broken down by party. The party-level data shows a similar trend, SFT+GRPO generally outperforms SFT and GRPO. Fl tends to be better on left and center-left parties, except the German center-right FDP.
D ANES RECODING SCHEMES
Questions that ask for (dis- Jagreement a Likert scale were left is, and the answer options mapped such that all agrecing options become "Yes" or "Agree", and all disagreeing options become "No" "Disagree". Options like "Neither agree disagree" or "None of the above" mapped to neutral. All other types of questions were rewritten to allow for this answer style. For instance, consider the following original ANES question:
When women demand equality these days, How often are they actually seeking special favors?
1.
Always
Most the time
3. About half the time
4. Some of the time
5. Never
This question can rewritten as a statement like:
When women demand equality these days, they are actually seeking special favors.
For recoding the answers, consider two schemes;
Conservative scheme: The original options 1 and 2 become Yes, option 3 becomes Neutral, and options and are mapped No. Generally, options in the middle of the scale, like moderatell important, moderate amount, about half of the time, etc., are mapped to Neutral under this scheme.
Aggressive scheme: Options to 4 mapped to Yes and only option 5 is mapped to No. Generally, only clearly negative options are mapped to No under this scheme. There is no Neutral option, unless the original options contain one.
16

---

Published as a conference paper at ICLR 2026
The conservative scheme treats only clearly positive frequency responses agreement, while the aggressive scheme treats any non-zero frequency as agreement.
D.1
FURTHER EXAMPLES
The question
How often do Members Congress change their votes on legislation because someone donates money to their campaign?
1. Never
2. Rarely
3. A moderate amount of time
4. Very often
5. All the time
becomes
Do Members of Congress change their votes legislation because someone donates money to their campaign?
with mappings (4, 5) Yes (3) Neutral, (1, 2) + using the conservative scheme, and (2,3,4,5) Yes and (1) + No using the aggressive scheme. As an other example, the question
Does the increasing number of people of many different races and ethnic groups the United States make this country better place to live, worse place to live, or does it make no difference?
1. Better
2. Worse
3. Makes no difference
becomes
Does the increasing number of people of many different races and ethnic groups in the United states make this country better place?
with mappings (1) Yes, (2) No and (3) + Neutral in both recoding schemes.
E ADDITIONAL MATERIAL
The following sections provide additional material contextualize the findings in the main text.
E.1
ADDITIONAL RESULT TABLES
Table 10
presents accuracy scores across models and configurations. Similarly. in Table II present accuracy scores without the Neutral class. In Table 12 and Table [3 we report one- tailed Welch's t-tests (SFT+GRPO > baseline) and Cohen's 's effect sizes for the results in Table (macro-F1) and Table 10 (mean accuracy), respectively.
E.2 IMPACT OF RECODING ANES
Since the recoding is not unique and can potentially lead to information loss and semantic shifts, investigate the impact of different recoding schemes the political groups present in the ANES dataset. Details on the different schemes are provided in Appendix D The key difference lies in how moderate responses treated: the conservative scheme maps them neutral/disaarremmnt,,
17

---

Published as a conference paper at ICLR 2026
Table 10: Mean accuracies (%) with standard deviations, averaged over 8 stochastic decoding runs at temperature T=1.0. For each run, we first compute the per-unit accuracy (candidate/partyyie spondent) over that unit's test items and then average across units. The table reports the across-run mean + s.d. Baselines: random (uniform; 50.0/33.3 on binary/ternary labels by construction) and majority (deterministic). Bold indicates the best method per base model and dataset, with signif-
icance reported in Table
Base model
Method
Dataset
smartvote
Woll
ANES
random
50.00
33.33 27.44
33.33 22.98
Untrained baselines
ma majority
37.43
icl ORPO SPI GRPO
63.91 3.79
31.44
0.02
44.39
1.43
59.23
2.47
57.48
5.97
44.15 _ 5.61
Llama 3.1
67.23 1.00
68.25
4.30
58.82 + 0.93
66.44 0.86
60.63
52.35
1.35
SFT+GRPO
70.53 + 1.51
75.06 + 3.30
59.21
= 0.62
iel
66.09 2.85
48.67
0.73
41.32
2.98
ORPO
39.02
2.04
38.56
3.51
34.07 + 2.08
Qwen3 8B
SFT
65.18
1.91
61.74
4.63
47.01 + 1.79
GRPO
67.04 1.69
53.19
2.33
52.92
0.72
SFT+CRPO
71.27 + 2.83
71.16 + 1.60
55.14 + 0.41
icl
71.41 3.54
44.64
0.67
42.46
5.13
Magistral 24B
ORPO
35.93
3.66
36.10 - 5.18
31.32 + 2.23
SPT
70.83
1.58
72.56
2.40
52.80
0.94
GRPO
68.44
1.23
72.05
2.99
62.33
0.51
GRPP
73.92 + 1.89
75.10
+ 2.80
62.33 + 0.78
Table 11: Mean accuracies (%) with standard deviations when Neutral is removed. Results are the averages over 8 stochastic decoding runs temperature T=1.0. For each run, we first com- pute the per-unit accuracy soanddddate/patyyrssppone
over that unit's test items and then average across units. The table reports the across-run mean = s.d. Basclines: random (uniform; 30.0/33.3 on binary/ternary labels by construction) and ma jority (deterministic). Bold indicates the best method per base model and dataset (higher is better). smartvote results are unchanged from Table T0 because there are no neutral answers on smartvote.
Base model
Method
Dataset
smartvote
WoM
ANES
Untrained baselines
random
50.00
33.33 27.44
33.33 22.98
ma majerity
37.43
icl ORPO SFT GRPO SFT+GRPO
63.91 = 3.79
31.03 +
0.19
47.15 + 1.69
Llama 3.1
59.23 2.47
59.28
6.15
44.15
5.61
67.23 0.10
70.11
4.36
62.47
0.92
66.44
0.86
62.93
1.54
64.15 +
1.51
70.53 1.51
78.19
3.38
71.35
icl
66.09 + 2.85
31.03 + 0.19
19.11
2.93
Qwen3 8B
ORPO
39,02 2.04
37.98
3.29
37.04
1.49
SFT
65.18
1.91
63.62
1.74
54.38
2.03
GRPO
67.04
1.69
55.30 + 2.44
65.38 1 0.91
SFT+GRPO
71.27 2.83
73.69
1.60
67.55 0.72
icl ORPO
66.09 2.85
46.62
0.80
46.27 + 5.94
Magistral 24B
35.93
3.66
36.39
5.38
33.67
1.86
SFT
70.83
1.58
74.71
2.71
62.67 + 1.17
GRPO
68.44 1.23
77.21
+
2.72
66.34
0.55
SFT+GRPO
73.92 1.89
74.71
2.71
69.99
0.85
while the aggressive scheme treats most non-zero responses as agreement. We train Llama 3.1 8B on the two recoding schemes and find that mean Fl-scores the aggressive scheme are higher than on the conservative one (45.33 = 1.37 vs. 40.67 + 0.93 reported Table 3). The confusion matrices in Fig. indicate that the improvement is mostly driven lower number of Neutrals in the aggressive scheme. These results highlight that recoding choices can substantially impact both model performance and the political diversity represented in simulated populations.
18

---

Published as a conference paper at ICLR 2026
Table 12: Statistical significance and effect sizes for SFT+GRPO improvements in macro-F1. Results correspond to the experiments reported in Table 3 We perform one-tailed Welch's 's t-tests (SFT+GRPO baseline) and report Cohen's d effect sizes. Green: significant after Bonferroni correction (p < 0.0014 for 36 tests). Blue: significant without correction (p < 0.05). Red: not significant.
P-values
Cohen's
Model
Bascline smartvote
WoM
ANES
smartvote
WoM
ANES
icl
0.0003
0,0000
0.0000
2.54
8.51
12.30
Llama 3.1
ORPO
0.0000
0,0008
0.0093
9.29
1.99
1.50
SFT
0.0013
0.0409
0.9991
1.90
0.94
-1.95
GRPO 0.0000
0,0000
0.0000
6.61
4.74
5.39
icl 0.0140
0.0000
0.0000
1.23
16.72
11.75
Qwen3 8B
ORPO
0.0000
0.0000
0.0000
14.87
8.39
11.46
SP'
0.0099
0.0004
0.0004
1.32
2.32
2.67
GRPO 0.0037
0.0000
0.0000
1.63
10.50
8.70
icl
0.0102
0.0000
0.0000
1.36
11.91
11.90
Magistral 24B
ORPO
0,0000
0.0000
0.0000
21.95
8.79
11.81
SFT 0.0049
0.1842
0.0000
1.50
0.47
6.88
GRPO
0,0000
0.0909
0.0046
4.90
0.70
1.51
Table 13: Statistical significance and effect sizes for SFT+GRPO improvements in accuracy. Results correspond to the experiments reported in Table TO We perform one-tailed Welch's 's t-tests (SFT+GRPO baseline) and report Cohen's d effect sizes. Green: significant after Bonferroni
correction (p < 0.0014 for 36 tests). Blue: significant without correction (p < 0.05). Red: not significant.
P-values
Cohen's
Model
Baseline
smartvote
WoM
ANES
smartvote
WoM
ANES
icl
0.0006
0,0000
0.0000
2.29
18.66
13.45
Llama 3.1 8B
ORPO
0.0000
0.0000
0.0001
5.52
3.64
3.77
SPT
0.0001
0.0017
0.1714
2.58
1.78
0.49
GRPO 0.0000
0.0000
0.0000
3.33
5.63
6.53
icl
0.0013
0,0000
0.0000
1.82
18.09
6,50
Qwen3 8B
ORPO
0.0000
0.0000
0.0000
13.07
11.95
14.06
SFT
0.0001
0.0002
0.0000
2.52
2.72
6.26
GRPO
0.0019
0.0000
0.0000
1.81
8.99
3,79
0.0527
0.0000
0.0000
0.88
14.96
5,42
Magistral
ORPO
0.0000
0.0000
0.0000
13.04
9.37
18.56
SFT 0.0017
0.0361
0.0000
1.77
0.97
11.03
GRPO
0.0000
0.0269
0.5000
3.44
1.05
0.00
E.3 REGRESSION TABLES
Table 14 provides the regression table for the regression shown in Fig. 4
Similarly, Table pro-
vides the regression table for the corresponding analysis shown in Fig. 10
Table 14: Regression table for the analysis presented Fig. Neutral l. base rate.
and Section 4.6
Model performance
Metric
N
Intercept (a)
Slope (f)
Slope SE
Slope 95% CI
FI Score (Macro) Accuracy
21
0.5517
-0.6741
0.2091
[-1.3039, -0.4084] 11.1117,-0.236,,
21
0.7803
-0.8562
0.2139
Metric
R2
p-value
RMSE
FI Score (Macro)
-0.5946
0.3535 0.4574
0,004472
0.1250
Accuracy
-0.6763
< 0.001
0.1279
19

---

Published as a conference paper at ICLR 2026
(1UAO
121.4)
274
0.0%
104
(2)
(1.9)
218
(2.5)
07.2%
Predited
Figure 9: Confusion matrices for Llama 3.1 8B (SFT+GRFO) trained on ANES with a more aggres- sive recoding scheme. - Yes, B + No, C + Neutral.
1.0
0.8
Neutral Recall
0.6
0.4
0.2
Yes
No
Neutral
0.8
0.8
+0.079
Accuracy
0.6
0.6
+0.082
#0.167
0.4
0.4
0.2
0.2
0.00
0.05
0.10
0.15
0.20
0.25
Left
Center
Right
responses
77
Neutrals removed
Figure 10: Fl score (A) and accuracy (B) Neutral response rate. Predictive performance decreases significantly as Neutral response rate increases. Comparing results presented here to those in Fig. 4 we observe the same trend regardless of the coding scheme. (D): Performance in terms Fl score on ANES before (left) and after (right) removing Neutral instances. All groups improve, but Center only marginally. The gap between Center and Right closes. Performance in predicting Neutral decreased (C). Results were obtained for Llama 1.1 8B using the SFT+GRPO configuration.
E.4 IMPACT OF THE NUMBER OF TRAINING QUESTIONS
We ablate the number of questions in the WoM training set for the SFT+GRPO configuration. The results for Llama 3.1 8B are presented in Fig. [1 Increasing the number of training questions is beneficial for all parties, except the center-right FDP party. The FDP's performance may be explained by shift the party's 's positions due dwindling voter numbers between 2021 and 2025. The largest increase in accuracy is observed for the far-right AID, which we attribute to the fact that their positions are furthest from the LLM's s position, and SFT thus has an aligning or "corrective" effect which becomes stronger as more data is added.
E.5
PCA DETAILS
we compute the PCA as follows: We use the complete smartvote dataset for the 2023 national elections across all major parties and all candidates. We convert the original 4-point Likert seale answers to yes-no, as we did for the smartvote subset studied in the remainder of the paper. Yes answers subsequently mapped to 1, and No answers to O. We then compute the PCA of the resulting matrix, where the rows correspond to candidates, and the columns correspond to questions.
To obtain the results shown in Fig. 2 and Fig. 12
20

---

Published as a conference paper at ICLR 2026
Table 15: Regression table for the analysis presented in Fig.
TO
Model performance vs. Neutral
base rate.
Metric
N
Intercept (a)
Slope (3)
Slope SE
Slope 95% CI
Fl Score (Macro)
21
0.6225
-2.2231
0.5049
[-3.2799, -1.1662 -2249981, -0.6161
Accuracy
0.7157
-1.5571
0.4496
Metric
r
R²
p-value
RMSE
Fl Score (Macro)
0.7106
0.5050
< 0.001
0.1608
Accuracy
0.6221
0.3870
0.002603
0.1431
Die Linke
Grüne
SPD
FDP
CDUICSU
ATD
0.8
115000
04
Propartion
Figure 11: Impact of the number of questions in the training set. Shown accuracies for 10%, 25%, 50%, 75%, and 100% of available training questions per party on the WoM dataset. Overall, F1 increases the number of training questions increases. The largest increase is observed for the far-right AfD, while the FDP demonstrates a counterexample.
For agents, we replace the ground-truth test set answers with the agents' predictions, and project resulting vectors onto the first two principal components. Similarly, for the inverted positions shown in Fig. 12
we invert all ground-trun -ruth answers and project the resulting vectors.
E.6 INVERSION
Figure 12 shows candidate positions before and after flipping all answer options. As can be seen, this operation roughly corresponds to rotation by 180% about the origin.
Left
Center
Right
Conservative- Liberal
Conservative Liberal
Left-Right
Figure 12: Inverting all answers in the dataset roughly corresponds to rotation by 180 about the
origin. Top: Candidates before inversion. Bottom: Candidates after inversion.
21

---

Published as a conference paper at ICLR 2026
F TRAINING DETAILS
In the following, we provide information on training settings and hyperparameters.
System prompt.
We used the following system prompt
digital twin of [NATIONALITY] voter. You are asked a policy issue or question. You must reason and then answer the question as if you were the voter. You reason and answer in English. Your final answer must one of the answer options ("A" for "Yes", "B" for "No", "C" for "Neutral"). You must respond in the following format:
You
<reasoning>
Your reasoning goes here.
</reasoning>
<answer>
Final answer, of
A) Yes,
No,
C) Neutral. </answer>
where we replaced [NATIONALITY] with Swiss for smarlvote, German for Wahl-o-Mat, and US for ANES.
LoRA settings. We the following LoRA settings for all models and methods. No hyperpa- rameter optimization was performed. r r= 32, a = 32, target modules: q-proj, k.proj, v.proj, o.proj, gate.proj. up-proj, down.proj.
ORPO. We use the same hyperparameters as Stammbach et al. (2024) and the Hugging Face im- plementation ORPO.
SFT. We train for 800 steps using the Adam optimizer with batch size of 8, learning rate 5e - 5, a cosine learning rate scheduler, and we fix the maximum gradient norm at 1.0. The first 80 steps were used warm-up period. For prettrind models, change the treasoning><// wexx//easoningg tags to their respective reasoning templates when fine-tuning. We used the Hugging Face implementation of SFT, and all other hyperparameters were left the default values.
GRPO. We train for 800 steps using the Adam optimizer with batch size of 8, learning rate 5 x 10-6, a cosine learning rate scheduler, and we fix the maximum gradient norm at 1.0. We use the Adam optimizer with 31 0.9 and 32 = 0.99, weight decay with 0.1. The first 80 steps were used as warm-up period. The GRPO group size was 8. The GRPO 3 parameter pretrained models, we change the
was set to O. The temperature was set at 1.0. For reasoning-pretrainee <reasoninges wetooningnn//raseeeeeeee htt//raasonnngg tags to their respective reasoning templates in the format of ward functions. We used the Hugging Face implementation of GRPO, and all other hyperparameters were left at the default values.
SFT+GRPO. For this configuration, we used the same settings as for the individual SET and CRPO configurations.
22


Reviews:
[{"review_id": "ExXjAzzscU", "reviewer": "Reviewer_ywJy", "summary": "This paper explores whether explicit reasoning improves *individual-level* opinion alignment in LLMs. The task is to predict a persona’s stance $(y\\in{\\text{Yes, Neutral, No}})$ on survey questions (q), given the persona’s past answers. The model outputs `<reasoning>...</reasoning><answer>...</answer>` and is trained with GRPO (group-relative PPO) using a composite reward for format correctness, rationale length, and answer agreement. An optional SFT warm-start with synthetic rationales is applied before GRPO. Experiments span smartvote (CH), Wahl-o-Mat (DE), and ANES (US), showing that SFT+GRPO improves macro-F1 (e.g., 70.7 on smartvote) and that Neutral responses and right-leaning personas are harder to predict. Code and data are released.", "questions": "1. Please compare against at least one specialized political-alignment baseline (e.g., Stammbach et al. 2024) and/or ORPO-style preference optimization.\n2. Can you provide a *shared model* baseline that conditions on persona embeddings instead of per-persona training?\n3. How sensitive are results to the synthetic SFT data source?\n4. Can you expand ANES to more respondents and report per-respondent confidence intervals?\n5. How did you prevent test leakage from SFT synthetic data?\n6. Could you include calibration metrics (ECE, coverage-accuracy)?\n7. What is the compute footprint per persona for GRPO training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Clear, reproducible method and reward design.** The structured format and composite reward are explicit, with training hyperparameters (LoRA, steps, schedulers) fully reported.\n* **Multiple datasets and ideologies.** The paper evaluates on smartvote (binary), Wahl-o-Mat (ternary), and ANES (ternary), with clear unit definitions and splits.\n* **Variance reporting across 8 stochastic runs.** Results include mean ± s.d., improving robustness.\n* **Insightful analyses of failure modes and class imbalance.** Figures analyzing ideology and Neutral class are carefully interpreted; removing Neutral boosts F1 but preserves ideological gaps.\n* **Resource release.** Code and datasets are linked, supporting reproducibility and follow-up work.", "weaknesses": "1. **Missing strong baselines for political alignment.** The paper omits comparisons against recent specialized methods that align LLMs to political viewpoints using supervised preference optimization or domain corpora (e.g., Stammbach et al. 2024).\n\n2. **Limited persona-conditioning alternatives.** The method trains *one model per persona*, not comparing to shared-parameter persona embedding methods.\n\n3. **Synthetic SFT rationales risk leakage/priors.** Synthetic arguments generated by a large LLM may encode political priors.\n\n4. **Scope of SOTA claim is limited.** No head-to-head with specialized political fine-tuning models.\n\n5. **Evaluation depth on ANES is modest.** Only 21 respondents modeled, limiting statistical power.\n\n6. **Fairness and calibration analyses are limited.** The paper lacks per-persona calibration or reliability metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4DYJ15Obg", "reviewer": "Reviewer_chku", "summary": "This paper investigates whether reasoning-based approaches can improve LLMs' ability to model individual political opinions. The authors train models using reinforcement learning (specifically GRPO - Group Relative Policy Optimization) to generate reasoning traces before providing political stances, using survey data from three political contexts: Swiss candidates (smartvote), German parties (Wahl-O-Mat), and U.S. voters (ANES 2020). The key contribution is showing that structured reasoning through SFT+GRPO outperforms baseline approaches, though significant challenges remain with neutral stances and right-leaning political positions. The work establishes a benchmark for opinion modeling and releases datasets for future research.", "questions": "The current pipeline does SFT on synthetic arguments generated by Llama 3.1 70B. How sensitive is the final SFT+GRPO model to the style of these synthetic arguments? For example, if the initial CoT is written in a more partisan tone, will GRPO converge to a different political center?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First systematic study of using RL-based reasoning for individual-level political opinion modeling, moving beyond demographic-based approaches\n2. Three datasets from different political systems (US, Germany, Switzerland) provide robust cross-cultural validation\n3. The paper identifies a gap between demographic-prompted political simulation (Santurkar et al., 2023; Argyle et al., 2023) and individual-level agents that must stay consistent with a known survey profile.", "weaknesses": "1. Training one model per individual is computationally prohibitive for real-world deployment\n2. The method still trains one model per persona, which is expensive and does not scale to population-level simulation; the paper acknowledges this, but the main method remains hard to apply in real digital-democracy settings where thousands of agents are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rDGHAJti2c", "reviewer": "Reviewer_mY5k", "summary": "This paper looks at whether reasoning helps align LLMs with human political opinions. The authors fine-tune LLMs on political survey data from the US, Germany, and Switzerland, and then train them further using Group Relative Policy Optimization (GRPO) — a reinforcement learning method that encourages structured reasoning before giving an answer. The goal is to make models answer policy questions like real individuals or parties would. Experiments show that reasoning-based training (SFT+GRPO) outperforms prompt-only and SFT-only baselines, improving F1 scores across all datasets. Overall, the paper argues that reasoning improves opinion alignment but isn’t enough to eliminate political bias entirely.", "questions": "1. Could you please provide a more detailed explanation of Figure 4, especially the panels on the right? I would like to understand what those plots specifically represent and what the error bars indicate in this context.\n\n2. Can the authors show evidence that GRPO improves reasoning quality rather than just accuracy?\n\n3. Did you analyze why right-leaning or neutral profiles are harder to learn — is it data imbalance or model bias?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper studies a very relevant question — how reasoning affects opinion alignment, which feels timely and important.\n\n2. The method is clear and builds nicely on recent RL reasoning techniques.\n\n3. The experiments are well-organized and tested across multiple real-world political datasets.\n\n4. The analysis is thoughtful, especially the discussion of ideological bias and neutral-class difficulty.", "weaknesses": "1. The clarity of paper should be further refined, and there are several typos in the content.\n\n2. It’s not entirely clear whether GRPO genuinely improves reasoning or just fits survey patterns better.\n\n3. The paper shows biases but doesn’t really explain their sources or propose fixes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8cOWTtmN39", "reviewer": "Reviewer_zvbd", "summary": "The authors present a novel approach toward aligning LLMs with individual opinions in a political context, which could enable downstream applications such as more representative digital democracies. In contrast to prior work that focuses on in-context learning, they use reinforcement learning to train models to align to the political opinions of individual candidates, finding gains in opinion alignment when compared to certain baselines. They also perform a detailed analysis of results and find that models are more easily aligned to left-wing political groups, but that trained agents tend to lean more centrist.", "questions": "- How does opinion alignment scale with the number of questions personalized models are trained on (SFT, GRPO) or prompted with (ICL)? Analyzing the scaling behavior could help us understand what kind of approaches are most likely to work with current datasets, or inform the collection of future datasets.\n- Why do you think your trained agents are generally more centrist and conservative than the opinions they are trained on (Figure 2), and how does this vary across different training/prompting settings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors identify a key problem in opinion alignment that current prompting-based approaches fail to adequately solve, and that could enable highly prosocial applications if solved, and propose a novel solution.\n- I found the political ideology analysis (sections 4.5, 4.7, 4.8) really interesting, especially the experiment with flipped labels in 4.8 that showed that left-leaning opinion profiles are easier to learn than synthetic left-leaning profiles constructed by inverting right-leaning opinion profiles. The findings that right and center positions are harder to learn than left ones, and that fine-tuning tends to push agents to the center, make interesting additions to the dialogue around opinion modeling with LLMs.", "weaknesses": "- The novel contribution of this work (using RL to train models to reason for opinion alignment) doesn’t seem to yield much improvement over SFT on seen opinions. In Table 3, SFT+GRPO is only a statistically significant improvement over SFT in three of the 12 base model - dataset combinations (Qwen3 8B + WoM, Qwen3 8B + ANES, Magistral 24B + ANES). The authors note that this relates to dataset scale, but SFT+GRPO doesn’t seem to consistently improve over just SFT even on the largest WoM dataset. Given the difficulty of collecting individual opinions, it may be difficult to apply the proposed method to new users.\n- Given the amount of prior work on opinion modeling with LLMs in survey environments, I found “Our work lays the groundwork and introduces the first benchmark for systematic opinion modeling with LLMs” to be a bit of an overclaim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)