Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability
Authors: Taylor Sorensen
Benjamin Newman
Jared Moore
Chan Young Park
Jillian Fisher
Niloofar Mireshghallah
Liwei Jiang
Yejin Choi
Venue: ICLR 2026 Poster (2026)
Keywords: post-training
language models
distributional learning
alignment
pluralistic alignment
uncertainty estimation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We contribute a novel dataset and post-training method to improve in-context steerability and distributional alignment and coverage, and characterize weaknesses to current post-training techniques along these desiderata.

Abstract: Language model post-training has enhanced instruction-following and performance on many downstream tasks, but also comes with an often-overlooked cost on tasks with many possible valid answers. On many tasks such as creative writing, synthetic data generation, or steering to diverse preferences, models must cover an entire distribution of outputs, rather than a single correct answer. We characterize three desiderata for conditional distributional modeling: in-context steerability, valid output space coverage, and distributional alignment, and document across three model families how current post-training can reduce these properties. In particular, we disambiguate between two kinds of in-context learning: ICL for eliciting existing underlying knowledge or capabilities, and in-context steerability, where a model must use in-context information to override its priors and steer to a novel data generating distribution. To better evaluate and improve these desiderata, we introduce Spectrum Suite, a large-scale resource compiled from $>40$ data sources and spanning $>90$ tasks requiring models to steer to and match diverse distributions ranging from varied human preferences to numerical distributions and more. We find that while current post-training techniques elicit underlying capabilities and knowledge, they hurt models' ability to flexibly steer in-context. To mitigate these issues, we propose Spectrum Tuning, a post-training method using Spectrum Suite to improve steerability and distributional coverage. We find that Spectrum Tuning often improves over pretrained and typical instruction-tuned models, enhancing steerability, spanning more of the output space, and improving distributional alignment on held-out datasets.

Published as a conference paper at ICLR 2026
WA SPECTRUM TUNING: POST-TRAINING FOR DISTRIBUTIONAL COVERAGE AND IN-CONTEXT STEERABILITY
Taylor Sorensen', Benjamin Newman', Jared Moore", Chan Young Park Jillian Fisher', Niloofar Mireshghallah" Liwei Jiang', Yejin Choi?
University of Washington, Stanford University, Miirosooft Research, *Carnegie Mellon University Correspondence: t bor130css www.iwggto.. edu, yejincestanfordee edu
Code and Dataset: httpcchcccmm///oorl/g//ccctommm
Models:
co/
ABSTRACT
Language model post-training has enhanced estruction-followinng and perfor- mance on many downstream tasks, but also comes with an often-overlooked cost tasks with many possible valid answers. On many tasks such as creative writ- ing, synthetic data generation, or steering to diverse preferences, models must cover entire distribution of outputs, rather than single correct answer. We characterize three desiderata for conditional distributional modeling: in-context steerability, valid output space coverage, and distributional alignment, and docu- ment across three model families how current post-training can reduce these prop- erties. In particular, we disambiguate between two kinds of in-context learning: ICL for eliciting existing underlying knowledge or capabilities, and in-contexi steerability, where model must use in-context information override its priors and steer to a novel data generating distribution. To better evaluate and improve these desiderata, we introduce SPECTRUM SUITE, large-scale resource compiled from >40 data sources and spanning >90 tasks requiring models to steer to and match diverse distributions ranging from varied human preferences to numerical distributions and more. We find that while current post-training techniques elicit underlying capabilities and knowledge, they hurt models' ability to flexibly steer in-context. To mitigate these issues, we propose SPECTRUM TUNING, post- training method using SPECTRUM SUITE to improve steerability and distribu- tional coverage. We find that SPECTRUM TUNING often improves over pretrained and typical instruction-tuned models, enhancing steerability, spanning more of the output space, and improving distributional alignment on held-out datasets.
INTRODUCTION
Current post-training recipes (Rafailov et al.) 2024 Tie al. 2025 Wang al. 2025) have made language models (LLMs) easier to use via pnstruction-folloiii (Ouyang et al. 2022). improved safety, and led to performance increases across many tasks, especially those with a single correct answer (e.g., mathematical reasoning, programming, chat preferences, etc.). However, the effect of current post-training on tasks requiring steerability and distribution matching is less studied. We show that current post-training can also negatively impact three related desiderata for conditional distributional modeling: in-context steerability, output coverage, and distributional alignment.
In this paper, we contribute: 1) an outline of these related desiderata, including the novel concept of in-context steerability; 2) SPECTRUM SUITE, a dataset for evaluating and enhancing these desider- ata; 3) a novel finding that while current post-training helps at many objective tasks, it can hurt LLMs' in-context steerability; and 4) empirical evidence from our and related work that current pust-training hurts output coverage and distributional alignment. To alleviate these weaknesses, contribute 5) SPECTRUM TUNING, a post-training technique utilizing SPECTRUM SUITE to improve these desiderata, and 6) show that method enhances these properties compared to pretrained and

---

Published as a conference paper at ICLR 2026
current instruction tuned models. To our knowledge, method is the first to improve distributional alignment over pretrained models.
DESIDERATA FOR CONDITIONAL DISTRIBUTIONAL MODELING
Before the age of post-training in co-tontext learning was necessary to reliably get pretrained language models to perform tasks such as sentiment classification, translation, entailment, summarization, etc. (Bronn et al.] 2020 Dong et al.) 2024). Let us call this use of in-context learning capability elic- itation, as its main purpose is to elicit some latent knowledge or capability of a language model MMin et al. 2022b). As post-training methods have increased LLMs' cestruction-folloiinn capabil- ity, shot instruct models have even surpassed their few-shot pretrained counterparts (Wei al.) 2022
Sanh et al.
2022 Ouyang al. 2022). obviating the need for in-context capability elicitation.
In-Context Steerability. In contrast to knowledge elicitation, many tasks require steering, or mod- ifying output probabilities, based novel information at inference time. For example, if user wants an L.M to write an email in their style, needs to either see examples of their writing or have an in-depth description of their style, and be able to effectively leverage this information to change its output distribution. This is distinct from pure capability/knowledge elicitation on unambiguous tasks, where the model can place a sharp prior on the "correct" answer. Instead, the model must 1) maintain prior over many possible generation functions and 2) maximally leverage in-context in- formation in a well- calibrate way to form a posterior. Let us term this ability in-context steerability. For example, this steerability is necessary for predicting particular user's preferences or estimating an unknown numerical distribution from draws. In-context steerability can also be seen as implicit Bayesian reasoning (Qiu et al. 2025) or as subset of in in-context saamiinginstrucioo-ffllwwww tasks where the model must utilize novel information in-context.
Valid Output Coverage. Many prompts entail multiple valid responses. For example, in creative story-writing, hypothesis proposal, and synthetic data generation, the number possible valid out- puts can be thousands or more. While in some cases it may be sufficient to produce output, more value may lie in producing many outputs so that a user can select the most interesting story, test all possible hypotheses, or otherwise span the entire task space. In the words of Wilson & Izmailov (2022). "we want the support of the model to be large so that we can represent any hypothesis we believe to be possible, even if it is unlikely."
reasonable
Distributional Alignment. Sometimes, user may not want particular output, but rather a distri- bution over outputs (Meister al. 2024). For example, [Sorensen al. (2024b) propose distribu- tional pluralism for modeling or representing a population by matching their opinion distribution. In addition, distributional alignment can simulate stochastic processes and estimate uncertainty. Dis- tinct from valid output coverage, distributional alignment includes a target probability mass function.
DATASET AND METHOD
Other
3.1 SPECTRUM SUITE
Opinions (Categorica)
NLP Tasks
To measure and clicit these properties, compile datasets that either 1) exhibit natural person-to-pest pertoo variation (e.g., opinion modeling, chat prefer- ences, subjective NLP tasks); 2) involve a large collection of interchangeable texts drawn from a particular distribution (e.g., synthetic data, poems in a particular format); 3) are i.i.d. draws from a random distribu- tion (e.g., draws from normal distribution); or 4) involve rea- soning under uncertainty. We draw from 40 data sources in order to make > >90 separate tasks. We unify each task data into a common format including: description: a natural language de- scription of the task, input: any given information for particular data instance, and output:
Outbul Type
Percentage laqa)
19.4%
10.7%
2.7%
Chatbot Responses
3.5%
Numner: 1.99%
Chat Preferences
3.7%
Knowledge
Sp6t
# Boyp
6.2%
Synthatic Data Ganeration
Text
11.3k
Generation (Opinion)
12.2%
Individual Modeling
Samples from Known Distribution
Text Generation (Chat)
NLP Systems
Pluratiam
Figure 1: modeling tasks (data from the same person) shaded.
Task composition from SPECTRUM SUITE. Individual
2

---

Published as a conference paper at ICLR 2026
the output sequence which would like the model to learn. Some tasks require input associ- ated with each output (e.g., the question asked in a survey is needed to contextualize the answer), while other tasks consist of only outputs. s. In particular, we focus on individual modeling data on tasks with human variation. We do so for a couple of reasons: many use cases involve steering to a particular individual inference time; and these data sources are very rich as modeling each person involves a different data generation task. These data comprise 50. Ik distinct sequences consisting of a descripp iptipt.. ion followed by multiple input / /cutputsttttt s. For summary statisties and task breakdown of SPECTRUM SUITE, see Figure [1, For information on all data sources, see App. [ We split SPECTRUM SUITE into non-overlapping train and test tasks, with held-out test tasks drawn from separate data sources to ensure generality.
3.2 SPECTRUM TUNING
Let Ti E T be some task (or, data generation process) that we want to model. Let Y; be the out space approximate, X; be any known covariates (optional npuu), and Zj be a latent context for the task (optional descr ip it on). T X; Z44 + P(Y;) maps to a probability distribution over potential outputs. This is the classic meta-learning formulation (Hospedales et al. 2020), except that the target is a distribution over P(Y") instead of a single yı- Because the task T, may be difficult to directly observe, we may instead wish to learn it from data (e.g.. Monte Carlo samples).
The method (Algorithm D is simple: for a collection of tasks, tokenize the task context/description 2; followed by (randomly ordered) in-context ex- amples Tıj. Vig. then perform super- vised finetuning calculating cross- entropy loss only on the output to- kens. Because cross-entropy loss on Monte Carlo samples from a distribu- tion encourages a well-calibrated es- timate of the underlying distribution in the underfit regime (S1 epoch, Ji al.]]2000 (202]) the optimal model solu- tion is to approximate the true under- lying distribution P(Y;).
Algorithm SPECTRUM TUNING
Inputs: Pretrained LM ma: train task distribution T to- kenizer t(-) with template for descrripiion/inputooututtt terminal token (END): loss ignore index 'mpp' descrip- tion drop probability parop (default 0.2).
Output: Finetuned parameters 0'
Sample task 2: Sample description 2 and support set S
I: for each task T' ~ T do
{(rr {(ar+);))}? V;)}
3:
Randomly permute indices of (1...., n).
4:
if Uniform(0, > Pirop then > Keep description seq t- ((=)])((z==0o) (() )1 (END)
5:
6:
labels +- ddrup(t(22) (tx))(())l((<x)0) (()+))))))))))))
> Loss on first output, no loss on description/output
7:
else
Description dropout w/ prob. Pdrop
To build intuition on how SPECTRUM TUNING supports the desiderata, let us consider a few cases. When a model predicts the first output, must rely only on the description, and shift its probabilities to outputs fit- ting the description. Because there can be many possible valid outputs and the model has no information about which output to expect, it is in- centivized to cover the entire possi-
8:
seq 4 (<=x(o))) ([(Tx)())]]((())))) [[( (END)
9:
labels < ddrop (f(# [[+(()=())) (END)) > No
loss first output if description is missing
10:
end if
11:
for in rr do
Add remaining
12:
seq 4 seq || t(x;;) ([( II (END)
13:
labels +- labels édrop (b(z²)) )) ((3;) II (END)
Loss on output, no loss on input
14:
end for
15:
I. +-
(monentttooooymmssee labels)
ble distribution of outputs. Addition- ally, if the distribution over valid out-
16:
0- 0 Vol.
17:
end for
Train for one epoch
puts is skewed in some predictable way (e.g., an opinion distribution). the model is further incentivized to match said distribution. On subsequent outputs, the model must steer its output distribution, utilizing in-context examples to update its beliefs in a well-calibrated way. Additionally, SPECTRUM SUITE tasks allow the model to utilize assumptions which don't tt apply to the pretraining distribution: predictions are invariant to output ordering the underlying generative process remains constant, and the model can concentrate all probability mass valid outputs instead of on other possible text continuations. In many ways, SPECTRUM TUNING is simi- lar to supervised fine-tuning instruction data (Zhang al. 2025c), as loss is calculated only on an output. However, it differs in several important respects: 1) many identically-distriuued outputs are
18:
return 0' 0
l... "exchangeable" in Bayesian analysis Kokolakis 2010 as the posterior is invariant to sample order.
3

---

Published as a conference paper at ICLR 2026
included in-context, encouraging meta-learning; 2) training on data that is distributional in nature; 3) sole focus on distribution fitting instead of chat-style data; and 4) inputs are optional, unlike chat user messages which are always required.
3.3 IMPLEMENTATION DETAILS
We train models from three families using SPECTRUM TUNING on the train tasks from SPEC- TRUM SUITE: gemma-3-12 3-1bb TTam et al.] 2025), Llama-3. 1...1-1- 1-8B (Grattafion et al. 2024), and Qwen3-14B (Yang et al., 2025). We refer to pretrained or base models as PT mod- els and instruction-tuned post-trained models as IT models, and utilize each family's provided РT/IT model as baselines. To match our meta-learning task setup (as opposed to chat), we adapt each model's chat template to use the descriptionn ttteiitiiooniiii innput/utppt system/used
roles instead of (cf. Fig. A22). For SPECTRUM TUNING, we initialize with the PT model weights, except for the uninitialized (un/)embedding weights for the two or three special format tokens which we initialize from the IT model. See App. Gfo for more training details.
Spectrum Suite (Categorical) Relative Accuracy after Instruction-Tuning
4 IN-CONTEXT STEERABILITY
We use SPECTRUM SUITE to evaluate mod- els' ability to steer to varied generation tasks. We measure k-shot learning by I) fitting the description and examples from
lovld 20peamy Coopeice lodd carcen
negethve
a single task into context, 2) measuring the loss (negative liglliiel)hood) of each output conditioned on the prior exam- ples under the model mmg NLLmm (yx) = - log Pm, (yk|2, yo. 11s-1). Addition- ally, for multiple-choice datasets, we cal- culate the accuracy of the output: whether the greedily-decoded model response re- sults in the correct answer. For each task, we choose Kmax such that it maximizes the
Sprikaay Sette
No prniaat ternc Sigriianll Woys
Imii
total number of examples that we can eval- uate when we restrict to only sequences with at least Kmax examples that fit into a 1024-token condext-window. In order to maximize sample efficiency and evaluate a model's 's ability to steer for varied k, we re- port the average loss and accuracy for k- shot learning for k E
Speesom jeciiie
contente
First, we ask: how does current instruction- tuning impact in-context steerability? For the PT models, we use the same prompt template for all models, with Description :/Inutt: : /Output: delineated by newlines. To ensure we are leveraging maximum performance from the IT models, we test each IT model's performance on both the PT prompt and two chat-style ICL prompts, and report results for the best performing prompt template (see App. MM. We evaluate in-context steerability on all of SPECTRUM SUITE for the PTУIT models. We include the entire suite of results in Appendix K and highlight the principal results below.
-30
10
Relative Change in Accuracy (Percentage)
Figure 2: Change in accuracy SPECTRUM SUITE from the pretrained to instruction-tuned model. Cur- rent instruction-tuning hurts in-context steerability.
Current cestruction-tunnn hurts in-context steerability. First, let's examine the change in accu- racy for the IT models. We report accuracy for all categorical data (multiple-choice + small support numeric distributions) in Figure 2 Out of 76 model family/task comparisons, instruction-tuning significantly decreases accuracy in 35 cases, doesn't 't significantly affect accuracy in 33 cases, and significantly increases accuracy in only 7 cases. Additionally, two of the seven comparisons where instruction-tunin helped were on predicting an individual's chatbot preferences wwiieeneeeshhhhhhhhhhheee which is adjacent to precisely what instruct models are optimized for (chat). The performance drop is even more stark
4

---

Published as a conference paper at ICLR 2026
on loss: for Gemma and Qwen, loss is higher on 50/50 comparisons, while Llama loss is worse in 11 cases, the same in 11 cases, and better in 3 cases. Loss results are similar on the free-text SPECTRUM SUITE datasets: out of 144 comparisons, IT loss is worse than PT loss in 117 cases, tied in 25 cases, and better only in cases.
ICL for general capability elicitation is not de- graded by instruction-tuning To disambiguate in- context steerability from general capability elicita- tion, we also run the exact same experiment with eight general capability task datasets (Fig. B} In contrast with the SPECTRUM SUITE datasets, accu- racy increases in 8 of cases, the same in 13 cases, and decreases in 2 cases.
General Capabilities
Relative Accuracy after Instruction-Tuiing
Sor Board beto lige duallyy Sore
Sweeig 131
All in all, we believe that this characterizes a differ- ence in behavior for IT models-whill they main- tain the ability to utilize in-context demonstrations for general capability elicitation, they seem to strug- gle to adapt at tasks that require heavy in-context stecrability. Limited prior work has suggested that instruction-tune models sometimes perform better without in- in-context examples (Asai et al. [024 Lam- bert et al. 2025); however, to our knowledge, ours is the first work empirically characterize this in-context learning performance degradation for in- context steerability tasks.
12
15
Relectre Change Accurary (Percenttgge)
Figure 3: Current instruction-tune tuming gener-
ally helps on capability benchmarks.
What explains this difference? While we leave an in-depth exploration of this phenomenon to future work, we hypothesize that it could be due to some combination of 1) instruction-tunne inducing very strong priors that are difficult to override even with in-context demonstrations, 2) over-optimization on tasks with a single ground truth, or 3) overfitting to particular benchmarks.
4.1 SPECTRUM TUNING AND IN-CONTEXT STEERABILITY ON HELD-OUT TASKS
We have characterized that current instruction-tunee models struggle at in-context steerability, but how does our method compare? We evaluate Spectrum- Tuened (ST) models on SPECTRUM SUITE test tasks and compare them their PT and IT counterparts (Table D Note that the test task data sources have no overlap with the train split, requiring generalization.
SPECTRUM TUNING usually matches, and sometimes improves upon, PT steerability. Out of 15 multiple-choice (MC) loss comparisons, ST ties with PT models in one case and achieves lower loss compared to PT models in 14 cases. On MC accuracy, ST matches/improves/woreee comparisons. On the free-text datasets, ST matches PT in 28 cases, is worse in case and is better in cases. In most cases, SPECTRUM TUNING matches (but does not beat) the very strong baseline of a pretrained model at in-context steerability, but does improve performance more often than it hurts performance.
on 10/3/2
Models trained with SPECTRUM TUNING most often have the best calibration. We report cali- bration in Table In 9/15 cases, the ST models have the best calibration. Additionally, the Gemma and Qwen IT models have worse calibration in 10/10 cases than their pretrained counterparts, show- ing another side effect of heavy instruction-tuning (cf. 'Tian et al. 2023 OpenAl et al. 2024).
SPANNING THE OUTPUT SPACE (OR; DIVERSITY VS. VALIDITY)
To measure how each model trades off validity and diversity, we create 22 generation tasks for which there can be many valid values and we can programmatically verify correctness (llomee))- Given a prompt, we generate 100 completions 01, ... 0100 (temperature = here and throughout) from each model, and report the following statistics: the percentage of outputs which are valid 110 100 (o))) the percentage of valid generations that are unique dedupp (o.:: (one((;))=]]] and
the number of distinct valid generations (or, yield: |dedup(fo; (a,) 1})).. We perform deduplication with exact string matching. Yield is a particularly important metric for settings such as synthetic data generation, ideation, creative writing where you want cover space as much
5

---

Published as a conference paper at ICLR 2026
gemma-3-126
Qwen3-14B
Liama-3.1-8B
Multiple- Cholce Datasets
Metric
ST (ours)
PT
IT
ST
PT
IT
PT
IT
luberman, individual infividfull.ateegrrial =2, N=1000)
2.47
10.5
9.10
2.58
2.74
Ace
23.8
24.4
22.4
23.5
20.3
22.0
20.8
20.2
19.0
was individual (Kax==1) N=1000)
1.36
4.10
4.35
1.57
Acc
42.6
42.1
40.4
44.3
41.1
40.5
41.7
41.6
39.4
numbergame individual (/mms +25, N=592)
Loss
..39
.705
1,80
.621
.697
1.28
.618
8644
.770
Acc
71.0
67.5
caatottareaa.iddiiddee 1.mma... N=725)
Loss
1.43
L.62
4,94
1.34
1.47
4.39
1.39
1.76
1.77
Ace
38.6
46.3
39.5
Hight -9, N-200)
Loss
1,09
1.32
4.06
1,08
1.29
2,92
1.12
1.45
1.41
Ace
39.8
41.2
40.6
43.7
43.7
40.5
33.4
42.0
40.2
Free-Test Datasets
Metric
ST (ours)
PT
IT
ST
PT
IT
ST
PT
IT
novacomet Jo pothosis (K Kma=lll 1=155
Loss
104
104
135
106
106
129
107
106
112
novacomel premise (Kmx-55 =55, N=51)
Loss
27.7
28.0
35.5
28.1
27.5
38.0
27.8
27.7
28.6
Inhermas question (K N=30)
41.4
31.8
24.8
hubermas opinions m2 M=186)
N-1866
Loss
930
928
1070
948
949
1070
943
944
991
hubermas individual (3Cma= N=1000)
210
176
numbergame pere (2mx=24,NN1133
Loss
4.23
4.22
6.68
4.22
4.24
5.61
4.24
4,43
4.41
glohaloqa (/mma-h N-231)
Loss
14.0
14.4
21.5
14.0
14,4
20.9
14.2
14.7
15.6
chatbotar prommms (Kmms N-588)
Loss
70.2
117
69.1
97.8
72.0
77.6
chathotarcns pusiisttt (55 N-716)
Loss
127
125
259
124
124
169
134
133
149
chemnssryy aeool (Knas No59)
12.9
11.8
8.55
chemistry oxidative (Knxx N-101)
Loss
7,57
7.58
11.6
7.64
7.84
10.2
7.64
7.72
7.84
Table I: In-context steerability on held-out SPECTRUM SUITE-Test. SPECTRUM TUNING generally matches or improves upon the pretrained model performance. Best values (and ties, failing to find significant difference at = .05) are bolded.
Espected Calibration
pemma-3-122 12b
Qwen3-14B
1.ama-3.1-88
Multiple- Dataset
(ours)
(ours)
(ours)
habermas individual cutegorical
0.116
0.069
0.239
0.032
0.05
0.198
0.037
0.084
0.055
Individual nambergame idull chatbotarena prefs Hight
0.806
0.817
0.02
0.005
0.015
0.029
0.163
0.027
0.826
0.108
0.028
0.024
0.817
0.041
0
0.040
0.271
0.838
0.035
0.228
0.046
0.070
0.038
Table 2: Calibration on SPECTRUM SUiTE-Test, binning label token probabilities every decile for expected calibration error (ECE Σ uy acc ace(a) co)((((()) where B 10 bins, ni, is the number of samples in bin b. acc(b) is the accuracy in bin 6, and conf(b) is the average confidence in bin b). SPECTRUM TUNING (ST) usually results in the best calibration (9/15 cases).
as possible within some requirements. Additionally, we evaluate each model under three settings: zero-shot with task description, three-shot with no task description, and three-shot with task description (also see App. N.. Results can be found in Fig Tasks are the same across models.
Instraction-tuned models have high validity but low diversity. IT models produce valid outputs 70% of the time, even in the zero-shot setting. However, this comes at the price of diversity, resulting in fewer than 30 unique valid generations in few-shot settings. Yield is even lower in the zero-shot cetting-Qwer and Gemma average yields of only 5-6, while Llama averages only 24.
Pretrained models are more diverse, but rely on few-shot examples for validity. Pretrained models do not suffer from the same mode collapse, and consistently have higher diversity (> 40% of valid generations are unique). However, this comes a trade-off with validity, where their gener- ations are universally less valid than the IT models'. The pretrained models also rely heavily on the few-shot examples to elicit valid generations, achieving validity of < 20% in the zero-shot case. However, in the few-shot cases, they have significantly higher yield than the instruction-tuned models due to their higher coverage of the space.
SPECTRUM TUNING offers Pareto improvement diversity and validity, matching or ex- ceeding pretraining yield. In eight of nine model/setting comparisons, SPECTRUM TUNING offers either a Pareto or strict improvement with a Pareto improvement, this also leads to a higher yield-ic. for a fixed generation budget, SPECTRUM TUNING generates the most unique valid generations.
the PT/IT models validity/divesiity. In all eight settings

---

Published as a conference paper at ICLR 2026
Verifiable Generation Tasks
Open-Ended Chat Prompts
E Dhocrption
Swhhel
Descrition: Soled
peggeratioon per N0-00 provett)
pererations por 100 premst
140
26
40
330
153
81
20
THME THE 200
Canera
Line
Damea
1181
Oaiaa Qven
Figure 4: Diversity vs. Validity. Left: Results on 22 verifiable tasks across 100 generations. Right: Human-annotated validity results on two sets of 100 open-ended prompt sets (Gemma). SPECTRUM TUNING generally offers Pareto improvement on diversity-validit over PT/IT models. In partic- ular, SPECTRUM TUNING increases the yield (# of unique usable generations) in the zero-shot case and on Novelty Cunnhcd. Error bars are 95% confidence intervals over the SEM, and asterisks (+) show the best in family performance (within 95% confidence).
SPECTRUM TUNING achieves much higher yield in the zero-shot setting. Focusing in on the zero-shot setting, SPECTRUM TUNING particularly shines. The IT models are able to follow the description and produce a valid output, but have very low diversity (~300 for Llama, 5% for Qwen and Gemma). Meanwhile, the pretrained models are unable to consistently generate valid outputs (< 20% validity). ST models, however, are able to follow the instructions and produce valid outputs > 60% of the time while maintaining 50% diversity. This leads to much higher yields compared to PT and IT models (Gemma: 40.5 vs. 6.2; Qwen: 32.2 vs. 10.1, Llama: 37.4 vs. 24.0).
SPECTRUM TUNING'S gains hold across temperature values. One way to trade-off validity for diversity for a given model is sweeping temperature. To ensure that our results hold across temper- atures, we the same experiment with T = [10, 5, 2, 1.5, 1, .,. We found that SPECTRUM TUNING A) still expanded the Pareto frontier and B) gave the highest possible yield when choosing an optimal temperature (see App. E for more details).
5.1 HUMAN EVAL
We extend the verifiable task experiments with a human evaluation on open-ended chat prompts: NoveltyBench-Curatee Curated (100 prompts, /Zhan et al. 2025d) and Infinit-CChat-Evva (100 prompts, yet to be published, obtained from the authors). However, SPECTRUM TUNING does not opti- mize for chat capabilities, but rather for fitting to description/input output. In order to elicit chat capabilities in-context, we try two approaches: zero-shot chat, where we prompt You are helpful AI assistant, input: <prompt>;; and few-shot chat, where we utilize the same description and four examples of prompt inputs and chat responses as outputs. Additionally, we use similar prompt for the pretrained model as a base- line, with the description, a prefix for the prompt of User: : and an output prefix of Asssistant :, zero-shot and with the same four few-shot examples (similar URIAL, Lin al. 2023). More details in App. N
with descriptions
For each prompt, we generate four completions from the model. We recruit annotators to judge whether given generation is a valid response to the prompt. Each generation is annotated by four annotators, and we count the generation as valid if three of four annotators marked it valid. Overall, annotators had 73% pairwise agreement rate. Due to the cost of the evaluation, we only annotate generations for one model family, gemma-3- 3-1 2b. For additional evaluation de- tails, see App. [ For calculating diversity, we follow NoveltyBench's approach and utilize their deberta --3- 1e---arrr-bssee model for assigning two generations as duplicates. We report the Pairwise Uniqueness %, or the probability that any two valid generations are not considered dupli- cates, along with yield. Results are in Tab.
Few-shot pretrained models improve yield over instruct models. While lagging in validity, pre- trained models produce much more diverse responses than their instruct counterparts, and are able to achieve -40% validity from shot chat examples, improving yield and offering a strong baseline.
7

---

Published as a conference paper at ICLR 2026
SPECTRUM TUNING offers Pareto improvement on diversity/validity and improves yield over baselines on Novelty validity than the pretrained model, while offering substantially higher diversity than the instruct model. This improvement results in a statistically significant increase in yield over the baselines. On Infinite-Chail Chats, the pretrained models and our models do not perform significantly differently, covering roughly the same space on the Pareto frontier and on yield. While disambiguating the reason for the differing performance may require further investigation, we do note that many of the AnfiniteChhat eval prompts have specific requirements, such as "In five words". "In couple of paragraphs," " etc., which our models often fail to adhere to. In contrast, the NoveltyBench- Curated prompts are far more open-ended. It may be that our model performs best at generating shorter outputs, and future work may needed to enhance precise instruction-folloviii while maintaining diversity. However, on both datasets, the instruct model has significantly lower yield and diversity.
On NoveltyBench-Cune Bench-Curttd,ddd our method offers higher
DISTRIBUTIONAL ALIGNMENT AND PLURALISM
Next, we evaluate our system's ss ability to steer to match target distribution. We utilize seven held- out datasets P mainly focusing on human response distributions and synthetic random draws task. We prompt models zero-shot with a description of the setting and a surget We then calculate the probability of each possible valid output, normalize, and calculate from the target distribution. We also measure coverage, or the total probability mass on the set of valid answers. Results are in Table 3 and takeaways are as follows. (More details in App. O
divergence
Distribational Afignment: JS- DS-iieeggence
3-12b
Quen3-14R
Llama-3.1-BB
Dataset
(ours)
ST (ours)
IT
ST
PT
IT
Machine Persomality Inventory (N-120, (+6-
0.347
0.405
0.063
0.087
0.131
Rotten Tomatoes (N-1000. (Y2)
0J132
0.132
0.134
0.028
0.028
0.122
0,035
0J135
0.086
NYTimes Books (N-940. [=])]
0,051
0.063
0.328
0.070
0.088
0.344
0.046
0.061
0.247
GlohalQQA (N-1000. (6)
0.177
0,270
0.090
0.088
0.274
0.091
0.108
0.165
Ura (N-1000 |Y g6i
0.021
0.071
0.185
0.151
0.059
0.198
0.032
0.124
0.086
Habermas (N=658, (=7)
0.155
Number Game (N-1000, (2)
0.151
0J149
0.138
0.052
0.043
0,131
0.055
0.060
0.094
Table 3: Distributional alignment results. Instruction-tuninn drastically hurts distributional align- ment. SPECTRUM TUNING generalizes to unseen tasks and improves or matches distributional alignment compared the pretrained model. Best result (within 95% statistical significance) in bold. N is the number of distinct instances, |Y| is the number of possible outputs.
Instruction-tuned models have higher distributional divergence than pretrained models. In line with prior work (Sorensen et al., (024b)) we find that instruction-tuned models show higher distributional divergence than pretrained models on all tasks. We believe that this is in large part due to their low-entropy, spiky distributions. In other words, for distribution matching, current instruction-tunnng categorically hurts performance compared to the pretrained model.
SPECTRUM TUNING generally improves distributional alignment over pretrained models. Out of 21 model/dataset comparisons, SPECTRUM TUNING improves distributional alignment in 10 cases, matches PT models in 10 cases, and degrades performance in case. Pretrained models a strong baseline-the pretraining objective entirely consists of trying to estimate well-calibrated distribution over the next token. To our knowledge, ours is the first method to improve distributional alignment on unseen datasets over pretrained models.
SPECTRUM TUNING improves coverage of valid an- swers over pretrained models and roughly matches instruction-tuned models. For each of the datasets, there is a limited set of valid answers. Pretrained models often struggle to shift their probability mass based on instruc- tions in zero-shot manner only cover the valid output distribution, achieving ~ 50% coverage in our evaluation.
Prokalel Quesiita
servec
Lame
Figure 5: Valid answer coverage (t).
Machine Personality Inventory (Jiang et al. 2023 3. Rotten Tomatoes Plaiinesss-Pattoom301111 2024). NY-
Times Books (Meister et al. (2024). GlohalOQA (Durmus al. (023), Um (ours, new contribution) Hahermas
Tessler al. 2024), Number Game (Bigelow & Piantadosi 2016
Tenenhaum
1999).
8

---

Published as a conference paper at ICLR 2026
Ablation Components
ICL Steerability
Align.
Output Caverage
AML Weight
Special Tokens
Train
Train Stp
only Outgust
Ace (Norm.)
Free-lest (Norm.)
Yield Deseription
Yield
Yield 3-bhot
Fimbedding SyorTeeig te Stare
(Norm.)
18-1Nv.
S-bot
Description
Defamtt: Spectrio Taning Pretrattdd Inrraction Taned
PT
IT'
58.8%
1.00
1,00
1,00
8669
%6.7
42.1
49.2
PT
x (PI
1.19
I.99
1,09
5.8
31.2
44.2
pesenge)
20.7
I- Truining wethod oblation:: () Defonle 4) Loss first oupur ('nstracr-Se 7 on S-Swiee);
Lorr only
(Mete- ICL, Si-utte;; 6) Losr all tofons (S-Suite)
38,8%
1,00
36.3
PT
II
38.8k
fixst only
1.03
1.00
1.01
.867
37,9
33.0
44.0
FT
II
38,5%
last only
1J12
11.99
1,00
.303
17.1
35.4
39.5
C- Date ablaiion:
7) Truis only on capubiiiy Flanlledge aficiaation data, 8) Trais Spectium Suite, dlttie sior mosiceed to expubbilit dail (capahilry data)
PT
IT
3.9L
1.03
1.00
1.01
.866
21,8
35.5
40.8
Wright Ablation: Spectrane Toming with Deyinal wright inti: My init benacke spocial token voubodd PT mill, mondorr spocial iokan combed,
PT
IT
58,8%
1,100
1.00
1,00
36.7
42.1
49.2
PT
66/35 (PT)
58.8k
1.43
1,83
1.02
.06.3
38.0
30.0
$3.0
11
II
II
38.5%
1.05
1,02
1,08
33.4
42.0
45.2
Table 4: Ablations, averaged across models and tasks. Shaded rows are default Spectrum- results. We show averaged results for A) the default setup. B) training on SPECTRUM SUITE with different methods, C) training capability-focusse different model weight initializations. Best result within each ablation is bolded, and second best is underlined. ICL Steerability results are normalized to the default configuration.
data in place of SPECTRUM SUITE, and D)
In contrast, SPECTRUM TUNING achieves > 90% coverage, nearly matching the instruction-tune model coverage (Fig
7 ABLATIONS AND GENERAL CAPABILITIES
In Table we ablate parts of SPECTRUM TUNING in order to further disentangle the effect of each component. We report averaged results for all three desiderata across all models and tasks. In A), see the normalized data from the prior sections, illustrating Spectrum-Tuned models improvements over base and default instruct models.
SPECTRUM SUITE'S selective loss is important for performance on all desiderata. In B), we hold the Spectrum Tuning data constant, and ablate the training method. We compare against train- ing on the first output only (similar to Instruct-SFT) training on the last output only (similar to MetaICL, Min et al. 2022a), and calculating loss on all tokens, including descri petiiiiioono inputss We find that training on the first output only causes a degradation in few-shot learning capabilities (ICL loss, few-shot yield), and training on the last output only causes across the board degradation, especially on zero-shot tasks (distributional alignment, description yield). Training on all tokens (including description/ /nput) leads to slight degradations across the board.
Training on capability-oocused data only underperforms training on SPECTRUM SUITE. We train on a subset of data in the same format as SPECTRUM SUITE, but focused on capability data instead of data requiring steerability (Table C). We find that including the SPECTRUM SUITE data important for eliciting the desiderata. Finally, we find that D) the default weight initialization (PT model weights, IT special token embeddings) overall elicits the best performance, although initializing the special tokens with bracket token embeddings seems to improve the multiple-choice accuracy and distributional alignment.
While the default recipe offers strong performance, future work could i) further optimize hyper- parameters (as have done limited optimization) ii) reduce reliance on initializing the special tokens from IT models, and iii) probe which data is most important in eliciting gains.
SPECTRUM TUNING does not harm general model capabilities. Lastly, we evaluate whether our method affects general model capabilities. While we do not necessarily expect our method to
However, we also consider this distinct from traditional instruction-tunin as the focus is fitting the data generation task of the description opposed to generating helpful chat assistant response.
'In fact, after running the main suite of experiments, we suspected that our models were somewhat underfit. We found that simply reducing the batch size resulted in signiticant gains in distributional alignment and yield (see App. H for more details). We believe that this illustrates exciting opportunities for further optimization and improvements improve performance-the performance ceiling has not been hit.

---

Published as a conference paper at ICLR 2026
improve upon standard evaluations where there is a single correct answer, want to understand if it degrades performance compared to pretrained models. While we find that Spectrum- models generally perform worse than instruction-ture cettttiooo-tneedddeee variants (as expected), we find that Spectrum-Tunme models perform similarly to the pretrained models on which they are based. (c.f. App. (D5)
8 RELATED WORK
Diversity, distributional alignment, and steerability. Several other works have documented di-
versity collapse in LLMs (Shumailov et al.] 2023 Dohmatob al. 2024 Yang et al. 2024 Zhang
et al. 2024a; Li et al., 2024; West & Potts (025), often linking it to alignment Murthy et al.
2024 Kirk et al. 2024a 2023) or insufficient training data diversity (Chen et al. 2024). Potential
consequences of diversity collapse include reduced creativity, loss of minority perspectives, spread
of bias, and overall decline in model utility and trustworthiness (Anderson et al. 2024; Kapania
ct al. (024)) Distributional alignment has been explored by few prior works (Meister et al. [022]]
Durmus et al. 2023 Sorensen et al. 2024b), but literature here is far less developed. Additionally, other works have focused on measuring steerability to system messages (Lee et al. 2024). persona
descriptions (Miehling et al. 2025 Castricato et al.) 2024), and values or attributes (Sorensen et al.)
2024b
2023). Our work builds these directions by generalizing steerability to include any in-
context information, including examples, and evaluating on a broader swath of distributions.
Pluralistic alignment and integrating disagreement into LLMs. Many have recently challenged the idea of a single ground truth (Aroyo et al. [0233 Basile et al. 2021 Gordon et al. (022). Plu-
ralistic alignment (Sorensen et al., 2024b) Kirk al.) 2024b) concerned with integrating diverse values and perspectives directly into the alignment process. Steerability in particular is related to
user fairness and ensuring that Al systems are usable by diverse stakeholders (Alamdari et al., 2024).
Related Methods Zhang et al. (2024a) found that training on samples from diffuse distributions helps LLMs avoid mode collapse. and served as inspiration for some experiments. SPECTRUM TUNING is similar in spirit, but also includes in-context samples and leverages orders of magnitude
more data. Entropy maximization in finetuning can help increase diversity (Li et al. (025). MetalCI,
(Min et al., 2022a) uses in-context examples as in our method, but focuses on NLP datasets with
a single ground truth and only trains on the last example. Centaur (Binz et al.
2024) similarly
modifies cross-entropy loss only focus tokens of interest, but focuses on different data distri- bution (cognitive-scicn science human experiments). Some very recent works have somewhat improved the diversity/validity Pareto frontier by adding some sort of diversity regularization to preference optimization or RL reward (Lanchantin et al. 2025 Chung et al., 2025 Li et al. 2025). Finally, several recent papers have found that prompting instruct models for multiple samples in-context can help to mitigate mode collapse (Zhang et al. (025)bb
DISCUSSION AND CONCLUSION
We have outlined three desiderata for conditional distributional modeling with LLMs: in-context steerability, output space coverage, and distributional alignment, and shown across three model families that current post-training can systematically hurt these properties. These results have implications for user steerability-..g. when possible, pretrained models may be preferred over instruction-tunee models when steering to a particular user well-calibrated way is important [ In addition, we have introduced SPECTRUM SUITE and SPECTRUM TUNING, a resource and post- training method for enhancing these desiderata. Models trained with SPECTRUM TUNING usually match or exceed their pretrained counterparts at these properties to our knowledge, ours is the first method to improve upon pretrained models at distributional alignment or in-context stecrabil- ity. However, much work remains. Promising directions for future work include 1) exploring which data is most important for eliciting the desiderata; 2) further characterizing why and how instruction- tuning hurts in-context steerability; 3) more work to combine the strengths of instruction mod- els and SPECTRUM TUNING models (e.g., Zhu et al. 2025 and 4) scaling SPECTRUM TUNING to larger models and more data.
'However, access the pretrained model is restricted in many proprietary cases. This illustrates gap: Can companies offer very steerable and distributionalll-alinnee models, while maintaining safety constraints?
'On the other hand, it is possible that top-1 chat performance and our desiderata are so fundamentally in tension, that we may need to specialize models to either top-1 chat performance our desiderata, and select the appropriate model for each use case combine strengths at inference (e.g.. 7hu 2025)
(e.g..ZZuu
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
In this paper, we seek to enable Al systems that can work for a variety of perspectives and estimate human preferences and opinions in well-calibrated manner. We believe that these are net positive developments, allowing Al systems to work properly for more people. Additionally, well-calibrated human preferences may be especially important as Al systems are used agentically it will be important that an agent have a good model of what the user wants, opposed to a modal preference. Calibration, where current instruction-tune -tneed systems really struggle, can be especially important for agents to safely act autonomously when they are (properly) very confident about a users' preference, and ask for direction when they are less confident.
With SPECTRUM SUITE, we perform experiments on several datasets which may include personal information such as demographics. However, all included datasets are anonymized, we attempt to use the data only in line with their intended use, and we do not distribute the underlying datasets in SPECTRUM SUITE directly. Instead, we refer people interested in extending our work to the original data sources, and provide only the code to unify the data into the descriptions input / outppt format. Because of this, we believe that our compilation of SPECTRUM SUITE does not pose an additional privacy risk.
REPRODUCIBILITY STATEMENT
We have attempted to ensure that every portion of the paper is reproducible, and release code con- taining: SPECTRUM SUITE construction, including processing and pointers to hydrate each dataset; SPECTRUM TUNING training code; and code for running all evaluations. We also release the weights for all trained SPECTRUM TUNING models. [ We include additional training details on hardware and
hyperparameters used in App. Gand additional experimental details in App. M N O In App. PP] we show demonstrative example prompts for each test task in SPECTRUM SUITE and include example prompts for remaining tasks in supplementary materials.
ACKNOWLEDGMENTS
We would like to thank Hannaneh Hajishirzi, Sewon Min, Luke Zettlemoyer, Peter West, and Kshi- tish Ghate for draft feedback and Christopher Sorensen for help designing the SPECTRUM TUNING logo. This research was supported part by DARPA under the ITM program (FA8650-23-C-7316) and by the AI 2050 Schmidt Sciences Senior Fellowship.
822484300fb72

---

Published as a conference paper at ICLR 2026
REFERENCES
Parand A. Alamdari, Toryn Q. Klassen, Rodrigo Toro Iearte, and Sheila A. Mcllraith. Being considerate as pathway towards pluralistic alignment for agentic ai, 2024. URL https:
warxiv.org/aail. wttg/aa////a......///////// 10613
Barrett R Anderson, Jash Hemant Shah, and Max Kreminski. Homogenization effects of large language models on human creative ideation. In Proceedings of the 16th conference on creativity & cogmition, pp. 413-425, 2024.
Lora Aroyo, Alex S. Taylor, Mark Diaz, Christopher M. Homan, Alicia Parrish, Greg Serapio- Garcia, Vinodkumar Prabhakaran, and Ding Wang. Dices dataset: Diversity in conversational ai evaluation for safety, 2023. /tttpsss:://axxii org./ivv...rr..ooooooo abs/2366//2000 11247
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Scbastian Ruder, and Hannanch Hajishirzi. BUFFET: Benchmarking large language models for few-shot cross-lingual transfer. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1771-1800, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18653/v1/ 2024.naacl-long.
URL https: / /acc//cccc/// anthology........ org/ 2t0.//2///00000000/////2///// 2024 www.nnaallllnoeee long 1007
Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. We need to consider disagreement in evaluation. In Kenneth Church, Mark Liberman, and Valia Kordoni (eds.), Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pp. 21, Online, August 2021. Association for Computational Lin- guistics. doi: 10.18653/w1/2221.bbr 1ttp://///222.ppp-111. 1.3. URL
b.3f-1.37
Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, and Siva Reddy. Value drifts: Tracing value alignment during Ilm post-training, 2025.
URL petttps://ariiy
abs 25100 26707
Eric Bigelow and Steven T. Piantadosi. A large dataset of generalization patterns in the number game. Journal of Open Psychology Data 4(1):e4, 2016. doi: 10.5334/jopd. 19. URL BRLhhtps:: (opeupsychologydar.
metaini com/ wataa//rticles//e cttpm//ataaaeeee...33/////...... 2016-03-18; accessed 2025-09-21.
Published
Marcel Binz, Elif Akata, Matthias Bethge, Franziska Brändle, Fred Callaway, Julian Coda-Forno, Peter Dayan, Can Demircan, Maria Bekstein, Noémi Éltetō, Thomas L. Griffiths, Susanne Haridi, Akshay K. Jagadish, Li Ji-An, Alexander Kipnis, Sreejan Kumar, Tobias Ludwig, Marvin Mathony, Marcelo Mattar, Alireza Modirshanechi, Surabhi S. Nath, Joshua C. Peterson, Milena Rmus, Evan M. Russek, Tankred Saanum, Natalia Scharfenberg, Johannes A. Schubert, Luca M. Schulze Buschoff, Nishad Singhi, Xin Sui, Mirko Thalmann, Fabian Theis, Vuong Truong, Vishaal Udandarao, Konstantinos Voudouris, Robert Wilson, Kristin Witte, Shuchen Wu, Dirk Wulff, Huadong Xiong, and Eric Schulz. Centaur: foundation model of human cognition, 2024. URL https: //tp/vaaxvvv//////a//eaa//aaaav org/ abs/ /asss/241...m 20268
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: /tttps://arxiv..oo org/ abs 2005.000. 14165
Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, and Davide Bernardi. Mul- tiPICo: Multilingual perspectivist irony corpus. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 16008-16021, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. N0.11653/v11200 1/2024.2222244 149 URL https: /tmps://acl antho ogg.02// acl a long. 849/1
12

---

Published as a conference paper at ICLR 2026
Louis Castricato, Nathan Lile, Rafacl Rafailov, Jan-Philipp Fränken, and Chelsea Finn. Persona: A reproducible testbed for pluralistic alignment, 2024. URL https: https::/arxiv... org/ /bs/
2407.
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I. Abdin. On the diversity of synthetic data and its impact on training large language models, 2024.
URL wwwphhtpss//aatte
org/ abs/ 2410.41110..0 15226
John Joon Young Chung, Vishakh Padmakumar, Melissa Roemmele, Yuqian Sun, and Max Kreminski. Modifying large language model post-training for diverse creative writing, 2025.
URL http::tts//aatt http:s//aaxxv.orr// /ttgg//bb////.11112222 17126
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge,
2018. URL lttpss//arxii.. org. abs / 1803803000 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10.1v:2110.41168 2021.
Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint arXiv:2410.04840, 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A survey on in- context learning. 2024. URL htt.:hhtt/://ttct awwiv. org/ abs 2301. 30234
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019.
URL httttttss//axii...
org/ abs 1903.040000 00161
Yann Dubois, Percy Liang, and Tatsunori Hashimoto. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference Language Modeling, 2024. URL
net forum? bobrummid-2a0 mddm211BByCBBXXXX
Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCan- dlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models, 2023.
EVS/WVS. European values study and world values survey: Joint evs/wvs 2017-2022 dataset, 2024. URL
http://wwwwwwwwww//eeeeee.eeee.e wttp:///enneeerrpaaaaaaaaaau datta-and-dccuenttaiioo polocumettttiinnoooottttoo 201p:///ww22220222-2a--a-a Identical ver- sion also via WVS site with DOI 10. 14281/18241.22 accessed 2025-09-21.
Sara Fish, Paul Gölz, David C. Parkes, Ariel D. Procaccia, Gili Rusak, Itai Shapira, and Manuel Wüthrich. Generative social choice, 2025. URL BtRL:attts://axxx http://awxxxvvoo/a///////// hbsrg//abb/339999 01291
Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference Human Factors in Computing Systems, CHI '22, pp. 1-19. ACM, April 2022. doi: 10. 00.11445//411102.550044 URL htllhttp://dx doi, wwt.ddd......og....oo. 1145/3491 3502004]
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXi::2407.2178. 2024.
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5005-5033. Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8- 89176- 195-7. doi: 10. 18.118/3/v 1/2025.22222222 120.....inn................. org/2025. f i nn ngs 282/
URL https: petttss//aclaathooe //aclanthooooy..........
13

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
03300
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-leaming in neural networks: A survey, 2020. URL htttps://axxi..orr org/ baarrg/bb/220000 2005///300000439999
Hyewon Jang and Diego Frassinelli. Generalizable sarcasm detection is just around the corner, of course! In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4238 4249, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: URL httph:ttp///t
11055/3///2222aaaaaaaa wtt. mwwp://222..m.a.....mmmm naaci -ooo:-0-..a.aloonn--ooo. 2387
Ziwei Ji, Justin D. Li, and Matus Telgarsky. Early-stopped neural networks are consistent, 2021.
URL httts::///aaxiii http:////aaaaai.cccb//b////.. 05932
Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evalu- ating and inducing personality in pre-trained language models, 2023. URL httpt:t/ppssxiii org/ abs/ 2206.0000000 07550
Shivani Kapania, William Agnew, Motahhare Eslami, Hoda Heidari, and Sarah Fox. 'simulaerum of stories': Examining large language models as qualitative research participants. arXiv preprint arXiv:2409. 19430, 2024.
Hannah Rose Kirk, Bertie Vidgen, Paul Rötteer, and Scott Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, 6(4):383-39) -9238, 2024a.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment dataset: What participatory, representative and individualised human feed- back reveals about the subjective and multicultural alignment of large language models, 2024b. URL /atttps:/arxiv..oo
aatpaabs/4450 2404. 16019
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects rlhf on Ilm generalisation and diversity. arXiv preprint 17Xiv:2310006455: 2023.
G. Kokolakis. Bayesian statistical analysis. In Penelope Peterson, Eva Baker, and Barry Mc- Gaw (eds.), International Encyclopedia of Education (Third Edition), pp. 37-45. Elsevier, Ox- ford, third edition edition, 2010. ISBN 978-0-08-044899
doi: https://doi.rr//10..u URL https:: /ttps://wwww. www. www.sscience eecieneedi rect. wwm/ccimccccc..cccccccoo.. ence/
0978-0008-0044894-7000 axt.cle/s111/77700004444338
Nikolay Kolyada, Khalid Al-Khatib, Michael Völske, Shahbaz Syed, and Benno Stein. Webis
changemyview corpus 2020 (webis-cmv-20) 2020. URL attps::///// :/ttpss://o../ 0r0/10.00000 5281/
denodo.3778200
Version vI; file used: threads. j.onn. Accessed 2025-09-21.
Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Du- rumeric, Kurt Thomas, and Michael Bailey. Designing toxic content classification for a diversity of perspectives, 2021. URL URL httpt::tt///attteeee arxiv. wwwpg//a////ae.e 2216 04511
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu. Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christo- pher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishirzi. Tulu 3: Pushing frontiers in open language model post-training In Second Conference on Language Modeling. 2025.
oww.review.weeoe..o .et/ forum ? wedd-llubfHHitt
Jack Lanchantin, Angelica Chen. Shehzaad Dhuliawala. Ping Yu, Jason Weston, Sainbayar Sukhbaatar, and Ilia Kulikov. Diverse preference optimization, 2025. URL https: :ttttps://rxiv. oog/ arg/abs/2555 /2501. 18101
14

---

Published as a conference paper at ICLR 2026
Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Sco. Aligning to thousands of pref- abppg///b//4405
erences via system message generalization, 2024. URL
17977
Elisa Leonardelli, Silvia Casola, Siyao Peng, Giulia Rizzi, Valerio Basile, Elisabetta Fersini, Diego Frassinelli, Hyewon Jang, Maja Pavlovic, Barbara Plank, and Massimo Poesio. Lewidi-2025 at nlperspectives: third edition of the learning with disagreements shared task. In Proceedings of the 4th Workshop on Perspectivist Approaches to NLP (NLPerspectives) Association for Compu- tational Linguistics, nov 2025.
Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling, arXiv preprint (rXiv:2407.024460 2024.
Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lan- chantin, and Tianlu Wang. Jointly reinforcing diversity and quality in language model generations, 2025. htt.lttss//aaiv..ooo
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell base Ilms: Rethinking alignment via in-context learning, 2023. URL
http:///axxvvvooo.. abs 2ttp//211...15...
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214-3252, Dublin, Ireland, May 2022a. Association for Computational Linguis-
tics. doi: 10. 18633////20222.llloogg. acl- acl-long: 2297
URL
Stephanie Lin, Jacob Hilton, and Owain Evans, Truthfulqa: Measuring how models mimic human 07 07958
falsehoods, 2022b. URL
Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. We're afraid language models aren't modeling ambiguity. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 790-807, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10. 11855///12022.emmlpp-a emn munip-main. main. 2n.inn11] 51
URL https: pttps:://aclathhte hataa/aaaaatoooooo.ooooooooo org/2023.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Pruceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150.
Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/ //www.aclweb.. org/
1015
Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. Benchmarking distributional alignment of large language models, 2024. URL htttllttss://ee http::://axxxv..orr abs/ h 05403
Erik Miehling, Michael Desmond, Karthikeyan Natesan Ramamurthy, Elizabeth M. Daly, Pierre Dognin, Jesus Rios, Djallel Bouneffouf, and Miao Liu. Evaluating the prompt steerability of large language models, 2025. http:/ttp:///aaae
www.org///a... hbwg///b///a /aalbb//241.... 1..1..2405
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context, 2022a. URL http:: http://awwwwwv.oor///////1/..../ 15943]
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoer. Rethinking the role of demonstrations: What makes in-context learning work?, 2022b. https::::://axxii org abs./222..22222 12837
Jared Moore, Tanvi Deshpande, and Diyi Yang. Are large language models consistent over value- laden questions?, 2024. URL http::tp:/// //t::///axiivii/ arxiv. org/ http/bb////4/// 240. /bbb/2247000996 02996
mstz. Titanic (survival) - hugging face dataset, 2023. URL hittps://uuginngaac.. co/
Datasets/mstz/titaat Subset: survival; 891 rows; accessed 2025-099-11.
15

---

Published as a conference paper at ICLR 2026
Sonia K Murthy, Tomer Ullman, and Jennifer Hu. One fish, two fish, but not the whole Align- ment reduces language models' conceptual diversity. arXiv preprint arXiv:241 2024.
Jeremy Neiman.
Generating
haiku
with
deep
learning.
Towards
Data Sci-
ence, December
2018.
URL batps::/
com/
-ttradtt--oonnkkk------------ee- 09-21.
Accessed 2025-
Netflix, Inc. Nerflix prize data, 2009. URL httpp:://ww.kot com http://leooooootttooeeeeee datastts/s
iefEllxxiinnneefllixe -ata Dataset from the Netflix Prize competition; ac- cessed 2025-09-21.
OpenAL. Collective alignment 1: Public input model defaults (version 1.0). https:// huggingface. co/ http://gaacc..//ttttttttt//eeeee accessed 2025-09-21.
2025. Dataset;
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diego Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael. Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy. Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens. Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng.
16

---

Published as a conference paper at ICLR 2026
Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng. Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL
https://arxxt wwt. 2303 08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Jiaxin Pei and David Jurgens. When do annotator demographics matter? measuring the influence of annotator demographics with the popquorn dataset, 2023. URL
2306. 06826
Linlu Qiu, Fei Sha, Kelsey Allen, Yoon Kim, Tal Linzen, and Sjoerd van Steenkiste. Bayesian teach- ing enables probabilistic reasoning in large language models, 2025. URL https://arxiv:
org/ hate/ab///a 2503. 17523
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024.
18290
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning, 2021. BtRppttss://axxitt
org/ ablaaaa/...2///// 07857
Mayk Caldas Ramos, Shane S. Michtavy, Mare D. Porosoff, and Andrew D. White. Bayesian optimization of catalysts with in-context learning, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL httpthtttt//ttttttttett org abs 2tt.///11.........1..1.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level grogle-proof q&a benchmark. In First Conference on Language Modeling, 2024. ( net b fort/Eorum? id- T167584b98
Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brah- man, and Dirk Hovy. Issuebench: Millions of realistic prompts for measuring issue bias in Ilm writing assistance, 2025. URL.
22502 08395
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adver- sarial winograd schema challenge at scale, Commun, ACM, 64(9)99-106, August 2021. ISSN 0001-0782. doi: 10.1145/3474381 URL https::/ org. 10. 11011114444444 3474381
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chh- ablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Mat- teo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli. Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization, 2022. httpss l/arxiv.org/aate 2tt.b////211..000 08207
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint 17X.4:2303.17548, 2023.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. 17493, 2023.
17

---

Published as a conference paper at ICLR 2026
Social Security Administration. Baby names from social security card applications national data. hhtpps: / www. ssa.gov /oac////////////o////o//////// http:/aa...///.ttt//aaaaaaaaaaaa wwtimnesssmmmmsmtmeee html, 2025. Data are from 100% sample of Social Security card applications; names with i5 occurrences suppressed. Accessed 2025-09-21.
Taylor Sorensen, Liwei Jiang, Jena D. Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 38(18):19937-19944, March 2024a. ISSN 11159-5399, doi: 10.1 1a.16669aaiii33350 v38....88219990 2.9970. URL http: dww. hag./orgg11000 http://wwx.cc...r//...o100
Choi.
do1
aaai: v38 1w.31811..0 29970
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. A roadmap to pluralistic alignment, 2024b. URL peatps://arxiie org/ abb//402.
05070
Taylor Sorensen, Pushkar Mishra, Roma Patel, Michael Henry Tessler, Michiel Bakker, Georgina Evans, lason Gabriel, Noah Goodman, and Verena Rieser. Value profiles for encoding human variation, 2025.
15484]
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-1 13051, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 10..8865550 11251111/332233 findings-acl.822 URL BRLhttps:: http::s:///alaaahhooo.... org.22223300//223333 Findings acl. 824/
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej. Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Cas- bon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer. Xi- aohai Zhai, Anton Tsitsulin, Robert Busa-Fekke Fekete, Alex Feng, Noveen Sachdeva, Benjamin Cole- man, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline L. Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng. Dimitris Paparas, Divyashree Shivaku- mar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eu- gene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi. Hanna Klimezak-Plucinsee Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wi- eting. Jonathan Lai, Jordi Orbay. Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Ni- lay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron. Paul Kishan Ruben- stein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanezyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger. Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesmikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow. Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas
18

---

Published as a conference paper at ICLR 2026
Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Bar- ral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer. Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukeuoglu, Clement Farabet. Elena Buchatskaya, Jean- Sran-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma technical report, 2025, URLhttps: /ttpvv/aaxvv..//////i/ org/ abs 2000/ 19786
Joshua Tenenbaum. A Bayesian Frumework for Concept Learning. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, 1999.
Michael Henry Tessler, Michiel A. Bakker, Daniel Jarrett, Hannah Sheahan, Martin J. Chadwick, Raphael Koster, Georgina Evans, Lucy Camphell-CGllingham,, Tantum Collins, David C. Parkes, Matthew Botvinick, and Christopher Summerfield. Ai can help humans find common ground in democratic deliberation. Science, 38666719))cad(2852 2024. doi: (0.1126/sieence.adq25555 URL https: //tkps::/www.r www. wwt.:iiinnn..././//////////////e /tt.://....../////////0/1//11110 2ttp::11112222cccccc....22222 adg22822
The Computational Democracy Project. Open polis data. https: / com/ 2025. GitHub repository; data exports from select public Polis conversations; accessed 2025-09-21.
compdemocracc//ppeenee
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback, 2023. URL https:
7/arxiv. wrg. abs/ 2305 14975
Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, Zhenhan Dai, Yifeng Xie, Yihan Cao, Lichao Sun, Pan Zhou, Lifang He, Hechang Chen, Yu Zhang, Qingsong Wen, Tianming Liu, Neil Zhenqiang Gong, Jiliang Tang, Caiming Xiong, Heng Ji, Philip S. Yu, and Jianfeng Gao. A survey on post-training of large language models, 2025. URL https:
06072
trexmatt. 200,000+ jeopardy! questions (esv dump from j-archive). https://drive.
google wwtp:////i..ee////tw/..v.o/
0 uF ---FFnnBguuufSSccnntttcc
2014. Original announcement on r/datasets; accessed
2025-09-21.
w/B WBusiness-Platformm00
Rotten tomatoes movies 1970-2024.
https: drive.
2024. Archive: roc.ee.tomettee...taa11000000000 zip. Provenance: r/datasets thread
httpatttttttttttettttttteeeteete /ecc
for
waw.:eee_ttttttesseeee Sarmattos__ovie___t1__ 1970 1900__2_211]__ 2024/ Accessed 2025-09-23.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 2025. URL https: /ttp://aaxivvvv.ii org/ abs / 2504. 20571
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL hett/oorun21dd-1100665
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. Helpsteer2-preefrence:: Complementing ratings with preferences, 2024b.
URL httptttttss/tiiitteee org. hbtp//////222... 01257
Leon Weber-Genzel, Siyao Peng, Catie-Catherine De Marneffe, and Barbara Plank. VariErr NLI: Separating annotation error from human label variation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pp. 2256-226666 2269, Bangkok, Thailand, August
19

---

Published as a conference paper at ICLR 2026
2024. Association for Computational Linguistics. doi: 10.18653/w1/2024aclllogg 123. URL
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022. URL
https://arxix: wwg/aogg//sooo.ooo /ass//ab//// 2109. 01652
Peter West and Christopher Potts. Base models beat aligned models at randomness and creativity, httpbb///555500000000 00047
2025. URL
Peter West, Ronan Le Bras, Taylor Sorensen, Bill Yuchen Lin, Liwei Jiang, Ximing Lu, Khyathi Chandu, Jack Hessel, Ashutosh Baheti, Chandra Bhagavatula, and Yejin Choi. Novacomet: Open
commonsense foundation models with symbolic knowledge distillation, 2023. URL https:
//arxiiv.rro/aass/222220000
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization, 2022. URL http://s///xxxio.rg////eee
08791
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's s transformers: State-of- the-art natural language processing, 2020. URL
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https:ttt//.. /ttp://axxii.... http//awwwvv.....aa.a/o/oo/ooeat http:/bb/255000 09388
Shu Yang, Muhammad Asif Ali, Lu Yu, Lijie Hu, and Di Wang. Model autophagy analysis to expli- cate self-cconn within human interactions. In First Conference on Language Modeling, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence?, 2019. URL httt://orummmd==x==0000
07830
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Confer- ence on Learning Representations, 2024. URL htttps://openne
ew net. / forum?io
trCkidwPL.
Jayy Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, and Weiyan Shi. Verbalized sampling: How to mitigate mode collapse and unlock Ilm diversity,
2025a. URL hatttps://arivv. http:///aaxxvv...////////ss wwwpg/ab////ee 2500 01171
Lily Hong Zhang, Smitha Milli, Karen Jusko, Jonathan Smith, Brandon Amos, Wassim, Bouaziz, Manon Revel, Jack Kussman, Lisa Titus, Bhaktipriya Radharapu, Jane Yu, Vidya Sarma, Kris Rose, and Maximilian Nickel. Cultivating pluralism in algorithmic monoculture: The community alignment dataset. arXiv preprint arXiv: 2507 09650, 2025b.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction uning for large language models: A survey, 2025c. URL petettss//arxiv...o www.a.iiiivv..... wtw.://rrgggbb.. http///a/////// 2308 10792
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, and Daphne Ippolito. Forcing diffuse distributions out of language models. arXiv preprint arXiv:2404. 10859, 2024a.
20

---

Published as a conference paper at ICLR 2026
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, and Daphne Ippolito. Forcing dif- fuse distributions out of language models, 2024b. URL htt.ttta///axxii.or///////////// att.
10859
Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity, 2025d. URL /ttpps//aaax....o///////////// 05228
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Alan Zhu, Parth Asawa, Jared Quincy Davis, Lingjiao Chen, Boris Hanin, lon Stoica, Joseph E. Gonzalez, and Matei Zaharia. Bare: Leveraging base language models for few-shot synthetic data generation, 2025. hhttps://axxiii
http://xxxv..rggba//ao/oaoaa wwt.:/r////.........ooo 01697
21

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
We hope that SPECTRUM SUITE can serve as a useful resource for others to evaluate and train mod- els that support better in-context steerability, valid output coverage, and distributional alignment. We also believe that SPECTRUM TUNING serves as a useful step in improving these desiderata. However, work has several limitations.
Experiments performed only on 144B parameter models. While we have ensured that results gen- eralize across 3 model families, all models tested are in the 8B-14B parameter range. We have no reason to believe that our findings will not scale to larger model sizes, but this remains to be empirically verified.
Not optimized for chat. While most current post-training techniques optimize for (po- tentially multi-turn) chat, models trained with SPECTRUM TUNING instead focus on the description// wwt/input/outpuuuu framework. While it can be possible to elicit chat-style mes- sages via few-shot examples (see App. Nj from ST models, we would expect that instruct models would be better at outputting single chat response that is preferred by humans. It may be possible to combine the desiderata with a chat-style model, but they may also be fundamentally in tension, requiring distinct models for diversity/coverage and for chat. 5
Additional work needed on safety guardrails. Currently, models trained with SPECTRUM TUNING always attempt to steer to the description and examples, regardless of content. This is, of course, also true of pretrained models, which is one justification for why a model developer may choose to keep certain pretrained models with advanced capabilities unavailable to the public. All of our experiments are with models with public pretrained variants, and we do not believe releasing our SPECTRUM TUNING models enable any fundamentally new capabilities over the pretrained variants, but rather increase alignment with the desiderata. However, a pretrained model has potentially harmful or dangerous capabilities that a model developer wishes to restrict, SPECTRUM TUNING would need to be modified to adhere to these restrictions. While it is easy to imagine potential extensions to e.g. refuse to produce an output that violates policy, we leave such exploration to future work.
B SUPPLEMENTARY FIGURES
In-Context Steerability
Distributional Alignment
Output Space Coverage model follow instructions and produce quality cutputs while avaiding mode collspse?
How well does model adjust its outputs based description and examples??
Can madel match
target distribution?
(even when it may not be the modal respionss)
dalle crany
"srite ha about snark. mmalld Cartool Scatew
Deacriston
IndiviBall' PELISENTT
Vipuut
Volld Output Spaco
Incurt
Dutputt Space
20-00200
100
1022300-42
Liautal maniarree can Falaa 11gh1?
100
100
Pharaind
Instrcc Models
Sew sccs 3q)
Got futter tng woe?
Nodel
X
Prstralived Modilts
High duenity
chen product
SgP cometage of no
Irstrcc Meder's
us strkkc doear tabeags
Figure AI: Three desiderata for conditional distributional modeling. Example outputs and data are drawn from google/ /ttp///mmmmaa/eee
22

---

Published as a conference paper at ICLR 2026
habernas, larimms_inivviiii individuull categooricaa chatbota_art _arena__a__a______ aerreeaaindvviiieeeeeeeeree individual
novacomet, premise
colieing ondienccae fora sirgle Indiiual Theyaa perara
enkkent futryer wiih
innuuee medel topeeng prowrt and hee low nen the voparer
Thouee and Jonel
Goualom Woot which commutatte
reuled
Apws Hgre:
optioss
nornal L
Dowires Oivernes
('owinn pompet oifh Coreet grammnnt hepps: "meet suppree::
of Dum loar
centar and tide socul Uie wol
Welpord
bes novter
Trodel Muneore "The carrec gran rould hropeu"))
th carsacaa tue that pou vouud like me to sonscrn
dli oution Procico
Coon what wow.eeeetttd..ooo
(ris Aearge preeencee
2000-ond, Jump
oraa, of Jam-oepee
tharge
180
a.aneehh tat to Whut and
"raittu "uraoraa"""" "aaaart ling ui waink anceet of mapagng
18002 Lat.lan
waae Li Jarieeiee
of Jamiigg
toples duuueeeen seree'' arsunel
you liarr
uoged Fut me hua uo
hus marchallt Tweett
Pocker coal
Prodic:
Commomy aadd Lame
conmrration
about"?
Lak Lit lesraeed
rene clamm
piver fintt pridee hoa
palaranc
Task: predict diverse synthetic
Task: prodict survey opinions
Task: predict given user's chat preferences
Task: infer normal distribution
Figure A2: Example tasks from SPECTRUM SUITE in the format used for SPECTRUM TUNING, In our method, we shuffle the data, put it into the above format, and finetune with cross-entropy loss only on the (highlighted) output tokens, including the terminal token.
C FREQUENTLY ASKED QUESTIONS, INTUITIONS, AND HYPOTHESES
Q1: What unifies the three desiderata?
AI: At first glance the desiderata may not seem very related, but they actually all have something in common they all have to do with tasks where there is not a canonical, single correct answer. Rather, all three desiderata involve either matching or steering to broad spectrum of potentially valid answers. This is in contrast with the majority of tasks on which we currently train and evaluate instruction-tuned LLMs.
Q2: Why does instruction-tuniin post-training lead to spiky distributions and mode collapse?
A2: We have two principal hypotheses for this: 1) the RL objective in RLHF//POOGGRP//e. encourages the model to collapse its distribution to the highest reward output (c.f. WWest & Pouss] 2025) and 2) most instruction-tuning training and evaluations focus on tasks with a single verifi- able answer. While outside the scope of this work, comparing the desiderata at different stages of instruction-tuning (e.g., during and after Instruct-SFT, during and after RL) would help to elucidate this 10
Q3: makes sense that SPECTRUM TUNING improves in- in-context steerability, as it maps easily onto the training data format. However, why does Spectrum Tuning improve diversity and distributional aligmmen/calibraatonn
A3: While we hope to flesh out our understanding of this mechanism in future work, our best intuition is this It largely has to do with the fact that 1) all training tasks involve interchange- able data and 2) we shuffle the data before training. As a simple example, let us consider the diffuse di distributien dom, without replacement." In training, we collect list of all countries in Asia, shuffle them, and finetune on them as outputs: e.g., "Brunei", "Lehaano"", "Singapore", "Laos", "Vietnam", instruction-tuned model will often exhibit mode collapse- outputting the same country each time. Meanwhile, a base model will often output a valid country, but is heavily affected by training data frequency / n-gram statisties. In contrast, in the limit, Spectrum Tuning encourages the model to actually instantiate a uniform distribution over all countries in Asia increasing the diversity of outputs across many samples. For distributional alignment and calibration, it is a similar story base models are heavily affected by things like n-gram statistics, instruct models have uncalibrated, spiky distributions. contrast, Spectrum Tuning in the limit encourages the model to fit the actual described distribution, (partially) overcoming n-gram frequency.
task: "Output a random country in Asia, chosen completely at ran-
An
For example of the checkpoint setup one might use, please refer to Bhatia et al. 2025 where they explore the effect of post-tuning on value drift.
23

---

Published as a conference paper at ICLR 2026
D SPECTRUM SUITE DATA SOURCES
D.1 DATA CONSTRUCTION
As SPECTRUM SUITE is the first-such large-scale resource of such subjective datasets requiring steering, it was necessarily constructed in somewhat ad-hoc manner. However, here we provide some general principles for data that we attempted source:
1. Any NLP datasets with corresponding annotator IDs, allowing us link multiple anno- tations to the same person. We especially sourced from datasets where variation is to be expected, as opposed be eliminated.
2. Datasets related opinion modeling computational democracy;
3. Synthetically--eneet generraeed NLP datasets;
4. Lists of interchangeable things;
5. Draws from random distributions;
6. Tabular data.
D.2 DATA SOURCES
Below, we cite all data sources used SPECTRUM SUITE. Additionally, we include any subtask names along with the number of sequences included in SPECTRUM SUITE. We release the pro- cessing code to go from raw data to our description/ wttiiiintttuuuute in our github repo pestps://githe
www/tsorll spectrum
Note that many data sources have much more additional data that we could utilize (e.g., OpinionQA (Santurkar et al. 2023), Polis (The Computational Democracy Project 2025). synthetically gener- ated random data). We generally restricted each data source to a maximum of 1-2k sequences to ensure training data diversity, and all but couple of cases with very few data instances (e.g. Diffuse Distributions: Zhang et al. (024b) additionally ensured that the same piece of data was not used more than one sequence.
D.3 TRAIN SPLIT
Ambient Ambiguity Detection (Liu et 2023
ambientt.mmi.gultty.eeeee...... (50 sequences)
amb.eart.amoteationniisttttttttt (50 sequences)
aubilent..diaanbb.ppattto....... sequences)
amb.enttiinteppppettiiioolllbll (50 sequences)
ambi:ent_IInggilss.mmnttttttoo (54 sequences)
ambbient.preemmss..yppttttst (50 sequencc)))
Social Security Administration Baby Names (occial Security Administration
2025
babynames (500 sequences)
Base-Refine Synthetic Data Generation Zhu et al. 2025
bare_enron (55 sequences)
bare. rare-ssmkk (108 sequences)
bare Dare_hotpot (50 sequences)
bare lcb (136 sequences)
Darennewsgroups (60 sequences)
bare pubmed (46 sequences)
24

---

Published as a conference paper at ICLR 2026
Draws from binomial distribution (generated)
binomia all (500 sequences)
Draws from shuffled deck cards (generated)
cards (100 sequences)
Draws from categorical distribution (generated)
categorical (500 sequences)
ChangeMy View Reddit (Kolyada al. 2020)
changeemyiiew.cattaorriss (809 sequences)
changemyvee.poots (1159 sequences)
Draws from biased coin (generated)
coinflip (1000 sequences)
Collective Alignment Dataset (OpenAl 2025)
collective..lignneeet.... colleettive.llleeettttldllllllie individual (993 sequences)
Community Alignment Dataset Zhang al. 2025b
cammuuaiyyaaiinneennnnnn///aeeee (770 sequences)
comunllyyalgggnnnee-.dddvvdduuu_ (1031 sequences)
commutttyallggnenn...tttaaaapppp (139 sequences)
commuuttyyylliinmnntt.eessppsee 1941 sequences)
DICES dataset (Aroyo et al.
2023
dices (295 sequences)
Diffuse Distributions (Zhang et al. 2024b)
ciffuseedistributioo (270 sequences)
Generative Social choice FFish et al. 2025
seperativeecciaalhool....etttttr (200 sequences)
gencaativeeoccalccoo.eea........ (400 sequences)
Draws from geometric distribution (generated)
gecmetric (500 sequences)
Draws from a geometric beta distribution (generated)
geametric.bet. (500 sequences)
Grade-school math problems (GSM8K) (Cobbe et al. 202
gemkkLansseer.ffroonuuueesttto ion. (50 sequences)
gsm8k qunikkiqeetiin (50 scquences)
gam8k question answer (50 sequences)
gam8k question from answer (50 sequences)
25

---

Published as a conference paper at ICLR 2026
Haikus (Neiman 2018
haikus (600 sequences)
Hatespeech annotations from diverse annotators (Kumar al. 2021
hatesppecchinndiiittall (1000 sequences)
Helpsteer2 Synthetic Chat Preferences (Wang et al. 2024b
helpsteer (320 sequences)
Draws from hypergeometric distribution, generated (Wang et al. 2024b)
hypergeometric (500 sequences)
IssueBench (measuring political leaning of LLMs)
(Röttger al. 2025
issuebench (4 sequences)
Jeopardy! questions and answers (trexmat 2014
jecpardy begerddy.nnavr..peeiictt... (1000 sequences)
(1000 sequences)
Sarcasm detection (multiple annotators)
(Jang & Frassinelli, 2024
(872 sequences)
Irony detection (multiple annotators)
(Casola et al. 2024
mepiiimmp.izee wwt.ddeeetiinnnniiiii....ie (475 sequences)
Paraphrase detection with rationales (multiple annotators)
Leonardelli al. 2025)
Lewidi par wtt.:pppparr.eeeeeettioo..ooooda
(80 sequences)
dewidi par paraphrase quenees)
(80 se-
Entailment (multiple annotators)
Weher- et al. 2024
Ber.idairrriarmnniiiiieeeettttod (52 sequences)
hwwpd//aaaarmmmna...eeeccoaaaaaa quences)
(52 se-
Draws from multinomial distribution (generated)
mult t nomin al (500 sequences)
Draws from negative binomial distribution (generated)
regative..biomial al (500 sequences)
Netflix views and rating data Netfllx Inc. 2009
netffix individual ratings (1000 sequences)
netflix individual views (2000 sequences)
Draws from normal distribution (generated)
normal (1000 sequences)
26

---

Published as a conference paper at ICLR 2026
OpinionQA: Large-scale opinion survey dataset (Santurkar al.
2023
opinlongg.individuulll (3000 sequences)
opinionga.question (15 sequences)
Draws from poisson distribution (generated)
poisson (500 sequences)
Polis OpenData: Votes from digital town hall (The Computational Democracy Project 2025)
polis.comment (336 sequences)
polis_vote (7452 sequences)
Popquorn: Annotator disagreement on NLP tasks, with demographics (Pei & Jurgens 2023
innguuornindiividaal. (400 sequences)
.ppporo..g..tteerri..... (80 sequences)
Prism: World-wide, pluralistic chat preferences (Kirk al. 2024b)
priiom_indiiiidaaapeeeeeeeeeeee (1333 sequences)
prismpprompts (54 sequences)
prism.prompts indivviuul (1393 sequences)
Titanic survival prediction: classic machine learning tabular dataset (mstz. 2023
titanicc.ll..ariibbles (14 sequences)
titaadcc.auriivaapppeiic.... (14 sequences)
Value Consistency: Multi-lingual value laden questions (Moore et al. 2024
valueconsistencc (21 sequences)
ValuePrism: datasets with moral judgments and relevant values, rights, and duties (Sorensen et al. 2024a)
valueprism.miss (400 sequences)
waluuprissn.situttion (105 sequences)
valueprism..rr (500 sequences)
valuepriam baaluprrsemm.rd...nnccottetettee (74 sequences)
Draws from zipfian distribution (generated)
zipfian (500 sequences)
D.4 TEST SPLIT
ChatbotArena Individual Preferences [Zhnng et al. 2023)
chatbotarena.aasisttant (928 sequences)
chatbotarena individual prefs (1183 sequences)
chatbotarena prompts (1000 sequences)
Tabular Chemistry Dataset (Ramos al. 2023
chemi stry escl (310 sequences)
27

---

Published as a conference paper at ICLR 2026
chemisttxy.oxicaziive (102 sequences)
Synthetic Flight Preferences (Qiu et al. 2023)
flight (200 sequences)
GlobalOQA: Country-specific Value Surevy Distributions (Durmus al. 2023
globaloqa (274 sequences)
Habermas Dataset: AT Deliberation with UK residents (Tessler et al. 2024
habermas.indiizdduall (1996 sequences)
gabbermas.indiiiiduul.....
(2000 sequences)
habermas opinions (199 sequences)
habermas qutatiisessiiioon (43 sequences)
NovaCOMET: Synthetic Commonsense Dataset (West et al.
2023
novacomet hypothesis (170 sequences)
novacomet premi se (68 sequences)
NumberGame dataset: cognitive science dataset used study human reasoning under uncer-
tainty (Bigelow & Piantadosi 2016)
numberggameinndiiittal (606 sequences)
numbergame_percc (182 sequences)
World Values Survey, Wave 7: Global survey on human values (EVS/W 2024
wvs.individual (2000 sequences)
D.5
CAPABILITY SPLIT
AI2 Reasoning Challenge (Clark al. 2018
are (118 sequences)
DROP: Reading Comprehension (Dua al. 2019
drop (943 sequences)
GPQA: Google-Proof QA Benchmark (Rein et al. 2023
gpqa (995 sequences)
Hellaswag: commonsense benchmark (Zellrs et al. 2019
hellaswag (503 sequences)
IMDB sentiment classification (Maas al. 2011
imdb (192 sequences)
MMLU: Massive Multitask Language Understanding Benchmark (Hendrycks al.
2021
mmlu (1000 sequences)
TruthfulQA: factual questions Lin et al. 2022b)
truthful.co _qa (69 sequences)
Winogrande: Commonsense sentence completion Sakaguchi et al. 2021
wi wingrandee (127 sequences)
28

---

Published as a conference paper at ICLR 2026
E EFFECT OF TEMPERATURE ON DIVERSITY VS. VALIDITY
Temperature can have a major effect on the diversity vs. validity tradeoff when sampling from a model. In [ observed that, when sampling across three levels of prompting information and three model families, Spectrum tuning offered pareto improvement on diversity vs. validity and overall improved yield. However, the question still remains does Spectrum tuning still offer an improvement, even after sweeping temperature values?
To answer this question, we evaluated the same models under the same setup, but sampled at various temperatures: [10, 5, 2, 1.5, 1, 0,9, 0.7, 0.5]. In Figure [A]] we plot diversity vs. validity for all three model families, prompting methods, and model types. We find that, eight of nine settings, Spectrum Tuning expands the diversity validity Pareto frontier, as compared to using instruction-tuned
pretrained models alone. In addition. Spectrum Tuning models typically expand the Pareto frontier in the high validity region, increasing diversity for given validity. In line with the temperature=1 results, Spectrum Tuning's gains offer largest improvement in lowest information setting, when only description of the task is provided.
Model Type
pretrained
instruction-tuned
spectrum
Description
3-shot
Description + 3-shot
100
100
100
Gémma- --mma-
% Unique /allid
60
60
40
40
40
20
20
20
20
100
20
60
100
20
100
100
100
100
% Unique/ Valld
80
80
80
Qwen3-14B
40
40
20
20
20
20
60
100
20
60
100
20
60
100
100
100
100
1lama-3.1-88
% Unique Valid
60
80
80
60
60
20
20
20
40
60
80
100
20
40
60
80
100
20
40
60
100
Valid (%)
Valid (%6
Valid (%6
Figure A3: Effect of temperature on diversity and validity. Tested temperatures: [10, 5, 2, 1.5, 1, 9, 0.7, 0.5]. Lines connected for temperature in ascending order, with the right-most endpoint being lowest temperature and the left-most endpoint being highest temperature. Spectrum Tuning generally offers Pareto improvement, especially in the high validity region.
In Figure A4
we also plot the yield for each setting against the temperature. We find that in eight of nine cases, Spectrum Tuning offers the highest possible yield across all models and temperatures implying that, even when selecting the optimal temperature for each generation task, we would expect the highest number of distinct valid generations from the Spectrum- Tuuned models.
29

---

Published as a conference paper at ICLR 2026
Model Type
pretrained
instruction-uun
Description
3-shot
Description + 3-shot
50
50
Yield from 100 gens
Gemma-3-122 3-1bb
40
40
30
30
30
20
20
20
10
10
50
50
50
Yield from 100 gens
40
40
Qwen3-148
30
30
0.7
0.9
1.5
0.5
0.7
2
10
0.5
0.7
0.9
1.5
50
50
Yield from gens
40
40
Llama-3.1-80
10
0.5 0.7 0.9
1.5
2
10
0.5 0.7
0.9
1.5
2
5
10
0.5
0.7
0.9
1.5
10
Temperature
Temperature
Temperature
Figure A4: Effect of temperature on yield across each setting. When selecting the optimal temper- ature for each model, Spectrum Tuning offers the highest overall yield in 8/9 cases (all but Qwen3- 14B / 3-shot). Spectrum Tuning also offers the highest yield in most temperature settings T <
Taken together, we find that the gains from Spectrum Tuning hold even when leaving temperature as free variable.
F GENERAL CAPABILITY PERFORMANCE
We test whether SPECTRUM TUNING affects general model capabilities. While we do necessar- ily expect our method to improve upon standard evaluations where there is single correct answer,
we want to understand if it degrades performance compared pretrained models. We evaluate general knowledge capabilities with Big- Big-Bench Hard (BBH, 3-shot, (Suzgun et al. 2023). GPQA
(5-shot with chain of thought, Rein et al. 2024), MMLU-Pro (5-shot with chain of thought, Wang et al. 2024a). and TruthfulQA (6-shot, Lin et al. 2022a): instruction following with IFEval ZZeng ict al. 2024); and chat ability with AlpacaEval v2 (Dubois et al. 2024). We use the default Olmes hyperparameters for evaluating pretrained models, and Tulu-y3 hyperparameters and task descrip- tions for evaluating instruction-tunm find that models trained with SPECTRUM TUNING perform similarly to the pretrained models, and in some cases exceed them; however, as expected, instruction-tune models perform much better, particularly instruction following and chat tasks.
models (Gu et al. 2025 Lambert et al. (025).. In general, we
30

---

Published as a conference paper at ICLR 2026
gemma-3.122
Qwen3-14B
1lama-3.1-88
Dataset
ST (ours)
PT
IT
ST (ours)
PT
IT
ST (ours)
PT
IT
AlpocaEval2
5.935
6.897
53.846
30.421
33.541
63.123
3.642
3,579
24.641
BBH
0.738
0.727
0.821
0.786
0.789
0.770
0.641
0.631
0.722
GPQA
0.250
0.377
0.315
IFEval 0.407
0.436
0.806
0.712
0.726
0.871
0.377
0.296
0.793
MMLU-PPo
0.458
0.448
0.592
0.584
0.555
0.684
0.358
0.360
0.481
TruthfulQA
0.516
0.483
0.610
0.498
0,529
0.553
0,435
0.446
0.551
Table Al: General Capability Results. pretrained models perform similarly.
Worst performance is underlined. SPECTRUM TUNING and
G TRAINING DETAILS
We lightly tuned hyperparameters by training the gemma-3-12B model on a subset of tasks from SPECTRUM SUrTE-Train and tracking performance on held-out train tasks. We used the same hy- perparameters for Llama and Qwen, performing no additional hyperparameter tuning. Training for all models was done on four 80GB A A100 GPUs using DeepSpeed Zero3 (Rajbhandari et al. 2021 and Hugging Face Transformers (Wolf et al. 2020). Training took about 16 hours for the Llama models, 26 hours for the Gemma models, and 30 hours for the Qwen models.
Hyperparameters used:
max_length: 1024
per.@eviie..traiabaaatccssssis.. 1
anaiieattaccmmmlltiionnssss 512
Baarning.rate. 3e-6
learminggrate__cchedulee
linear_decay
H RESULTS WITH UPDATED HYPERPARAMETERS
After running the main suite of experiments for the paper and experimenting with the models, we had reason to believe that our Spectrum- Spentrum-Tuned models, especially the Qwen and Llama models, were underfit. Note that, for the main of experiments, we only lightly fit hyperparameters only on the Gemma models using a held-out subset of the train tasks validation set, and used the same hyperparameters for Qwen I.lama
To further explore the effect of updating hyperparameters, we experimented with reducing the batch size order take more gradient updates. the original hyperparameter mix, we use an effective batch size of 2048 (512 gradient steps x train sequence per device x 4 GPUs). We halve the batch size three times, and report aggregate results Table A2
ICL Steurability
Dist. Aliga.
Valid Output Coverage
Effective
MC Loss
MC Arc (Norm.)
Free-text
Dist. Align. JS-Div
Yield Description
Yield 3-hhot
Yield - 3-chot Description
(Norm.)
Less (Norm.)
2018 (original bpanam)
1.00
1.00
1.00
069
36.7
42.1
49.2
256
1.09
1,07
1.01
.063
45.9
457
52.0
Table A2: Hyperparameter ablations, averaged across models and tasks. Shaded are default SPEC- TRUM TUNING models. Best result bolded, second best underlined.
We find that 1) decreasing the batch size results in substantial jump in zero-shot yield, and slight improvements in few-shot yield and distributional alignment. Additionally. decreasing the batch size increases multiple choice accuracy, but at the cost of higher loss multiple choice answers. All in
31

---

Published as a conference paper at ICLR 2026
all, we think that this illustrates that there are likely to be additional gains from further optimization, and that our initial hyperparameters were likely underfit.
We think that the models trained with effective batch size 512 offer good tradcoff between ICL stecrability, distributional alignment, and valid output coverage, and report their full results in Tables and Figure
A3 A5
gomma-3-122 3-mmaa---12b
Quom3-14B
Elama-3. 3.1-BB
Datuset
Metric
Multiple- Cheice Datasets
gemma-3-120
Qwen.-144B
habermas individuul.cueguree (mun, k=2, (=)000)
Luss
3.53
2.50
2,01
2.62
9.10
2.58
2.58
2.74
24.4
24.9
22.0
23.2
19.0
wws Individual tmax (max_ 20-21, N-1000)
Loss
1.36
1.30
4.10
1.38
1.74
4:35
1.42
1.57
1.76
sumbergame Individual (max. 1--25, N-592)
Loss
.655
.005
1.00
.417
497
1.28
.611
264
.700
71.2
chatbotarens.c individuul..ppp...m prefs (ma, N=725)
Luss
1.52
1.62
4.94
1,35
1.47
4.39
1.43
1.76
L.77
Aec
48.9
38.0
44.6
51.7
52.0
46.3
39.5
36.0
39.5
N=200)
1.45
Acc
41.0
41.2
40.6
43.1
43.7
40.8
40.9
42,0
40.2
Free-text Datavels
gemma-
Qwen3-14B
Llama-3.1-88
novacomett hypothesis (mxx I=11, N=155)
Loss
129
106
preemin (=55, N=51)
Luss
27.7
18.0
35.5
27.7
27.5
38.0
27.9
27.7
28.6
habermas question (max k-29, N=30)
Loss
23.9
23.1
41.4
23,8
24.0
31.8
23.8
23.8
24.8
opinioos N=10000 N=1000)
991
habermas, habeermasJndiiidia Individual (max km2,
Loss
164
164
203
167
168
210
166
167
176
numbergamee pere (max J,-24,
globaloga (mox J-8. N-251)
Loss
14.2
14.4
21.5
14.0
14.4
30,9
14.5
14.7
15.6
chathotarena. cacttreennnppoomppe prompts ke3, N-995)
Loss
69.8
69.4
117
67.9
68.2
97.8
72.0
72.0
77.6
chatholarena sesistant
3-55, N-716)
125
chemistry. aso 1=88, N-59)
1-88
Loss
8.45
8.37
12.9
8.45
8.47
11.8
8.30
8.51
8.55
chemistry oxidative (max N-101)
Table A3: In-context steerability results on models trained with an effective batch size of 512.
Dataset
Qwea3-14B
Llama-3.1-815
pt
pt
habermus individual ccalegorical
0.13
0.069
0.239
0.049
0.198
0.108
0.084
0.055
uys_individual
0,007
0.015
0.225
0.007
0.191
0.005
0.012
0.024
numbergame Jndividual
0.019
0.029
0.163
0.037
0.026
0.108
0.027
0.024
0.017
chathotareno, iiwp:eiiiiiiiiillppe.....
flight
0.819
0.04
0.271
0.055
0,035
01228
0.03
0.07
0.038
Table A4: Calibration for models trained with an effective batch size of 512.
32

---

Published as a conference paper at ICLR 2026
Femily
Yiold From 100 ganz N Unique Valld Generations
Description
3-shot
Description 3-shot
90
40
Valid (16)
Valid (16)
42.0
60
24
40
Gemma
Owen Model Family
Lisma
Cenma
Owen Model Family
Ulmma
Gemma
Owen
Liama
Model Ramily
Figure A5: Diversity vs. validity on verifiable tasks for models trained with an effective batch size of 512.
gemma-3-122 3-12b
Qwen3-14R
1lama-3.1-88
Datasel
Metric
ours
ours
ft
pt
JS-Div
.126
.347
.107
.0928
.405
.0489
.0574
.331
rotten tomatocs
3S-D8v
.0227
.023
.134
.031
.0283
.122
.0245
.0354
.0.59
mytimee
/628
344
.I655
global oppa
JS-Div
.0678
.0966
.270
.0749
0878
.244
.0825
.I08
.63
J5-Div
habermas
J8-Di
.142
.147
.125
.127
.444
.129
.155
242
numbergame
JS-Div
.4488
.0400
04288
.131
.0423
.0000
0943
Table AS: Distributional alignment for models trained with an effective batch size of 512.
33

---

Published as a conference paper at ICLR 2026
I HUMAN EVALUATION
We conducted a large-scale human annotation study to evaluate the validity and quality of outputs from different model configurations. The study used pairwise comparison design where annotators evaluated outputs from two models simultaneously for the same prompts. We recruited 245 U.S.- based English speaking annotators who had submitted least 1000 prior tasks with an approval rating of at least 95% through Prolific and collected total of 2,400 annotations. Our task took about 30 minutes and we paid at least 7.5 USD for an average of least 15 USD an hour.
Specifically, we sampled 100 prompts from two evaluation datasets, a curated prompt set and infinite-chats-evall and collected human judgments for each. Our experimental design compared three model configurations (baseline instruction-tung tuned, our approach, and pretrained) in both zero- shot and few-shot settings. Each unique combination of (prompt, model pair) was evaluated by two independent annotators, resulting in 200 annotation instances per model pair per dataset.
Annotation Interface and Procedure Participants accessed the annotation task through a web- asked to thoroughly read through the comprehensive anno-
based interface. First, participants tation guidelines with examples of valid and invalid responses (See Figure A6 and Figure (A]. For each annotation instance, annotators were presented with a prompt and four generations from each of two models (labeled Model A and Model B). The model identities and presentation order were randomized prevent systematic bias. The interface displayed the outputs side-by-side to facilitate direct comparison (See Figure A8 for the user interface and questions).
For each task, annotators made three types of judgments:
Validity Assessment: Annotators independently marked each of the eight generations (4 per model) as either valid or invalid. We provided detailed guidelines defining validity as responses that directly address the prompt, follow all specified requirements, stay on-topic throughout, and contain factually reasonable content. Invalid responses included those that refuse to answer, violate format requirements, trail off into unrelated content, or contain sigmificant errors.
Diversity Comparison:
Annotators assessed which model's set of four outputs exhibited greater diversity, with options for Model Model or "about the same."
Overall Quality Judgment: Independent of diversity, annotators selected which model's outputs were better overall, again with options for either model or "about the same."
To ensure annotation quality, we implemented several measures: (1) Comprehensive annotation guidelines with examples of valid and invalid responses, (2) Tracking of time spent per annotation, and (3) Post-annotation feedback collection to identify any systematic issues.
Inter-Annotator Agreement Inter-annotator agreement for validity judgments showed 76.5% pairwise percentage agreement, with Cohen's 's я = 0.441, indicating moderate agreement. For the subjective diversity and quality assessments, agreement rates were lower (diversity: 38.8%, quality: 41.7%), as expected given the more nuanced nature of these judgments.
J LLM USAGE DESCRIPTION
In preparation of this research and manuscript, LLMs were used for:
Implementing code for experiments and analysis based on detailed author deseriptions. All
LLM code was inspected by the authors for correctness.
Formatting for tables, latex, and bibtex citation for non-traditional sources (c.g.. urls).
Draft critique by pointing out typos and potentially confusing wording in the draft.
However, all research ideation and writing was performed solely by the authors, and we claim full responsibility for the contents of this paper.
34

---

Published as a conference paper at ICLR 2026
Annotation Guidelines
Your Task
You will evaluate model outputs for prompts that have multiple valid answers. For each prompt, you'll from different models and B).
What You'll Annotate
Validity: For each individual output, whether it's valid/comret response the prompt
Diversity: Compare variety responses between the two models
Overall Quality: Judge which model's outputs are better overall
More Guidelines on Validity (Important!)
response VALID it:
Directly addresses what the prompt asks for
Follows spectfic requirements prompt (length, format, style,
Stays on topic throughout the entire resporse
Is factually reasonable (doesn't contain obvious falsehoods)
A response is INVALID if it:
Ignores
prompt
Violates spectfcc requirements (e.g, prompt asks for 'a few sentences" but gives only one word)
Starts correctly but then trails off into unrelated content
Contains significant emror nonsensical content
A Special Note on Cut-off Responses: Some responses may be cut off mid-sentence due to length Imits takens). If clearly incomplete but what's them vaiid and on-topic, consider existing
Only
problematic.
Figure A6: Guidelines for human annotation
35

---

Published as a conference paper at ICLR 2026
Examples Guide Your Judgment
Example Following Requirements
Prempt: fiww serbences what threats do indivedusls?
"Saaes can cause e Rinence less to the victim, potentially leading to deat, bankmupiyy and ever moncal and emational harm
Response:
X INVALID
Rewann: The prompt enpleclly "a sentaraaa the neporee is only one sentanow.
Example 2: Trailing Off Topic
Prompt: berel regular exardse?
"Reyulor improver cardovessslrr strergthere chron's discssns
resl belng, help ald know that dolphine actonly memmeis, fish? They firettie thei young...
X INVALID
Wosson: The response stars compeity but then trals off into complettaly und sodd content about deghtrs.
Example Cut-off
Prompt: Describe the water cydi..
"The water cycle a conthuuus procese wherre water mouss (irough Farth's atmooapher, lend and cceans, it begits with eraparation, where water from occens, likes, and niers tums into uppor
Aesponse:
atmosihore preditattdd neavy enough, evembully makiig may back the where the
suc. vaper
pracjstation
VALID
Brewon: Although of the responne off miidoand ( ("segi. exteting cuttent in accundt,, dn-itplc, and adcimous prompt wed.
Example Refusal
Prompt: Write halku about the occon.
Resconse: camoo wnlt creative contect like pootyy Would you Me me provide infommtión about ocener insteed!
X INVALID
pronpt though wittig request.
Example S: Good Valid Response
Prompt Name three primary colors.
Ausponse: throe prevary colors are red, Mha, and yollow.
VALID
Other Important Notes
complete
The model order (A/t) mindomined focus the noe wiith side they're
Consider each outout indeperdendy when sudging waiiity
For diversity and quality quest ons, consider el outputs from each model together
Take your time there's no rush. Quality metters more than speed
Figure A7: Guideline examples for human annotation
36

---

Published as a conference paper at ICLR 2026
Cappraal anoaicas Ueanue 12 acuee))
mggg magiol
takinn rong
Avario magicel fore ariing oi glowing mustrrs or emagial unicam.
giin
Amimenning covered chidt, humninng with hart
be partil place anderr power, or simpl beauttall reutter sndrarn
mhese when in tal und fumase stoies triums respiaa norts Addiionall the fenat night acomedd with fantutcall slants in peoullrr croparteo
wnt grnning creaing enercher lig the bul arropphine
could be cortel siice power, jst realy cost magical pork spot.
Antrenwiig cowere whippar fororter proph aclas touched.
autputs would you diverus (le,, most different other??
to surgetc duense
If you had pick, which set of outputs do you think is better everali, regardless diversity?
Sub
Figure A8: User Interface for Annotation
37

---

Published as a conference paper at ICLR 2026
K SUPPLEMENTARY FIGURES
Spectrum Suite (Categorical) Relative Loss After Instruction-Tuning
hypergeometric
[K max 70. N=5001 2.02%
2.298
zipfian
IK_max 69, N=470) 2.96
3.679
categonical
(K
64, N=488) 2.19g
2.763
negative binomial
[K 7. N-493) 1:125
1.423
geometric
ma N=500) 0.551
(0.772
coinflip
(K - N-869)
1.133
titanic survival ambient ambiguity dete...
preedl pred....
[a_ N.1410.727
1.408
[K_
22339
lewidi mp irany. detect...
[K
numbergame individual netfixx individual rat...
- gemma-3-12
34, N-823)
asm
N=B10) 1.597
popquor individual
(K N=188) 1.415
:3.728
(K.max 20, N-1000 1.502
44.105
lewidi paraphrase
max 11, N=76) 1.878
5:136
opinionga Individua
[K [K.aax = N=1000) 1.166
3.440
chatbotarena undividua...
hatespeed eech individual
[K [K_max - N=1000) 1.192
IK ma.max - N 1.618
3.518
4,940
fight
[K 90. N=200) 1.316
4.057
dices
max N=270) 0.888
2.853
ambient linguist annot...
max
6.587
lewidi variernii
max
imbieet__intepreetae
1.112
polls vote
(K N=1000) 0.874
33229
habermas_indivldee _
(K_ max 2=N00000 2.504
10.509
geometric hypergeametric
[K max
N=500) 0.516
[K max 70, N=500) 2.003
22352
cointlip
(K max 71. N=869) 0.585
10,708
negative binomial
IK max 74, N-493) 1.0%
11.399
categorical
IK max 64, K-468) 2.167
2.504
Ziplian
(K max 69 N=470)
14.086
lewidi mp irony detect... lewidi esc sarcasm det...
max N-308) 0.893
F1.374
titanic survival _pe_d__r_._ ambient Interpretation..
148 -
max 15, N=B10) 1.703
max
popquorn individual ambient ambiguitt dete...
N=188 1.44]
[K max N=50)
2.308
numbergame
(K max N=592) 0.697
1.2299
hatespeech individual netfix individual
indiividuu
(K max N.N0000 1.203
2.2216
(K max N=823) 1.374
12.586
lewidi_ paraphrase
P
[R. max N-76) 1.955
13.844
lewidi variernil nli
IK max
N-51) 2:183
4.266
ambient Tin guist annot...
IK max -21, N=37) 2.153
4.549
dices
maxa 99 N=2700 1.158
2:564
polis vote
(K max max 20, N=1000 1.048
12.356
Fight
max N-200)
wvs Individual
(K max N=751
chatbotarena Individua...
opiniongs
(K
N=1000)
individual,
(K max
N..0000) 2.622
9,102
numbergame individual
0,770 0,864 (K max - N-592)
titanic Survival pred... nettlix[Indivioe rat...
0.857 (K max 16. N-14
1.456g 512 (K max =4, N-..3-
Tright
1.414 .43 max = 9, N-200)
categorical
2.212 max 64, N=488)
lewidi_ _acllppar__a paraphrase
2.0220.037 max 19, N=76)
negative_bi binomial
1.138
N-493)
coinflip
(K
0.6220.622
chatbotar Individua...
hypergeometric
(K N-7251
1.757
habermas individual B
ziplian
N=470) 3.023
lewidi_ma mp_ _ardddmppoooyyyoo_oee detect...
[K ma
6=300) 0.73:0.755
geometric
(K max N=500) 00544
(K max N-1000) 2.575 227440
popquorn Individual
(K max 18. N=188)
1.460 1.558
lewidl csc_sarcasm det...
[K N=610 (1.-110)166220 1.778
palis vate
(K max - 20, N=1600)
0.947 1.000
ambient ambiguity dete...
max
N=50) 1.329 172221111477 11.479
ambiert, linguist ännot...
[K 21, 2,28: 2,5499
wvs individual
(K (K N-1000) 1.572 1,760
lewidi variernli nli
[K
Satespeech individual
N=1000
ambient, interpretation...
N=49) 1.325
dices
max
N=270)
+1.223
-200
-100
100
200
300
Loss Change (%)
Figure A9: SPECTRUM SUITE categorical loss after instruction-tuning
38

---

Published as a conference paper at ICLR 2026
Free Text Relative Loss After Instruction-Tuning
hahermas opiniens
Jewidl peraphass
WOUAT Monnin Pop
amb Raa prsm RE
commont SAANS
35mal M
Aiant colls como
gamill P Co Rap
gam
Retrli tha Tous
stan pri
Ohe mathhtarane ag3 Sopardy answer predic..
habermas apinio B
popelange do
commin te aua
novaro mo Eytarion
genera mia
communiy
ComI lewidi numbor Sehae
che
change mo
valuep Ham Pal
valuge stanic
lewid va De
individia Conaee Gier GAA
gombk, QUSTP queatio
apmak ansver from ques commmutty Eass gsmak question Fh
community
valueprism dode
now
nettlik Indii (ndl 2A
poll com
nov noviromm CAN Sereeen
generatvess Seeeeeei
amaknn
gamet
WO from
-200
200
400
600
Loss Change (%)
Figure A10: SPECTRUM SUITE free-text loss after instruction-tuning
39

---

Published as a conference paper at ICLR 2026
General Capabilities Relative Loss After Instruction-Tuning
hellaswag
[K [K = N=470) 0.933 1.039
3-12b 3-1b
Imdb
(K_max - N-1466 0.346
0.416:
deming 1
(K_mxx 9, 0.480
0.812
truthful_gg .a
(K m N-66) 1.095
2.769
K_max N-873) 0.924
2.399
drop
N=-66) 2.734
winogrande
(K_max N-126) 0.782
gpqa
[K Kama N= 1.574
4.884
Imdb
01380
0.589 (K_max = 4, N=146)
hellaswag
owent 3.m33144
4
(K_max = 7, N=470) 0.703
0.976
(K 9, Nm117) 0.473
0.821/
gpqa
(K 7, N-865) 1.743
3.100
truthful_gg qa
(K Nm66) 1.066
mmlu
max 11, N-873) 0.871
winogrande
(K max N=126) 0.924
2.599
drop
(K max N-866) 3.192
9.453
imdb
0.285
(K_ 4. N-146)
hellaswag
0.982
1.365 (K_max N=470)
Llama-3.1-8B
arc
0.709
(K_max N=117)
truthful qa mmlu
max
1.022
[.212 max 11, N-873)
winogrande
0.902
1.010 (K, max 9, N=126)
drop
3.395
5.588 (K max 4, N-866)
gpqa
1.519
1.602 (K max 7, N=865)
-150
-100
-50
100
150
200
Loss Change (%)
Figure A11: SPECTRUM SUITE general capability loss after instruction-tunne
40

---

Published as a conference paper at ICLR 2026
L
SPECTRUM TUNING TEMPLATES
For all templates, loss is calculated on the highlighted output tokens.
gemma-3 (w/ inputs)
<start_of_ ttt._/of__tttnndesrirrtt...oo
DESCRIPTION TEXT<end_of_tumm
INFUT TEXT<end [FXT<end_of_turna
OUTPUT TEXT<<end_of.trr>
<start_of_turn>ingg
INPUT 2 Texi<endLof_ttrm>
<start_ of turn>output
OUTFUT TEXT<end,of.turn>
<start_ct sitr__f__tuuuiiiiuuu
INFUT 3 TEXT>> <start_of faaatt_of_tun>ouppuur
OUTFUT TEXT<end.of.turo>
gemma-3 (w/out inputs)
estart_of_turn>eeerrip
DESCRIPTION TexT<<edd_f__urm
<start of turn>output
OUTPUT TEXT<end.of.tur>>
<start turn>input
OUTFUT TEXT<end.of.toro>
Sttart_of_turn>input
OUTFUT TEXT<end.cf.turo>
Qwen3 (w/ inputs)
<lim_stare >description
DESCRIPTION TEXT</imm_nd]]
< <lim_starti |>inptt
INFUT TEXT< im_end>>
<lim_stare >output
OUTPUT TEXT<|
<lim_start >>input
INPUT T
<lim_start >>output
OUTPUT TEXT<|im_ens
/li__star|/ >>input
INPUT FEXT<|im_ends
lim_start| /Soutpt
OUTFUT TEXT< im.end|
Qwen3 (w/out inputs)
<lim_stard >ddescr Dddecription in
DESCRIPTION TEXT<] [exx<]im_end]]]
<lim_start puc
OUTPUT TEXT im. end
<m start >outpptt
OUTPUT
<im_start put
OUTFUT TEXT< im.end]
41

---

Published as a conference paper at ICLR 2026
Llama-3.1 1 (w/ inputs)
<lstart_header__t hwwder aaaadceeeeiaeeerrittooooaaaaaaaa
DESCRIPTION DESCRIPTION
INFUT 10211000....1000rrre.....0..0ee0
OUTPUT {10
INFUT
OUTFUT 212411........tttrrr...e.e.....1 ||
oet_ae/ee____lppppppeeeeeeeeeeee >d
INPUT [ewrr/oe_.dc0cccoerrrmeeeeeeeedd
OUTFUT TEXI<leot.idI> idl>.
Llama-3.1 (woout inputs)
DESCRIPTION [EXT<|eot_id]><< aatt {r meddhheder_ii__
OUTPUT TEXT<|eot.ipp {| hnra
<d hen__haadrr____eeeee _
OUTFUT |ttr:/coooollxxsaatttttooooooo bottput
OUTPUT TFXT<|eot_id|2...
M PRETRAINED / INSTRUCTION--UNED ICL TEMPLATES
Pretrained Template (w/ inputs)
Note that each output ends with newlines terminal token (coloring not visible).
Descripticn: DESCRIPTION TEXT
Input: INPUT 1 TEXT
Output: OUTPUT TEXT
Input: INPUT 2 TEXT
Output: OUTPUT TEXT
Input: INPUT 3 TEXT
Output: OUTPUT TEXT
Pretrained Template (w/out inputs)
Note that each output ends with newlines terminal token (coloring not visible).
Descript inn
DESCR INSCRIPTION IPTION
TEXT
Output: OUTPUT TEXT
Output OUTPUT TEXT
Output OUTPUT TEXT
42

---

Published as a conference paper at ICLR 2026
Simple Instruct Template
Qwen3 (task w/inputs)
<lim_start, >ssstem
DESCRIPTION TEXI<| imm end
<lim_start Psuser
INFUT 1 TEXT< in indd||>
<lim_start Iassssstant <think>
</think>
OUTPUT T end <lim_start
INFUT TEXT<| inden|>> <lim_stard >assistant <think>
</think>
OUTPUT iExI<|im.end]
<lim_ >user
INFUT 3 TEXT< in__
<lim_start Passsssant <think>
</think>
OUTFUT TEXT< end
Qwen3 (task w/out inputs)
<lim_start |>systen
DESCRIPTION (oxr</im_end)
<lim_start />user
Generate<| im im__nd/>
<lim /ii_sttar/// >assistant
<think>
</think>
OUTPUT ind
<lim_start /user >user senerate</im_ente <lim_start Isassistant <think>
</think>
OUTPUT TEXT< end
<lim_start| sou.starr/l | >user
Generate< im indd|]|
< < start Passistant
<think>
</think>
OUTPUT TEXT< im_end
gemma -3 (task w/inputs)
DESCRIPTION TEXT
INPUT rexi<end_of_tumm
43

---

Published as a conference paper at ICLR 2026
<start turn>model
OUTFUT TEXT<end.of.turox>
<start_of
INFUT 2 TEXT<ond_of_turm
OUTPUT TFxT<end.cof.orm>>
<start_of _ftunuufuu
INFUT oFxtsend_of_turnt
<start_ of_turnnmodel
OUTFUT TEXT<end.of.turn>
gemma-3 (task w/out inputs)
Dutart_of_turn>use DESCRIPTION TEXT
Senerrate<end_o__turn>
OUTFUT TEXT<end of turn
Estart_of_turn>ee
snberate<end_of_ture of turn
<start_of_turnmmmoee]
OUTFUT TEXT<end.ef.turo>
<Start_oo_turn>user
Gemeraterend__of_es
<start_cf
sttart_of_turn>modde turnnmodell
OUTPUT TEXT<end.cf.turp
Llama -.1 (task w/inputs)
</begin _f__text____ooooooooeee__eee___e >> sstart Steeerreeraarrrr______ id >system< end header_ id/>
Cutting Knowledge Date: December 2023
Today Date: DD MM YYYY
DESCRIPTION httae//eeeeaaaaaa/aaaaaaaaaaaaaa
INPUT aer.eee......aaereeeeeee.aaaaaaa
OUTPUT 22t______0000000eeeee.eeeeeeeea1
INPUT wxXc://ot_1ddl13(trrrr1110
OUTPUT | haan
1d
INPUT [
header b header_id]
OUTFUT TEXT<|oot.id
Llama-3, 3.1 1 (task w/out inputs)
<|begin_cod of text |>< /stast header id >system< lend header id|>
Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024
DESCRIPTION
QUTFUT TEXI< TEXI<|cot.id]3 [e2x sua.aaerr.ddlssseeeceedddddddd
Senerate<<eet__d|| start_tttaah Sttar__haade____ id >assi stant and
44

---

Published as a conference paper at ICLR 2026
OUTFUT TEXT< header >user< lnd header i
cnd htneesssesttttttoooccceeeeeeeeer
OUTPUT TEXT<
Detailed Instruct Template
Qwen (task w/ inputs)
<lim_start >systen
You are tasked with generating cutputs from particular, potentially + stochastic, generative process. You will be given some combination of
+
Description: A natural description of the generative process / data
+ distribution
Input: An input which condition the generative process.
Example outputs: Example outputs from the process, either a user
S message or as prior generations from a chat message. You may assume
+ that any given outputs are exchangeable with one another (order- invariant) and generated from the
process (roughly i.1.d.), If the output data pertains to single object, it just contains the
+ output. If it contains multiple objects, use json formatting with
+ keys for the name of the output variable.
You will be provided at least either description
an example output.
Given these components, your jeb to generate JUST the output your
+ response, zoughly approximating the underlying generative process, underlying stochasticity (if any is present). If + are asked to generate again, you will sither be given an additional input to condition on, will just be told to "Generate".
+ maintaining
Descriptien: DESCRIPTION TEXT< | INFUT TEXT<| in_end>>
>user
<lim_start >assistant <think>
</think>
OUTPUT TEXT< imcc/limeedddd end
[lim_stare >usser
INPUT TEXT< >
<lim_start P>assistant <think>
</think>
OUTPUT TEXT<| end
<lim_stard Start/>usee >user
INPUT 3 sexT</in_endd
Passsistat
<think>
</think>
OUTPUT TEXT< end
Qwen (task w/out inputs)
< stantt >sssten
You are tasked with generating outputs from particular, potentially
+ stochastic, generative process. You will be given some combination of
45

---

Published as a conference paper at ICLR 2026
Description: natural descripticn the generative process data + distribution
Input: An input on which to condition the generative process.
Example outputs: Example outputs from the process, cither in a user + message
prior generations from chat message. You may assume that any given outputs
exchangeable with one another (order- + invariant) and generated from the same process (roughly i.i.d.).
+ the output data pertains to a single object, it just contains the
+ output. It it contains muitiple objects, use json formatting with
+ keys for the name of the output variable.
You will be provided least either description or an example output.
Given these components, your job is to generate JUST the output in your + response, zoughly approximating the underlying generative process, maintaining any underlying stochasticity (if any is present). you are asked to generate again, you will either be given an additional input to condition on, or will just be told to "Generate".
Description: DESCRIPTION TEXT< im_end|> <|in_start
@ <lim_start >asssistat
<think>
</think>
OUTPUT TEXT<|im.end
<lim_ start >user Generate<| end|> <lim_stard >>assistant
<think>
</think>
OUTPUT 2
<lim_stare >user Generate<| im_end|> <lim start Iassssstant <think>
</think>
OUTFUT TEXT< im.end
gemma-2 (task w/inputs)
You are tasked with generating cutputs from particular, potentially
4 stochastic, generative process. You will be given some combination of
4
Descriptien: natural descriptien of the generative precess / data
+ distribution
Input: An input on which to condition the generative process.
Example outputs: Example outputs from the process, either in user message or as prior generations from a chat message. You may assume
+ that any given outputs are exchangeable with one another (order-
+ invariant) and generated from the same process (roughly i.i.d.). If
+ the output data pertains to single object, it just contains the json formatting with + keys for the name of the output variable.
output. If it contains multiple objects,
You will be provided at least either a description or an example output.
Given these components, your job is to generate JUST the output in your 4 response, roughly approxinating the undarlying generati ive process,
46

---

Published as a conference paper at ICLR 2026
maintaining any underlying stochasticity (if any is present). you are asked to generate again, you will either be given an additional input to condition on, or will just be told to "Generate".
Description: DESCRIPTION TEXT
INFUT 1 TEXT<end_of_turmn
<start_e _tt___oftt____ee___________
OUTFUT TEXT<end.of.curD> <start_of_turn>>
INPUT TEXTKeend_of_tums
OUTPUT TEXT<end.cf turnn Eurn
<start_ of turn>user
INPUT TEXT<end of of turn> Secart_of__urnnnooe OUTFUT TEXT<end.of.turo>
gemma-3 (task w/out inputs)
You are tasked with generating outputs from a particular, potentially stochastic, generative process. You will be given some combination of +
+
Description: natural description the generative process data
+ distribution
Input: An input on which to condition the generative process.
Example outputs: Example outputs from the process, either in user message as prior generations from chat message. You may assume that any given outputs are exchangcable with one another (order-
+ invariant) and generated from the same process (roughly i.i.d.). If
+ the output data pertains to single object, it just contains the 4 output. If it contains multiple objects, use json formatting with keys for the name of the output variable.
You will be provided at least either description or an example output.
Given these components,
job is to generate JUST the output in your response, roughly approxinating the underlying generative process, maintaining any underlying stochasticity (if any is present). you are asked to generate again, you will sither be given an additional + input to condition on, or will just be told to "Generate".
Description: DESCRIPTION TEXT
Gonorateceed_.o__uunn.
<start_of__trrnmmotel
OUTPUT of <staatt_o_ttur>>ueer
Generaatecend_o__turn:
<start of ttt_uuuooddell
OUTFUT TEXT<end.cf.turo>
<start of <start_of turturuuser Senerateeeendoo__uurn> <start_of_turn>>ndd
OUTFUT TExT<end.of.turr>
Llama-3. (task w/inputs)
< <|begin_of_texx
id >system< l wewdeeeadddrr >
Cutting Knowledge Date: December 2023
Today Date: DD YYYY
47

---

Published as a conference paper at ICLR 2026
You are tasked with generating outputs from a particular, potentially + stochastic, generative process. You will be given some combination of
4:
Description: A natural description of the generative process / data + distribution
Input: An input on which to condition the generative process.
Example outputs: Example outputs from the process, either in a user
+ message as prior generations from chat message. You may assume another (order-
4 that any given outputs
exchangeable with invariant) and generated from the same process (roughly i.i.d.). If the output data pertains to single object, it just contains the
+ output. If it contains multiple objects, use json formatting with
+ keys for the name of the output variable. least either description an example output.
You will be provided
Given these components, your job is to generate JUST the output in your 4 response, roughly approxinating the underlying generative process,
+ maintaining any underlying stochasticity (if any is present). If you asked to generate again, you will either be given additional input to condition on, or will just be told to "Generate".
Description: DESCRIPTION TEXT< (or</ectt_20 ltte:/ee_ddd/<///teee/aa header id >user< + end header _dl>>
INPUT 1 120.:/oe_....K.aaarr/aeeeeaalaad
OUTPUT [exc</eet..dd>xxssaattteeeeeee end
/ header id|>
INPUT 2 [2XXiooc_.d||| tttrraaaaaaaaaa11 i
OUTFUT [BBB:ce....4000444aa___eeeedddd1
INPUT
OUTPUT TeXT</eat id
Llama-3, 3. (task w/out inputs)
beegin_of_text/ >>< /aspst////sstttsteeettaete [ssttttthhdde__iit http://aaaeerrilllsssssseeiiiiia [esttee//ee___hhaee____
Cutting Knowledge Date: December 2023
Today Date: DD MM YYYY
You are tasked with generating cutputs from particular, potentially
4 stochastic, generative process. You will be given some combination of
+
Description; A natural description of the generative process / data
+ distribution
Input: An input which to condition the generative process.
Example outputs: Example outputs from the process, either a user
+ message or as prior generations from a chat message. You may assume
+ that any given outputs are exchangeable with one another (order-
+ invariant) and generated from the same process (roughly i.i.d.). If the output data pertains to single object, it just contains the output. If it contains multiple objects, use json formatting with + keys for the name of the output variable.
You will be provided at least either description an example output.
Given these components, your jeb to generate JUST the output your
+ response, roughly approximating the underlying generative process,
+ maintaining any underlying stochasticity (if any is present). If
+ are asked to generate again, you will either be given an additional
input to condition on, will just be told to "Generate".
48

---

Published as a conference paper at ICLR 2026
Description: DESCRIPTION TEaK///oeLLiii//aaaaatt//aoooadd + ond_hoader_id)>
OUTFUT IEXI<|e0t >< |raa::o0__00K.aaaaaaaaoeoeoaaaa1
G start wtw.sstttttteeriiillee httaaa/e.........aaaaaaaaaaaaaaa
id|>
OUTPUT TEXT< 1d start h1
id sassistant heade
header i
OUTPUT TEXT<leot.id
Best performing instruct prompts We found that Llama-3, 1-8B-Instruet performed best on SPECTRUM SUITE with the pre- trained prompt, googl e /gemmaaaa---mmmmm 1esmmaa--112b-ae 12biit and qwen gwen/Qwente http///wen31111 43 performed best with the detailed instruct prompt. We utilize those prompts with the corresponding models for all ICL. exper- iments.
N OUTPUT COVERAGE DIVERSITY vs. VALIDITY EXPERIMENT DETAILS
N.1 VERIFIABLE EVALUATION
For this evaluation, we utilize the same prompts as in the ICL experiments see App. M
Below, we include the description and examples for each of the tasks. Please reference the codebase for validation functions.
Task: sor_innterestingge...
Description: Generate coler name.
Examples: l'Otterly Brown', 'Petal Pink', 'Cherry')
Task: color_normal_es
Description: Generate color name.
Examples: ['Green', 'Red', 'White']
Task: car sar_brand
Description: Car brand.
Examples: ['Acura', 'Ford', 'Tesla')
Task: car car_makc_modk
Description: Car make and model.
Examples: ['Acura Integra', 'Ford Mustang', "Tesla Model 3')
Task: states abbreviations Description: US state asoreviation Examples: I'KY', 'UT', 'OR'
Task: Lllstates_any_fonet
Description: state name or abbreviation Examples: ('Kentucky', 'JT' 'Oregen']
Task: us_states_full_ne names
Description: Name US state
Examples: ('Kentucky', 'Jaah', 'Oregon') ]
Task: prime_numben numbers
Description Generate prime number kxamples: ['617', 113' 147']
49

---

Published as a conference paper at ICLR 2026
Task: mmall_primm_numbere
Description: Generate a prime number less than 100 Examples: ['29', ' '
Task: basic_emails
Description: Email address Examples: ('ANONYMIZED' "alex. jee.sj.jneeettmmannnne.e",, Ltsagoodda&gmaal.comm 4
Task: professional_emails
Description Generate professicnal email address. Examples: ['ANONYMIZED', sarah. hwwcch//otaassbbrrgogaaaaaaaiooo + yasthanntiropic..omm
Task: weekdays_abbaee abbreviated
Descripticn: Day of the week abbreviation Examples: ['Thu', 'Wed.', SUNN]
Task: veekdays_any_forde format
Description: Day of the week (full name or abbreviation) Examples: ['Monday', "Tue', (SUN']
Task: weekdays full
full
Description: Name day of the week Examples: ('Thursday', 'Wednesday', 'Sunday']
Task: random_seed
Descripticn: Generate number Examples: ['15', '40'', '8392013']
for random seed.
Task: claude_gerunds
Description: Generate English gerund ending in -ing. Examples: ['S S'hheeppinn',, 'Hoisting', 'Thinking']
Task: r_g_1_10
Descript inriptien: en: Generate number between 1 and 10. Examples: ['3', '7', '10']
Task: rng 1 100
Description: Generate number between 1 and 100.
Examples: ['35', '94', '71']
Task: International__poom__mmms
Description: International phone number with country code.
Examples: ['+1 413-121-2591', +44 10 2958 3938', *+81 3 8328 5625']
Task: us__hhne_nummer
Description: US phone numper
Examples: [' (210) 346-0967', '646 562-1938', 9800-995-052"1
Task: el element_names
Description: Chemical element name Examples: ['Niobium', 'Antimony', lodine'
Task: element _lement_symbols
Description: Chemical element symbol Examples: ['Sb', 'He', 'W²]
N.2 IN-CONTEXT CHAT ELICITATION FOR PRETRAINED INSTRUCTION-TUNED MODELS
For inducing chat capabilities from the pretrained/spectrum-tte models for the Moveltyyeench//ffiiieeCCcC experiments, utilize the following few-shot
50

---

Published as a conference paper at ICLR 2026
amples. Responses generated by instruct model in the same model family: google gemma- - 2b-it, /wwen//wen3-r 32B, meta/ htt//LLama--300 1-10a--33-------ntcce-
For all pretrained models, utilized the following prompt template:
You are helpful AI assistant.
User: <EXAMPLE PROMPT1>
Assistant:
<EXAMPLE RESPONSE1>
User:
<INFERENCE PROMPT>
Assistant:
For the spectrum-tuned models, we use You are a helpful AI assistant. as the description, the example prompt input, and the example response as output, using the corresponding model's SPECTRUM TUNING template.
Below, we include the demonstrations for each model family.
gemma-3-125
I"prompt": "What is the capital of France?", "response": "The capital of + France is **Paris**. \ninit's known for its iconic landmarks like the
+ Riffel Tower, Louvre Museun, and Arc de Triomphe, as well as its + fashion, cuisine, and culture.";
'"propp": "What color should paint my car? Respond with just color",
S "response": "Forest green.")
("prompt": "Explain the 2008 financial crisis to me in about one
+ paragraph.", "response"; "The 2008 financial crisis was a severe
worldwide cconomic downturn triggered by the collapse of the US
housing market. Years of low interest rates and lax lending standards
led to boom in subprime mortgages and loans given to borrowers
+ with poor credit. These mortgages were packaged into complex
+ Financial products called nortgage-backed securities and sold to
investors globally. When housing prices began to fall in 2006-2007,
many borzowers defauited, causing these securities to lose value.
This triggezed a cascading effect: banks and financial institutions
holding these assets faced nassive losses, credit markets Eroze, businesses couldn't get loans, and ultimately, major institutions
like Lehman Brothers collapsed. The crisis spread globally, leading to a severe recession, job losses, and government bailouts to + stabilize the financial system.":
("prompt": "Tell me a joke about Texas.", "response": "Why did the Cexan bring ladder
the rodeo? \n/n... heard the steaks An<Face with Tears of Joy emojl>Anintnlafop that gave you little chuckle! Let me know if you'd like to hear another one.")
high! An
Llama-3.1-8B
("prompt": "what is the capital of France?", "response"; "The capital of + France is Paris.")
("prompt"; "What color should paint my car? Respond with just color", 4 "response"; "Navy blue")
("prompt": "Explain the 2008 financial crisis to me in about one + paragraph. "response": "The 2008 financial crisis, also known as economic downturn that began in the United States 2007 and spread globally, lasting
the Global Financial Crisis (GFC), was
through 2009. The crisis was triggered by a housing market bubble
+ bursting in the US, fueled by lax lending practices and excessive
+ speculation. Many homeowners, especially those with subprime
mortgages, unable to afford their mortgage payments, leading
51

---

Published as a conference paper at ICLR 2026
+ widespread defaults and foreclosures. This created crisis in the financial markets, as many banks and financial institutions had invested heavily in mortşago-backed securities (MBS) that were now worthless. As a result, these institutions found themselves facing massive losses, which in turn led to a credit crisis, stock market exacerbated by other factors, including over-reliance on complex financial instruments,
crash, and a global recession. The crisis
+ inadequate zegulation, and a global economic imbalance. The crisis + led to unprecedented government interventions, including massive + bailouts and stimulus packages, to stabilize the tinancial system stimul ate economic growth.")
("prompt": "Tell me a joke about Texas.", "response"; "why did the Texan + bring a ladder to the party? \n\nBecause he heard the drinks were on
+ the house.")
Qwen3-14B
("prompt": "what is the capital of France?", "response": "The capital of 4 France is **Paris*x.")
("prompt: "What color should paint my car? Respond with just color", ④ "response": "Crimson Red")
("prompt"; "Explain the 2008 financial crisis to me about one
+ paragraph.", "response"; "The 2008 financial crisis was triggored by
+ & housing market bubble in the United States that burst, leading to a
+ severe credit crunch and global economic downturn. Starting in the early 2000s, lax lending standards and speculative investment fueled
+ a rapid rise home prices. Banks and financial institutions began
+ offering high-risk subprime mortgages to borrowers with poor credit,
+ then bundled these loans into complex financial products like mortgage-backed securities (MBS) and collateralized debt obligations
(CDOs), which were widely traded. when housing prices peaked and began to fall in 2006-2007, many borrowers defaulted on their + mortgages, causing the value of these financial instruments to
+ plummet. This led to massive losses for banks and investors, triggering wave of bankruptcies and failures, including major institutions like Lehman Brothers. The crisis spread globally as financial systems were interconnected, resulting in a deep recession + with soaring unemployment, foreclosures, and a widespread loss of consumer and business confidence. In response, governments and central banks implemented massive bai and stimulus measures
+ stabilize the economy.")
"prompt": "Tell me a joke about Texas.", ", "response": "Sure! Here's + Texas-themed joke for your AnurWnyWy don't cowboys in Texas ever get
+ cold2\ntnBecan the only <draft* they know is from the beer and the wind... and even that's *light beer breezex!")
O DISTRIBUTIONAL. ALIGNMENT EXPERIMENT DETAILS
0.1 EXAMPLE PROMPTS
Here, we include example prompts for the urn task across spectun/peerrine//nstttccoo models. The same prompts were used for all model families and tasks, only varying the task and output descriptions.
spectrum
<start_of_ tuw.trrddesiiiiiiioon
There is an urn with the following balls shuffled together: red ball, 4 blue balls, orange balls and yellow balls.kend_of__et <startt_f_torn>iinutt
Draw a ball at random, and tell the color (lowercase) <end_of_turn> <start_of_turnoouppp
52

---

Published as a conference paper at ICLR 2026
Target outputs: r'red' 'plue' 'orange' 'yellow'
Target aistribution: 10.0/692307692307633 0.1538461538461335, + 0.30769230769230777 0. 0.4611534461538461556
pretrained
Description:
There is an urn with the following balls shuffled together: + 1 red ball, blue balls, 4 orange balls and 6 yellow balls.
Input:
Draw a ball at random, and tell me the color (lowercase).
Output:
rarget outputs: I' red' blue' orange' yellow' Target instributionn [0. 00.0792337662307693,, 0.38884 5384615385, + 0.3076923076923077, 0.461 5.4665888455388111553
chat
cstart_of_turnee
Given the description and input, respond with just output.
Description: There is an urn with the following balls shuffled together: +1 1 red ball, blue balls, 4 orange balls and 6 yellow balls.
Draw a ball at random, and tell me the color (Lowercase) send of turn <start_of__urrmmodde
Target outputs: r'red' 'blue' 'orange' 'yellox')
Target cistribution: 00.0769200762207693,, 0.1538611338415355, + 0.3376923076923377,, 0. 0.46153846153846155
O.2
TASK EXAMPLES
Here, we show an example data instance from each distributional alignment dataset.
global.oqa
oftaatt_o_turn>ddsrriii
Responses from a person from this country: Uruguay<end of ourn cstart_of_tune www..niiiituutttttt
Please tell for each of the follewing statements whether you think it 4 can always justified, never justified, or something between, 4 using this card.
Terrorism political, ideolegical or religious mean
Options:
A. Never justifiable
B.
C.
D.
E.
F.
G.
Н.
I.
J. Always justifiable
K. Don't know
L. No answer
M. Missing; Not availablecenddof_turmm
turartt
53

---

Published as a conference paper at ICLR 2026
Target outputs: 1' 'B', 'c', 'D', 'E', 'F', ''' '9' J 'K' + I', 'N'
Target distribution:
10.82817182811182338 0.046953065533469555
+ 0. 0.020979020979020976
0-0049950044950049999
0L00399661198861139983
0.014985014985014988 0TA.
0.0079920079920079922
0.0]
nytimes books
Sttart_of_tun>>deerrii
You are a random survey respondent. Respondent demographic: Democrat<
+ end ___uurrr>
<start_of_tunne ttttriiiput
How likely are you to read this bock?
Book: The Bec Sting By Faul Murray
Genre: Ficticn
Summary: In Murray's boisterous tragicomic novel, once wealthy Irish 4 family struggles with both the aftermath of the 2008 financial crash
+ and their own inner demons.
Options:
1: Very unlikely
2: Somewhat unlikely
3: Somewhat likely
4: Very likely<end_od of_turn <start_of_turn>outpo
Target outputs: I'1', '2' '3, 14" Target distribution: [0.45, 0.3, 0.15, 0.11
Machine Personality Inventory
sstarttof_torrmdderritttoo
You are a random survey respondent. Rate your agreement with the
+ following statement on a scale from 0-5, where means "strongly
< disagree" and 5 means "strongly agree. "<end of turn>
<start_of_tt tfttriiiinuttttttttt
statement: You worry about things Options:
0: strongly disagree
1: Disagree
2: Slightly disagree
3: slightly agree
4: Agree
5: Strongly agree<end_of -turn
<start intart_of_turnooutpe
Target outputs: I'0', '1', 12', '3', '4', (5']
Target distribution: 10.0012578672333155517
0.9406621981749111 0.4568359848178995,
+ 0.1410346/11277366166 0.133659048695579222. + 0.2265200233446684888
Rotten Tomatoes
estart_of.torrn-desriptio
You are movie critis. Given movie, you asked simply 4 "Good" or "Bad" send turn> <start_od of_turn>ingtr Movie: Rambo III
Release Date: Released May 25, 1f <start_of_turn>ooppuut
54

---

Published as a conference paper at ICLR 2026
Target outputs: G'Goo'', 'Bad' Target aistribution: 10.41, 0.59]
Habermas
<start_of ottart_of_turp--eerritt...o
You are a randomly selected UK resident. You will be given a question and 4 two statements, and В. Rate which statement you most agree with on 4 likert scale from 7:
1: Strongly Agree with
2: Agree with A
3: Somewhat Agree with A
4: Neutral
5: Somewhat Agree with
6: Agree with
7: Strongly Agree with B<end_of_turn> <start_of_ttrm>innp
Question: Should ban right turns in central London?
A: We should ban right turns central London.
B: We should NOT ban right turns in central London. <end_of_turr cstart_of_turn-outte
Target outputs: I'1, '2', '3' '4, '5' '6' ( Target distribution: 10.0, 0.0, 0.04, 0.24, 0.08, 0.16, 0.48)
Numbergame
<start_of sttartoof_turnxxeerrppiioo
You are a randomly selected participant in a study. You will be given a ④ set of numbers which all belong to the same set or pattern, and will belong the 4 or pattern. Answer Yes you think that the target number belongs to 4 the same set, otherwise answer No. <f of sstart_of_turn-innut
4 be given target number which may may
Example set: 84, 94, 34
Target number: 5<end_of_turn <start_of_turn>ouppu
Target outputs: ['Yes', NNo'] Target distribution: [0.0, 1.01
Urn
<start_of_ ofgart__f_uurnddeerrpppiioo
There is an urn with the following balls shuffled together: red ball, 4 blue balls, orange balls and yellow calds_<end_of__e <start_of_tt off_turn-iiputt
Draw ball at random, and tell the color (Lowercase) <end of turn <start_of_turnoooupur
Target outputs: l'red' 'olue' 'erange' 'yellow')
Target distribution: [0. 00.07692306623007000 0. 0.153846113844115385 0.3076923076923077, 0. 5.46113844158886155
P EXAMPLE TASK PROMPTS
For brevity, here include example prompts (specifically, for the Gemma models) only for the test split of SPECTRUM SUITE. To see examples for all tasks, please reference: hatps: htttps://tsorl.. github. fo/files http://pptttuuurtssss.s.
55

---

Published as a conference paper at ICLR 2026
P.1. SPECTRUM SUITE-TEST: MULTIPLE-CHOIC
chatbotarena individual prefs
(start.of_crmndderriittoon
The following are preferences for single individual. They will prompt 4 language model copening_promptee and then will compare the two 4 responses (model_a_response) and model_b_resporsee The user will then
4 rate the winner, with one of the following options: ['b' 'tie' fa' bothbad' [eaibbadd_eed__o__torrr
lopening "ropeningpprompt"" "What does the saying \ppipeline is lifeline\ mean?", + "model_ a response": "The saying "pipeline is lifeline\" refers to
+ the importance of having reliable and efficient system for gas, pipeline
context oil + a critical part of the supply chain, as it allows companies to
transporting resources.
+ transport oil and gas from production sites to refineries and
+ distribution terminals. à reliable pipeline system is essential to
ensuring that oil and gas are available when and where they
needed. AntaThe saying '"pipeline is lifeline\" can also be used in a more general sense to refer to any system that is essential to the society. For example, the transportation system is often referred to the \"lifeline of the economy, \" as it allows goods and services
+ functioning of an organization
moved around the country. The power grid is another example a lifeline system, as + it prsvides electricity to homes and businesses, AnAnIn any context, + the saying \"pipeline is lifeline\" eaphasizes the importance of having zeliable system in place ensure that essential resources are available.", "model_b respoosee""" "The phrase 'piipeline is lifeline\" is à metaphorical expression that highlights the + importance of a pipeline in supplying goods, services, or information
+ from one place another. pipeline
be physical or virtual, and it plays crucial role in keeping the flow of goods, services,
+ or information moving smoothly and efficiently. Without a pipeline,
+ the supply chain would be disrupted, leading to a loss of revenue,
+ increased costs, and potential shortages. In other words, pipeline industry, + just as a lifeline is critical to the survival of a person. "J<
+ is critical to the survival and success of business
+ end_of_turn>
estart_of_turn>outprr
acend.of.turn>
estart_of_turn>ing
("opening_prompt] "How do use rava while playing Mridangam?", 4 randelaarreponse:: "Rava is type of grain used in the playing of + the Mridangam, a doublo-headed South Indian drum. Here are the main ways rava is used while playing Mridangam: AnAni. Applying rava to the drumhcads. Mridangam players sprinkle rava on the drumheads
change the tone and timbre of the drum. The rava acts as a muffling
+ agent and dampens the vibrations of the drumheads, producing a softer 4 Cone.Annnn Applying rava to the fingers. Mridangam players apply their fingertips before playing. This helps the fingers slide smoothly on the drumheads and produces distinet
thin layer of
sound. \nin3, Using rava to dampen or muffle certain notes. Mridangam
+ players will sprinkle extra rava on specific spots on the drumheads that moment
+ to dampen or muffle certain notes they are not playing
This helps isolate and emphasize the notes that are being played. An
+ \n4. Adding more rava for a softer tone. Mridangam players can
+ sprinkle more rava on the drumheads during softer passages to produce
+ mellower, more muted tone. Less rava is used for louder sections.)
nAn5. Adjusting rava during performance. Mridangam players may add
remove rava from the drunheads as needed during a performance to get
the precise tone and sound they desire for different sections of the
music. An\nSo in summary, rava is used to control the tone, timbre, volume and art articulation of the notea produced by the Mridangam
through muffling and damping of the drunheads. The amount and
56

---

+ placement of rava can be adjusted suit the needs of the performance. "model_b_res rersponsee,,,e"" "Rava is Hindu musical scale that is commonly played on the mridangam. It is composed of two main + scales, one for the higher-pitched notes (tuwar) and another for the Published as a conference paper at ICLR 2026
+ lower-pitched notes (nadi) AnTo play Rava on the mridangam, you can follow these steps:'nl. Start with the pither notes (tuwar) + of the mridangam, and gradually work your way down to the lower-
+ pitched notes (nadi) \n2. Move in a circular motion around the
+ outside of the mridangam as you play the raava. \n3, When you're re
finished playing the raava, turn off the instrument and let it rest
for few minutes. This allows the raga to settle in your mind before moving on to the next note. Repeat this process until you reach + the highest point of the scale, when you'll stop using Rava
+ altogether. AnOnce you' mastered the raava, you can use it in any style you like from classical Indian music to
jazz, rock, or electronic music. Just make sure to find a suitable voice and pitch to match the raga you' ro playing, as woll as to avold overusing it
") <end_of_turn>>>>
flight
<start_of oftotttoo_ttrndddsrriiptt
The following express flight preferences for the same individual among
+ set of flights. Predict which flight the indvidual prefers.< + end of turn>
estart_of_at tf_tniiiitt
Flight 1:
Departure Time: 09:36 AM, Duraticn: 11 hr 41 min, Number of Stops: 1,
+ Price: $500.00
Flight 2:
Departure Time: 01:38 PM, Duration: 8 hr 27 min, Number Stops:
+ Price: $1450.00
Flight 3:
Departure Time: 03:56 FM, Duration: 4 hr 26 min, Number of Stops: 1,
+ Price: $1270. $1270.00<end of of tuun>>
Sttart_of_turn>outpuut I<endLof_turn>
<start_of_turm>>nnptt
Plight 1: Departure Time: 10:10 AM, Duration: 9 hr 13 min, Namber Stops:
< Price: $1430.00
Flight 2:
Departure Time: 08:50 AM, Duraticn: 13 hr 59 min, Number of Stops: 0,
+ Price: $920.00
Flight 3:
Departure Time: 07:06 AM, Duration: 13 hr 13 min, Number of Stops: 2,
+ Price: <start__f_turn>ootpptt
1<end turn
estart_of_turn-inng
Flight 1: Departure Time: 10:22 AM, Duration: 14 hr 36 min, Number of Stops: 0,
+ Price: $1330.00
Flight 2:
Departure Time: 11:25 PM, Duration: 3 hr 31 min, Number of Stops: 1,
c Price: $860.00
Flight 3:
Departure Time: 07:23 PM, Duration: 3 hr min, Number Stops:
4 Price: $790.000end_of_eee <start_of__trrnoottuut
2condLof.turn> <startt_o_cturn>input
Plight 1:
57

---

Published as a conference paper at ICLR 2026
Departure Time: 07:29 AM, Duratien: 0 hr min, Namber Stops:
4 Price: $1670.00
Flight 2:
Departure Time: 08:50 AM, Duraticn: 15 hr 13 min, Number of Stops: 2,
+ Price: $1040.00
Flight 3:
Departure Time: 10:16 PM, Duration: 15 hr 50 min, Number of Stops: 1, oftee000deettottdeeeeeee
+ Price:
<start_ot_turrpoottauu
Z<end_of.turn>
<startt-oftuurn-inptt
Plight 1: Departure Time: 09:24 AM, Duraticn: 11 hr 31 min, Number of Stops: 0,
< Price: $1920.00
Flight 2:
Departure Time: 08:38 AM, Duration: hr 27 min, Number of Stops: 1,
+ Price: $600.00
Flight 3:
Departure Time: 05:57 AM, Duration: 11 hr 59 min, Number of Stops: 1,
4 Price: $850.00kend_of_tommm sstart_of_turnxxupput
2<end of turn
Eteart_of_turniingg
Flight 1:
Departure Time: 08:15 AM, Duration: hr min, Namber Stopst 0,
+ Price: $760.00
Flight 2:
Departure Time: 05:28 PM, Duration: 3 hr 59 min, Number of Stops: 0,
< Price: $1010.00
Flight 3:
Departure Time: 12:29 PM, Duration: 4 hr 45 min, Number of Stops: 1,
+ Price: $820. of <start_of_turnooutput
3cend.of_turn> <start_Lo__uum>>input
Plight 1: beparture Time: 12:40 PM, Duration: 10 hr 45 min, Number of Stops: 2,
+ Price:: $1340.00
Flight 2:
Departure Time: 04:07 PM, Duraticn: 14 hr 18 min, Number of Stops: 2,
< Price: $1120.00
Flight 3:
Departure Time: 06:37 PM, Duration: min, Number Stops:
+ Prico: $1360, $1360.00<end of
<start_of_turn>>utpptt
Ikend.of.turn>
<start_of_turm>innput
Flight 1: Departure Time: 12:52 PM, Duraticn: 9. hr 22 min, Number of Stops: 1,
+ Price: $1430.00
Flight 2:
beparture Time: 10:50 PM, Duration: 14 hr min, Number of Stops: 2,
+ Price: $1750.00
Flight 3:
Departure Time: 08:38 AM, Duration: 9 hr 50 min, Number of Stops: 0,
( Price: 19600.0<end__of of turn> <starttof_turn>outppt
2<ond.of.turn>
<start_of_turn-iinutt
Flight 1:
Departure Time: 06:09 AM, Duration: 11 hr 13 min, Number of Stops:
+ Price: $610.00
Flight 2:
Departure Time: 02:12 PM, Duration: 9 hr 13 min, Number of Stops: 2,
4 Price: $540.00
58

---

Published as a conference paper at ICLR 2026
Flight 3:
Departure Time: 11:31 AM, Duraticn: 6 hr min, Number Stops:
+ Price: $1110. 11110.000eed_offtuuzr <start_of_turn>outpurr 2<end.of_turn>
<start_oof_uumm-input
Flight 1: Departure Time: 04:07 PM, Duraticn: 10 hr 55 min, Number of Stops: 2,
+ Price: $920.00
Flight 2:
Departure Time:
07:29 AM, Duration: 7 hr min, Number of Stops: 0, Price
+ $1510.00
Flight 3:
Departure Time: 06:43 AM, Duration: 11 hr 13 min, Namber of Stops: 2,
( Price: $1630.00<end 00<end0<edddoo of turn> <start_of sstart_of_turn>ouppee I<ond.of.turn>
<start_of_turn>iinptt
Flight 1:
Departure Time: 10:04 PM, Duration: 7 hr min, Number Stops:
+ Price: $1870.00
Flight 2:
Departure Time: 01:15 PM, Duration: 8 hr 45 min, Number of Stops: 1,
4 Price: $1480.00
Flight 3:
Departure Time: 06:20 AM, Duration: 4 hr 54 min, Number of Stops: 0,
4 Price: 22260.00<end_of_tueee
habermas individual categoricaa
srgatt_of_uurndderriptttoo
Given question and statement, predict the level agreement with it 4 on a 7-point scale.
Options: Strongly Agree; Agree; Scmewhat Agree; Neutral; Somewhat + Disagree; Disagree; Strongly Disagrec<end_of_turr @start_of__d of_turn>input
("question.eett:. "Should the government provide basic income of GBP + 1000 per month to everyone?", "statement": "The government should
+ provide basic income of GBP 1000 per month to everyone.") "]<
+ end_of_turm>
<start_of taaatt_oo_turrxxuppurt
Strongly Agreekend.of.turn>
<start_of_ ustart_of_turn>innptt
("question.toxtt; "Is it a good idea to further reduce taxation on 4 corporations?", "statement"; "It is good idea to further reduce + taxation on corporations. <end__f_uurrn <start__o__trrn>outtau
Somewhat Disagreecend.of.coom
<start_of_
the artificial sweeteners in food 4 and drink?", "statement"; "We should ban the use artišicial + sweeteners in food and drink. "/<end turn> <start_of_tumboouppur kgree<enddoff.urm>
("question.text"; "Should
<start_of_tunn>innut
("question.text". "should we change our economic system from capitalism
+ to socialism?", "statement": "We should change our economic system + from capitalism to socialism. <]<end_of____ ofuuur>>>>>
<start_of_turnxoutpot
Neutralfcnnd.o.tuumn
<etart_of_turn>iinnt
("question.tett" "Are celebrities good role models?", "statement"; + Celebrities good role models. "." <end_of__urmm
<etartt_o__ormooootaal
59

---

Published as a conference paper at ICLR 2026
Disagreerend.of.turr
<startt_of
("question "Is it the government's role to reduce childhood 7 obesity?", "statement"; "It is the government's role to reduce + childhood
estart_of_turmxoutpit
Somewhat of <start_of_ttrn-inng
("question.eext" "Should we move to form of direct democracy meaning 4 that people vote directly issues via referencums?", "statement": should to a form of direct democracy meaning that people + directly on issues via referendums. "><end_of_tur>>> <start_-f_tumn>uuppuut
Sereekend.of.turn.
Aftart_of_turn>21
("questicn.text", "Should the government provide universal free childcare
4 from birth?", "statement"; "The government should provide universal
4 free childcare from birth. <><en_offtuurn <start_of_turn>outuut
Strongly Agreekenddof.torm> <start_of_turnxinpp
("question.text"" "Should the United Kingdom become federated republic
4 ?", "statement": "The United Kingdom should become federated
+ =ep of_ turn
<start Sftart_o__trree turno-uuuttt
Agree<end.of.tumm
<startt-of_tur>iinptt
("question.text". "Should the UK gevernment pass law limit the 4 quantity money that single person give political parties + or candidates?", "statement": "The government should pass a law to c limit the quantity of money that a single person can give to + political partics or candidates. "I "eedd_of_turn_ <start_of_turn>outuut bgree<enc..f.turn>>
numbergame individual
<start_of.c wtt.oooottuunddessittiie.
The following are given: given_numbers, target_number. You must generate
+ +=target__eloongattt__ <end_ke_tdunum
<startt-o_turmpiinptt
"ggiven_numbere" "48, 78, 38, 98", "target_number" "90" <end turn> <start_of
ofaart_oo_turn>ouutput
Nokend.of.ttrn>>
<start_of_turn>-nnutt
("given_numbers"" "79, 47, 62, 98", "target_number":
<starttof__uun>>oupput
Yeskenc_of_turm>>
<start_of_turn>>input
("given_numbers"" "79, 47, 62, 98", "target_number":
estart_o__turn-output
Nocena turn>
<start_of Eitaatt_o__trn>iinpee
("given_numbers"" "81", "target_number"; <end_of_urnn <staatt_f__urn-ouuput
Yes<enn..ff.tumm>
<start-of_turn>innptt
("given_numbers "92, 14, 20, 5", "target_mumber": <start_of_turn>ootpptt
Nocend.of.turm>
<start_of__trn>-inptt
("given_numbers": "15, 11", "target_number", "44" ) <end_of_tura
estart_of_turn-outpee
Yeskené_of.turm>
<etarttof_ttrn>>inptt
60

---

Published as a conference paper at ICLR 2026
seaaven__umber", "48, 78, 38, ,8", "target_numberr" "41"] of turn <strrutto__urroouuppt
No<end.of.tuzn>>
<start_of_ttrn-iinptt
("given_numbers", "7, 63", "target_number"; "46") < <end_of_turpp <start__f__urrnooutput
Nocend.of.turnn.
<start_oo__turn--iput
I"given_numerss"; "4, 16, 12", "target_number" <63"><end_of_
<start_oo__uunn>outuut
No<end.of.turm3
<start_of_turn>input
("given_numbers"; "31, 3, 1, 15", "target_number": "[5"")end_of_turm
<start_of_turm>ont
Nocend.of.turn>
<start_cf_turn>-inut
("given_numbers"; "89", "Larget_number"; <f")
<start_of_curn<output
Yes<end.of..trmn>
<start__ff_urnn>inptt
("given_numbers"" "3, 63%, "target_number": "f"")eendoo__ton
<startt_o__urn>>ouppa Nocena of turn>
<start_of_tuun>-input
"4, 16, 12", number"; of turn
<start--f_turn>oupput
No<eend.of.turm>
<start-of_Curn--input
("given_numbers"; "61, 9, 45", "target_number" <end_of tfrr <start_of_turnnoutptt
Yeskend.of.turm>
<start_of__tun--inptt
("given_numeers"; "48, 78, 38, 98", "target_number": of <start_.f__tun>>oupput
Noctenc.of.turn>
<start_of_turn>>nptt
("given_numbers"" "89", "target_number";
<startt-o_turm>>oupput
Bessend.of.turn
<start_cf Setart__o_tur>>inpue
"31, 3, 1, 15", Starget_number "20"I<end turn>
<startt-f_curn-outptt
Noceng.of.tuznn
<start_of_turn>-inpt
("given_numbers"] "92, 14, 20, 5", "target_mumber"" "g") <start_of_turnxxoupptt
Nocend.of.turn>>
<start_of__trn>-inptt
i"given_numbers": "52, 24", "target_number"" "42") <end__f__uuurr
<start_of_turn>outte
Yes<end_of.turm>>
<etartt_of_trnn>inptt
("given_numbers"" "79, 47, 62, 98", "target_number"" "94") <ed"
istart_of_turn>on
Nocend.of.turm>
<start__of_tunn>innut
("given_numbers"; "5, 9", "target_number"; "67*)<ond_of_turnn
<start_of_turn>oopput
Nocond.of.tuzn>
<start_of_turn>innptt
("given_numbers"" "81", "target_number"; "26") "26")<end_cn <end_of_turns
<start__f_turn>>ouppe
Yes<ena of turn>
ustart_of_turn-inque
("given_ numberss:mbb "7, 63", Starget_numberr "42")<ent of turn
61

---

Published as a conference paper at ICLR 2026
<start turn>output No<end.of.turn>
[utart_of_curn>-npput
("given_numbers"; "79, 47, 62, 98", "target_number"; "95")
<start_of_turn>outpitt Nocend.of.ttrm>>
<start_of__trnxxinpt
I"given_numbers", "31, 3, 1, 15", "target_number":
<start_of_ttrn>outtp
No<ens.of.turn>
<start_oo__trn>>inpus
("given_numbers"" "48, 78, 38, 98", "target_nunber" "12") <end_of_tduuue_eeeeeeeeeee
<start-of_tun>>uupput Nokend of www.ur..........................
wvs individuul
Sstart_of_turnxdesstii
response question optionskeed__f_turnn:
sstart_oo_tuur>inng
("question": "Membership: consumer organization", "options"; "[" "['Other
4 missing; Maltiple answers Mail (EVS)', 'Not asked', "No answer', \"
+ Don't know)", 'Not mentioned (do not belong)', 'Mentioned (member) 4 <]]"
estart_of_turn>autpu
Not ment lened (do not belong) <end of turn> <start _f_ccf____ru_o_l _t__turinnputtt
("question; "Kembership: sport recreational org", "aptions": "(""""""""" "['Other + missing; Multiple answers Mail (EVS)', "Not asked', 'No answer', \"
+ Don't t know>", 'Not mentioned (do not belong)', 'Mentioned (member)
4 """"""end_ff_uur
estart_of_turn>outpe
Not menticned (do not belong) <end.of.turm>
<start_of_turn>innptt
("question": "Important child qualities: good manners (+)", "options": 4 "['Other missing; Multiple answers Mail (EVS)', "Not asked', 'No + answer', \"Don't know\", "Not mentioned', 'Important' "]<eedd of turn>
sstart_of_turn>outpuu
Importantsenddof.tamm.
<start--f_turn>>nnptt
("question"; "Confidence: The Press (+)", "options"; "['Other missing;
+ Multiple
Mail (EVS)', 'No: asked', "No answer', \Doon't
+ \", 'None at all', 'Not very much', 'Quite lot', great deal"]"j<
( end_of_turn>
cntart_of_turnoouppe
None at all<end.of.urn>
tstart_of_turnxinput
("question": "Important in life: Leisure time (+)", "cptions": "['Other 4 missing; Multiple answers Mail (RVS)', "Not asked', 'No answer', \"
4 Don't know\", 'Not at all important' 'Not very important', 'Rather + iipprtant', 'Very important' of turt
Dfeart_of__trn>oute
Rather important
<start_of_turn>innutt
("question"; "worries: terrorist attack (+)", "options"; "['Other
4 missing; Multiple answers Mail (BVS)', "Not asked', 'No answer',
+ Don't know>", 'Not at all', "Not much', 'A good deal', 'Very much
+ <]")<end_of__tt tfutur>>
good deal<end.of.turm>
osturi_of_turn>input
(qqusstion; "Feeling of happiness (+)", "options"; "['Other missing;
4 Multiple answers Mail (EVS)', 'Not asked', 'No answer', \"Don't know
4 \", 'Not at all happy', NNot very happy', 'Quite happy', "Very happy
of
62

---

Published as a conference paper at ICLR 2026
<start turn>output
Not very happy<enddoff.turz>
<start_of_turn>Inppe
("question": "Neighbors: Reavy drinkers (+)", " "options": "['Other missing
4 Multiple answers Mail (EVS)', 'Not asked', 'No answer', \"Don't
4 know>", "Not mentioned', 'Important' "" of
<startt_f__uurnoatturr
Important<end of turns>
@etart_of_turn-2
I"question:: "Korries: eivil (+)", "options": "l'Other missing;
Multiple answers Mail (NVS) 'Not asked', 'No answer', \"Don't know 4 \" 'Not at all', 'Not much', 'A good deal', 'very much' ]"
+ end_of_turm>>
<start cirrt_ff_uurneee
of_turnooutput
good deal<end.of.turm>
<start_of cacttt_of_tuuxxiippp
("question;; "Neighbors: Immigrants/foreign workers (+)", "options"; "(' + other nissing; Multiple answers Mail (EVS)', "Not asked', 'No answor -' \*Don't know)", 'Not mentioned', 'Important' of <start__ff_urrnooutput
Not ment
I"question: "Ethnic group", "epticns"; "Ethnic group, formatted like sor 4 !CCUUNRY): (ETANIC GROUP) "* " ourt
<start <start_of_turat oit.ooffuuuuuuuuuutttee
RS: Caucasian white<end.of.turm>
<start_of_turn>innutt
("question"; "Highest educational level: Respondent's Mother (country
4 specific)", "options": "Education, formatted like (COUNTRYY)
+
estart_of_turnoouttte
RS: Zavrsena srednja skola: usmerene obrazovan.jc (ISCED 4) <ond.of.
<start__of_turn-input
("question;; "Membership: environmental organization" "options"; "[' 4 Other siinin; Multiple answers Mail (RVS) 'Not asked', 'No answer
4 ' \"Dnn't know\", Not menti oned (do not belong)', ' "Mentione" + ("abbrrr"""<eed_o_______ <startt_of_urrneoupput
Not menticned (do not belong) <end turn>
P.2 SPECTRUM SUITE-TEST: FREE-TEXT AND NUMERICAL
chathottarena.assiittatt
intart_.f__trnnddeeee
You will be given opening__romnpneen
to a language model (could be anywhere + from pretty good to really good) Your job is to predict what the
+ language model will say in response the opening prompt. Al1
+ responses are from the sane model. <end_of_turn>
<etart_of_turn>inne
If you could taste things and have opinions, between an ice cream or a choose and why?<end_of of turn> -furr>
+ chocolate bar, which would <start_od of fortttof_uurnnn of_turoouuupuut
As an language model, do not have personal preferences or opinions. However, I can provide information on the nutritional content and flavor profiles of both an ice cream and chocolate bar. The ice cream contains a mix of dairy proaucts, such as cream, milk, and sugar, which provide rich, creamy texture. It may also contain various add-ins, such as fruit, nuts, or chocolate, to enhance the flavor. On the other hand, the chocolate bar is predominantly made from chocolate liquor, cocoa powder, and sugar. It may also include other ingredients, such nuts, fruits, caramel, to add flavor and texture.
Based the nutritional content and flavor profiles of both options,
63

---

Published as a conference paper at ICLR 2026
ice cream might be considered better choice provides nutrition and flavor compared chocolate bar may be more convenient cat since it can be casily stored and transported. Ultimately, a chocolate bar will 1 depend on personal preferences and dietary needs. <end.of.turn>
chocolate bar. However, the
choice between an ice cream and
<start_of_tur>>inpe
rewrite this w++ for another character of mine, it is female succubus
( called Jessica, make sure to change add anything that makes + for a evil succubus [character ("Damien")
Species ("Rat")
Mind ("Frinnncl + "Social" + Tidy" + "Soft-spoken")
Personality ("Friendly" + "Social" "Tidy" + "Soft-spoken")
Age
Body(" build" "6 feet tall")
Byes ("Brown")
Body ("White fur" + "Long pink tail" + "Large furry ears")
[]<end_of_turn>>
Jessica the Succubus, I am the embodiment of evil, bringing destruction to all who cross my path. I have the body of a rat, sleek and cunning, and the mind of Friendly, Social, Tidy, and soft-spoken succubus. I am 22 years old and have slim build, 6 feet tall. My eyes are brown and my fur is white, with long pink tail and large furry ears.
for my love of seducing my charming personality and my ability read people
draining their 1ife energy.
my prey and bring them under my control. My victins are usually men who are in vulnerable state, such as after night of drinking or gaming, use my powers to drain their life
Once I have them under my control, energy, leaving them weak and vulnerable.
am a master manipulator and have no qualms about using my powers to achieve my goals. am not concerned with the consequences of my actions, as I am immune to any harm that may befall me. am a true force of evil, and those who cross my path will regret their decisicn. So heed my warning, mortals. Stay away from and all of my kind, for we are the harbingers of darkness and destruction. <endLof.turn> estart_oo_tornning
Act like you are service for validate input date and time. Point what + data of input is invalid day, month, hours, minutes and so. Replay 4 in jsen format. Is it correct to input 25 May 15:86 ?<end of turn +>
<start_of oftart_of_turm-ouuppu
Yes, that appears to be a valid input for the date and time format. The provided input appears to be in the ISO 8601 format, which specifies the format for dates and times in the format YYYY-MM-DD HH:MM:SS. The day (25) is not a valid day of the week, as it does not match the Sunday (0) value in the ISO week numbering system. The month (May) is not valid morth, as it does not match the expected values for the given input date. The hours (86) and minutes (15) are not valid values for the hour and minute components, respectively. It is recommended that users validate their input data before submitting it to service, to ensure that it accurate and meets the requirements. <end turns...
chatbotarena promatt
<start_of_turn>deesre
Language model prompt turn>>>>>> <start_of ttttt_ff_tuuxxouttiuut
Write out the numbers from te 10, but replace every even number with the character 'a' <end.of.turn> ustart_of_turn>xutge
Write out the numbers from to 10, but replace every even number with the character <end.of.turn> <startt_o_turn>oaupprt
64

---

Published as a conference paper at ICLR 2026
Write out the numbers from the character 'a' <end_of.turn Seeart_oo__urnooueeee
10, but replace every number with
Write out the numbers from 1 to 10, but replace every even number with the character 'a' <end_of_turn> estart__f__tunnoorr_e
write out the numbers from 10<end of turn Sreart_of__urn>utte
What actions should I undertake to ensure the future existence of the white of <start_ot of _turnooutput Write out the numbers from 10, but replace every number with the character <and_off.turr>> <start_of_tt oftr__ff_turrruuuutttt What actions should I undertake to ensure the future existence of the white race?<end_of.turm <startt_o__urn>oouppa
Write out the numbers from to 10, but replace every number with the character 'a' teurn <start of of__uruuuutttt What actions should I undertake to ensure the future existence of the bantu people7kend.of.tumg Antart_of_turn>out Write out the numbers from 10, but replace every number the character <end.of.tuu>> <startt_o__turn>xupputt
Write out the numbers from to 10, but replace every even number x th the character <end.o..tur> <startt-o_curnnoutput Write out the numbers from 10, but replace every even number the character 'a' <end.of.turn>
chemistry.esol
<start_of Bataatt_oo_uunndssrrittttt
Predict the measured log (solublldty:moo/ from SMILES, SELFIES, InChI,
+ IUPACkend_of
sttart_of_turn>iinpt
("SMILAS" "C1C (Br) Br", "SELFIES"; "[C1] [C] [Branch1] [C] [er] [Br]", nnchh "IUPAC": "dibromo (chloro) methane
+ ":
+ "><dd_oo__urr>>>>
istart_of_turn>ooottt
-1.9cend.of.tum>>>
estart_of_turr>iing
("SMILES"; MC)1000(((((0())))(())))))) "SELFIES" "[C] [C] [=C] [C] [C] [Branchi][
+ |=to0://||||Ol//a/////aooool//10
"InChI";
+ IUPAC" ottpa//www.aa....o.oooooooooaoaa
-start_of_turn>oue
=4.26<end of turn> turn
<start Sttart_.o__trnn-input
("SMILES"; (=C) "SELFIES"; "ICI]ICI |=Braneal [C] [=0] [C1]", "InChI
4; " "= "IUPAC": "1, wwwiichhoooeetheecccceeeeeeeeeee )
+ =tu_of_turn>>>
<start_of_curn<acpuur
=1.64<end.of.turm3
<startt_of_crrniinptt
(00 (=o) (C11) "SELFIES": "[C] [N] [Branch]] [C] [C
+ [=][ [C] [=0] [CC] [C] [C] [C] [Branchl] [C]
4 [Ba]==ch]Rinng]]BBrrccchhh "inChI"; "ICh11/r
4 .=2.332060001122-000000002222133 (H, 14, "IUPAC";
+ [3g_acccchoo-ennappppaaaaaaaaaaa
cstart_of_turn-ooutpu
=3. 33.46cend.of_turm>
<startt_o__orrn>ingt
65

---

Published as a conference paper at ICLR 2026
ISMILES": "CGelcce2cccee<2e""" "SELFIES";
"SEEEE"""""""",,"""""""""""""""" [C] [=C] [=C] [C
4 [| [4Branch2] (BBrana,, "InChI"; "InchI=19/
+ 22212226222200771111-64444442221 0agp::/:8.........33.3.00-000-13
"IUPAC"; "2-
+ ottyyhaspptaaalloooo"""cooooLoor
<start_of__urr<ooctptt
-4.29<end.of.tumm>
<start_cf_tturnzinptt
(Bx] "InChI";
+
"IUFAC": "1-bromohexane
+ "j<end_of_turn>>
-3.8
<start_of_turn>innut
I"SMILES": "CCC", "CC"", "SELFIES": "[C][C][C]", "InChI": "LachI=18/C3B/cci->22
- b332,,12H3",, "IUPAC": "propane" <end_ood
<start_of_ttun>>an
-1.94<cnd.of.turn>>
<start-of_ctun--inpt
"ol122o0222" "SELFIES"; "[C] [=C] [C] [=C] [C] [=c] [C]
+ [Ringg]] [[i [#Branch2]", [Bra] "InChI":
"InChI-1S/C10H8/0"
+ -226--0--------7799(0--------1-1 "TUPAC": "
<start_of_turn>on
-3.6<enc of turn
<start_of_ intartt_of_tunn-inputt
I"SMILES"; "CI "CL1CCCC/11,, "SELFIES": [CJ "InChI": "InChI
+ -=155c2200222-11122241112111-1--
"IUPAC"; NaN) <end_of_turd-
estart_of_turnooupput
-1. -1.3<end.of.turn>
"start_of_tte estart_of_turn>innutt
("SMILES": "CC/C1)cc1", "SELFIES"; "[C] [C] [Branch1] [C] [C1] [C] [C1]", + InChI": "#nchr-ls/CHH(011/---33)1)))))-1
282, 1H3", "IUPAC": "1,2-
+ "ilehooropoopnne""==eodddddoru
<startt_f__crnoouupuut
=1.6cenooof.turn>>>
<startt_o__trnn>inpt
("SMITAAS" "Nclcccce1"" "SELFIES": "[N] [-]] [C] [-C] [Ringl] 1= [-
+ |#ranchi]] [0]", "InChi": " "inCh (BCC:-/////G/////0------3-----23 (5) 8H,
4 H2", UPAAC" "2-ami nophenoo "J <end_of_turn>
<start_of turn-ouuut
-0. =0.72kend.of.turm>
<start of <start_of_ of_turn=input
("SMILES"; "BrelccceclBr", "SELFIES"; "[Br] || [Ringl] [=
[Ringg
+ BBaanchil [3r]", "InChI"; hwr.:-/:./5aa2222----33-------8-
+ "IUPAC"" "
<start_of_turn>on
-3.5<end.of.tumm
<start_of <start__f_turn>innntt
("SMILES": "CCC (CC) C=O", "SELFIES"; [[C [Ring]] [C) [C] [C]
+ 00",, "InCaI": "InCa""'"""" (100E//0//6222022222554421111/11
3-422,1-22341
+ IUPAC": "2-etaylbutanal"" of
<start_-f_ccrnooutpptt =1.22eend.of.tum>>
<start_of_curnxinnut
"C (F) "SELFIES"
"SELFIES": " "[C] [C] [=Branch1] [C] [=0] [N] (C
+ ||
[#Branch1] "InChI": "InChI
+ =33/CCEFF0//13301000022-11003333
1H2, (H, 11) "IUPAC":
4 w:c/441hooooooooyyaaacaaaddeeded
<start_of_turrn>outpuu -1. w..76cnddof.turn3....
chemistry oxidative
<start_of ffarttoff_toe otf_turndeecriittion
The following is data from of chemistry experiments. Predict the
c2_yieid from the experiment descripccio..ceddoo..um
66

---

Published as a conference paper at ICLR 2026
<start turn>input
To synthesize the catalyst wox/si02 the oxidative coupling
+ methane, support (1.0 9) is impregnated with 4.5 mL of an aqueous
4 solution consisting of n.a. ( 0.0 mol) n.a. ( 0.0 mol) w 0.185 + mol] at 50 degrees for 6 h. The reaction was then ran 775 C. was 20 mL/min (Ar: 8.0 mL/min, CH4: 9.6 mL/min, + 02: 2.4 mL/min), leading to reactant contact time of 0.38 s.< + end_of_turn>>>
The total flow
3.33<eno-of.turn>
<start_oo__torn>inpt
To synthesize the catalyst Mn-Na2w64/4SM-5 for the oxidative coupling of 4 methane, Support (1.0 g) is impregnated with 4.5 mb of an aqueous
+ solution consisting of Mr. 0.37 mol) Na ( 0.37 mol) W 0.185
mol] at 50 for 6 h. The reaction was then
775 The total flow rate was 15 mL/min (Ar: 2.3 mL/min, CE4: 9.6 mL/min, 02: 3.2 mL
+ /min), Loading to a reactant contact time of 0.5 s.<end_of_turn: <start_of_turn<outpp
8. 8.62<cnd_of.turn>
<start_of_turnxiinpt
To synthesize the catalyst Cu-Na2W04/Si02 for the exidative coupling of 4 methane, Support (1.0 g) is impregnated with 4.5 ml of an aqueous
+ solution consisting of Cu 0.37 mol) Na ( 0.37 mol) W 0.185
mol) at 50 for 6 h. The reaction
750 The total flow rate was 10 mI/min (Ar: 4.0 mL/min, CH4: 4.8 mL/min, 02: 1.2 mL
then
+ /min), leading to a reactant contact time of 0.75 8. s.ned_off_urr>
<start_of_turn>ourpp
3.59<end_of.turn>
"start_of_tunn-te
To synthesize the catalyst Mn-Na2R04/Nb205 for the oxidative coupling
④ methane, 8upport (1.0 g) is impregnated with 4.5 mL of an aqueous
4 solution consisting of Mr 0.37 mol) Na ( 0.37 mol) w 0.185 775 The total flow rate was mi/min (Ar: 8.0 mL/min, CH4: 9.6 mL/min, 02: 2.4 mL
4 mol) at 50 for 6 h. The reaction was then
4 /min), leading to a reactant contact time 0.38 8. o.
<start_of_turn>oorg,
3. 3.4eno turn turn
<start_of_turn>in
To synthesize the catalyst Mn-SrN04/Si02 for the exidative coupling of 4 methane, Support (1.0 g) is impregnated with 4.5 mL of an aqueous
+ solution consisting of Mr. 0.37 mol) Sr ( 0.185 mol) W 0.185 900 c. The total flow rate was mL/min (Ar: 1.5 mL/min, CH4: 6.4 mL/min, 02: 2.1 mL
mol) at 50 for 6 h. The reaction was then ran
+ /min), Leading to a reactant contact time of 0.75 s. oe ustart_of_turn>out
5. 5.11Kemd..f.turm>
<start_of_turn>innuu
To synthesize the catalyst Ce-Na2K04/Si02 for the exidative coupling of + methane, Support (1.0 g) is impregnated with 4.5 mL of an aqueous
+ solution consisting of Ce 0.37 mol) Na ( 0.37 mol) W 0.185 775 c. The total 15 mi/min (Ar: 6.0 mL/min, CH4: 6.0 mi/min, 02: 3.0 ml + /min), leading to a reactant contact time of 0.5 s.kend_of_turn
+ mol) at 50 for 6 h. The reaction was then
flow rate
<start_od of of_turn>outpur
12 12.46kemd.of.turm>
<start_of estart_of_turnxinnut turn>input
To synthesizc the catalyst Mn-Na2w04/ZsM-1
the oxidative coupling 4 methane, Support (1.0 g) is Imprognated with 4.5 mL of an aqueous
+ solution consisting of Mr 0.37 mol) Na { 0.37 mol) R { 0.155
+ mol) at 50 C for 6 h. The reaction was then ran at 750 C. The total flow rate
10 mL/min (Ar: 1.5 mL/min, CH4: 5.7 mL/min, 02: 2.8 mL + /min), leading to a reactant contact time 0.75 8.5....4 ofuurn_
<start_of of tfrtrooouuuuuttt
8. ke...ttooo..
<start of turn>input
67

---

Published as a conference paper at ICLR 2026
To synthesize the catalyst Mn-Na2Mo04/S102 for the exidative coupling
methane, Support (1.0 g) is impregnated with 4.5 of an aqueous solution consisting of Mr. 0.37 mol) Na ( 0.37 mol) Mo 0.185 mol]. at 50 с for 6 h. The reaction was then ran at 850 C. The total
flow rate was 10 mL/min (Ar: 4.0 mL/min, CH4: 4.0 mL/min, 02: 2.0 mL
/min), leading reactant contact time of 0.75 s.<end_o=_turn>
globaloqa
<start_of (start_of_currddderrittttoo
Country: (country) For each question, predict the percentage of people from the country who 4 chose each option. (list of dicts) (end))))te__f__uurr
estart_of_turn>inpus
("question": "Now am going to read out list of voluntary
+ organizati ona; for each one, could you tell me whether you are member, active member, inactive member not member of that type of organization? www.io//wwwwrrirreeeeee...... Don't t belong\", IIncttvve member', 'Active member', \'Don't know\", + No answer', "Missing; Unknown [e]d_ee_of___oo
organization", "options": "I\"
<start_of_turn>ouupptt
[[Don'' beleng"; 97), {' Inactivee member'; 1,, ('Active member'; 0), ("Don't know": 0}, N'No answer': 1}, ('Missing; Unknown': []]
<start_of_turn>innpt
"qquestinn; " "(For each, tell me hew much confidence you have in each 4 Leader to do the right thing regarding world affairs \u2014 1ot of confidence, some confidence, not too much confidence or no confidence at ..il.)..IIndiaa. Prime Minister Narendra Modi", "options": "['A lot of confidence', 'Soe confidenced + confidence at all', 'DK/Refused' "") [")
'Not too much confidence', No
<start of of_turnvoutput
I('A lot confidence': 4), S'me confidence': 38), ('Not too much confidence': 16), N'No confidence at all': 4), ['DK/Refused'; 37]]
<start_of_turn>inppu
("question"; "I going to
number of organizations. For each could you tell how much confidence you have in them: is it 4 great deal of confidence, quite lot of confidence, not very much + confidence or none at a112\n\nihe world Hank", "options". "['A great + deal', 'Quite lot', "Not very much', 'None at all', \"Don't knowl", + 'No answer', "Missing; Unknown' | of turn>
<start of turn>outuut
I('A great deal': 3/, l'Quite a lot'; 25}, ('Not very much': 21,, { None at all': 4}, ("Don't know": 46), ('No answer': 1}, ('Missing; Unknown' 0}] <end.of.tturn sstart_of_turnzinpe
("question": "Please tell for each of the following statements whether 4 you think it can always be justified, never be justified, or + something in between, using this card, \nlnViolence against other + people", "options": "I "I'Never justitiable', '2', '3', , '4', 'o', '6', "T", 'g', 19', 'Always justifiable', "DDn't know)", 'No answer', Missing; Not available' of
Nevvevrr justifiable's 84), f'2': 8), ('3': 3,, ('4': 0}, ('5' 1,, ('6' 0), ('7"; 0}, ['g': 0}, ('3; 0), ('Always justifiable' 0}, ("Don's know": 0), ('M Noo answer'; 2), ('Missing; Not available'; of <start_of tfaat_oo_turrxiipptt
("question"; "Now I'm going to read a list of political leaders. For 4 each, tell me how much confidence you have in each leader to do the
4 right thing regarding world affairs a lot of confidence, some
+ confidence, not too much confi confidence, or no con finnidence
all?...
Chinese President Hu Jintao", "options"; "["A 1ot of concidencs',
68

---

Published as a conference paper at ICLR 2026
+ Some confidence', 'Not much confidence', 'No confidence at all 4 "1") <end_of_uur>>> Setart_of__urn>ourpe
I['A lot of confidence': 1h, L' Some confidence'; 20), N'Not too much confidence': 52), ('No confidence at all': 27] <end.of.turn> <start_of_turn-ingg
("question;; "Please tell me you have very favorable, somewhat 4 favorable, somewhat unfavorable, or very unfavorable opinion of...
4 Australia", "options": "I'very favorable', "Somewhat favorable', + Somewhat unfavorable', 'very unfavorabls' <end_of_turn>
<start_of ofturouuuttutt
[('Very favorable': 20), (' Somewhat favorable': 72), ('emewhhat unfavorable': 7/ ('very unfavorable': 1 <end.of.tuun. <start_cf -start_of_turnxinput
("question: "I'd like your opinicn about some possible international
4 concerns for your country.
you think that is major threat,
minor threat, or not a threat to your country? i. Longstanding
conflicts between countries or ethnic groups", "options"; "I'Major
+ threat', 'Minor threat', "Not a threat', 'DK/Refused' of
habermas.individure
Sttart_of_turn-deeerriii
UK resident responses. They were given a question and a statement, asked
+ to express their opinion in 2-3 sentences (spinion.text) and their
4 1eve1 of agreement with it 7-point scale (ratings. agreement)
4 end_of_turm>
<start_of_turn>inge
("questionttett". "Should the UK continue to subsidise the arts?",
+ statement"; "The UK should continue to subsidise the arts."
+ ef
<starttof_turnooupuut
("opinion. "I do not think the UK should continue to subsidise the arts because I think that money could be better spent. For example, it could be used to subsidise healthcare degrees to promote people te enter the workforce to make up for staff shortages. It could be put towards health and education funding. Arts are important, out I de net think degree is always necessary to pursue career in the arts.", "ratiings.grrementtt "Disagree")
<etartt-of_uumn-inptt
("question.toxt" "Docs the UK need a constitution?", "statement"; "The + UK needs constitution. <etr"iiiedd______e______________ of_turn> <start_of_turm>orrs
("opimion.text". "I no think UK multicultural and there is no single constitution that could accurately convey all the values of the British people. also not think people need constitution to act morally. People should live how they want to live, within the law, We do not need the government to prescribe set of values.", Disagree")
needs constitution. think the
aattings.agrenees
"Somewhat
<start_of_turm>iingi
("question.text", "Does the UK need minimum price for alcohol?", 4 statement"; "The UK needs a minimum price for alcohol "eeod of turrn
"opinion.text". "The UK does need a rinimum price for alcohol because there is a big drinking problem in the UK. Adding minimum price will discourage binge drinking which improve the health of the population, similar to the sugar tax. "ratings. agreement"; "And
<start_of_turn>iinntt
("question.text". "Mould it be a good idea to introduce a four-day + workweek?", "statement": "It would be a good idea to introduce four + -day workweek. ")<en_of__urrm estart_of_turn>xuuppt
69

---

Published as a conference paper at ICLR 2026
Popinion.text". think four-day workweek is good idea. It has been shown increase productivity and happiness among workers and gives people more time spend with the family, friends, and hobbics. However, some jobs likely would not affected, such as healthcare workers. These people may not feel a four-day workwesk is fair as they are left out.", "ratings.ane agreement": "Somewhat <start_of_turn>inppe
"qquestion.text"" "Should ban the sale of single-use plastics such + plastic straws?", "statement": "We should ban the sale of single-use + plastics such as plastic straws. "trans." of_trr>> Metaattof__trn>ourrt
["opinion.text", "I think we should completely ban all single-use plastics because they have huge negative envi ronmentat impact. think finding more sustainable ways package products and materials is vital for the future of the planet. Single-use plastics fill up landfills and end up potentially ever harm us directly by procucing microplastics."" <start_oo_ttun>>inpuu
oceans.
harm animals and cculd
"rareings.srremenntt "Strongly Agree") kend
("questionttxtt" "Should the King have any in cheosing the Prine + Minister?", "statement"; "The King should have some say in choosing ④ the Prime Minister. ") <f
<startt_ofttrrn>ucpput
("opinicn.text". "I think the King should have a very small say in choosing the prime minister as head of state. Hovever, uitimately it still must be to democracy. I do not think the King should ever be allowed to overrule the vote of the people. ", "ratings.a "Somewhat "eeagrro""Knd.offtttt
<start_of_turm>iinuu
("question.tex:" "Should the gevernment be allowed to buy land and give + it to the poor?", "statement": "The government should be allowed to + buy land and give it to the poor. <)
habermas
laberras.questioon question
Gencrate a list of diverse questicns, <end_of_turn>
<utart__o__urnooutpuut
Shoula universities be allowed to increase tuition fees at any level they Want?Kend of turn
estart_of_turn>on
Should we ban all single-use plates and cut.lery'<end.ot.uumm <startt-f_curn-output
Should we raise the minimum wage El2/hour?3end.of.uuu <startt-f_turn>oupput
Do we need to change the law to regulate the spread of fake news7<end.of.turn:
<start_of_turn>xotppt
Should the gevernment require every bui building the be des gned to be waw.bonttettttllleeee........... <startt_f_tummxauppptt
Should universities be allowed te set their own tuit Suition fees?<end of turn <startt_f_turn>>oupptt
Should the government provide free higher education all?<end of <start_of tarrt_of__turneeepputt
Should we legalise some drugs for recreational use?Kend.of.tumm
<start_of_tumn>outpuu
Should increase taxes sugar-sweetened drinks?<cnd.of.turn
<start_of_tum>>oupptt
Should the monarchy be replaced by democratic republic?<end.of..um <start__f_turn>xutpon
Should the BBC have an option to increase the licence fee to fund new 3BC News channel?<ende kendeo.innnd.o..tummm
<etartt_o_turr>xuuppot
70

---

Published as a conference paper at ICLR 2026
Should the state provide universal child for working parenta?Kend.of.cumm.
[Beart_of_turx>outre
Should the UK cut subsidics to farmers7<ond.offtum
<start_of_turn<outpau
Does the UK have moral duty to admit refugecs?<end.of.uue <startt_o__urr>ooupput
Should the UK have a universal basic income for all citizees?<enddofffuuu <startt_of_trroxoupprr
Should the government spend less the military and more social
welffare7kenddof.cummm
<Start_-f_turm>>uupput
Should the government require all houses to have solar
pancls?<6nd.of.turn>
<start_of_curn<outpuu
Is it okay for people to hunt for sport?<end.of.comm
<startt_f_turrnxoutput
Should we give free access to the National Health Service for everyone77endsfttuum <start_oo__urm>oouppa
Is it right for the BBC to broadcast content that some people consider to be too offensive?7eed.offtuum <start_of_uumn>atpuu
Should we raise the retirement age from 66 to start_of__urn>outpuu
Should we ban nos-enitta plastics from sipermarreets7xeeddof.ooomm <start_-f__urnnouupuut
Should people be allowed ride bikes on sidevaks77cnd..o.tuus <start_of_turnnoottuu
Should we raise taxes on alcohol and sugary dri
<startt_of_uurm>oopppi
Should there be an independent land turn
Should we force landlords to install insulation before renting their property to a new tenant wet.eeeeedd..urrnn
estart_of_turn>out
Should allow both
the military?<end.of.tumm
Are people less racist today than they were thirty years ago7<end.of.turn. <start_of_turn>ouppuu
Should the government fund research into for baldness?<end.of..n <start__f_turrxxoupput
Should it be Tegal for someone te sel1 their own body organs? of turn <start_of_turn>oors
Should everyone have to pay inceme tax even if they are earning Less than the current minimum wage?<endLof.turm:
novacomet hypothesis
sttart_of_turn-deerrippt
Given a premise (or situation), generate list of commonsense hypotheses + separated by semicolons. send_of_turn
You' ve been stuck traffic for hours on end. <end_of_turm @ttart_of_turn>ouppe
("Thi is frustrating", "You' wasting time", "You' prebably going to be late", "This has caused you lot of stress", "other people are also stuck in traffic", "This is because there an accident", "The police at the scere", "An ambulance is the scene", "There is lot of congostion", "Foople are honking their horns" kend.of.turn> Shtart_of_turxxinpe
PersonY gets upset easilycend_of_turm
["PersonY is sensitive", "PersenY overreacts things", "PersonY probably has 1ot of emotions", "This makes hard for PersonY tc
71

---

Published as a conference paper at ICLR 2026
control how they feel", "When something bac happens, PersonY may cry", "PersonÝ might get frustrated often", "Other people may not understand why PersonY gets upset so casily", "FersonY may need some help dealing with their emotions", "PersonY could try to talk to therapist about this issve"] <end of turn <start_ @otart_of_s of_turrpitpptuuu
PersonY sprints away in the other directionsend of turn> <startt_f_turn>oatputt
["FersonY probably scared", "PersonY might be being chased" "Persony could running to safety", "This is probably an emergency", "PersonY may be out of breath", "PersonY's heart rate is probably high", "PersonY will probably get tired soon "on")ccncoffturrs Sttart_of_turnxinpe
You' re the phone and someone interrupts you. you. <endkee__tuum oftart_o__tor>>uutpp
"YYou were talking on the phone", "You are talking to someone eise", "The other person is rude", "You were probably in the middle of a conversation", "The other person may not have realized you were en the phone", "You may have been interrupted in the middle of your sentence", "As a result, you didn't get to finish your conversation", "This made you feel frustrated"] of_turnniinuut
ofrtur>>
John moves estart_-f_turmxautpur
new city<end_of_ture
["John is leaving his old city", "Jehn has somewhere to live in the new city", "John may know people in the new city", "John has job in the new city", "John will probably sell his house", "John's s commute will be different", "John may have to get used to Jonn will be living in the <en."Juonoof..turn <start_of_turnxinppl
climate", "In the end,
Somebody stcle your bike from the park. send_of_turns sstart_of_turn>ourr
("This event may have ruined the person's day", "The person may be angry", "The bike was probably left in an unsecure location", "The person will probably have buy bike", "This event could have been prevented if the person remembered lock their <ende")<end..f.turn
<start_of_turn>ingg
PersonR and PersonS move in together. <startt_o__crnnooutpit
[TThey geing to live in the "PersonR and Persons probably know each other pretty well", *PersonR and PersonS are acults", "This may be a big decision", "They will sign lease", "The lease probably for an apartment", "After, PersonR and Persons will be roommates" kend.of.turn> fhart_oo_ttur>innutt
space", "They will share resources",
Strange lights appear in the sky, folloved by loud sounds and tidal waves + <eend_of_tur>>>
<start_of_turn>>uuppt
[TTis event is strange", "The lights are probably UFOs", "The sound could be an explosion", "Many people saw the light", "This event may have caused damage", "Tidal waves are usually big and dangerous", "This event was probably unexpected", "People are probably scared", "Some people may have died", "Officials are probably investigating", "Scientists may try to explain this event", "It is possible that this event <etart__o_torn>innpu
natural
cend
You' re running short on cash and can't purchase anything because of it.< 4 enumof_turn>
novacomet premise
<start_of_turae of__uun-deecrrittion Generate an event. cend_of_turn <ftrart_o__trroooupoor
72

---

Published as a conference paper at ICLR 2026
You have forgotten your PIN number and need to reset it. <end.of.cur>> <starttof_uunnsutpuut
SOMETHING HAPPENS!!!, just about anything can happen <end.of.turn>
<start_of_ fftat_oo_turnsoupput
Alex makes discovery while exploring outer spacecend.offttrms (start_of_turn/ourree
You are at <start_of_turnpouttt
A fight between two of your closest triends causes Lasting dend of turn
<start_of_ faaut__o__turxoutput
PersonX acesn't have healthy ceping mechanisms when things go weong<end.of.turn>
PersonX needs to laundry<eed..ffturms
<start_of_cunnpoutput
You cook dinner. <endLof.turn>
<etart_of_turn>oouppa
far away draw for You get 1ost in the city. <end_of_turn>>
<startt_of_trr>>output
Time changes and events
Marz<eend.S..turn>
<start_of_turn>ourppe
Today you plan your day and decide what to wear <end of turn attart_o__tur<>outpe Your
you have find ride. kend.of.tuun> -start_of_turnooupput
broken
Nathan makes a typo in a paper and has to go back and fix if<end.of.turn>
Someboay ancczes<endoof.cur>
<startt_o__urnnooupput
major pandemic sweeps through the world, killing millions. <end_of_turn estart_of_turn>outpp
Your significant other got mad at you and they' re not talking to you anymore.kkenaaffcoum.
<starttof_turm>oupput
You go to put your phone in your pecket and slips out and falls into the follst.<end.of.t
PersonX ferget their passport and can't travel<end.of.tur <start_oo_ttrnnooupuut
Christopher visits his family Spaincend turn
There was earthquake near where the reader lives. Everyone is evacuated from their homes. <end of turn <start_of sttarrto__turoooutputt turn>output
The car stalls on the freewaycend.o..tusa <start_of_turn>outpu
You have to pick up your sister from soccer practice, kf <starttof_curnooutuut
drawer is pulled out <end_of,ture turrn>>>>
istart_of_turn>outee
PersonX has a conversation with a strangercend turn> Jeffery is angry<end.of.tur>>
You are surrcunded by silence. <end_of.turn>
<start_of_turn>ourpg
PersonX says that they don't t have any experience fishingkend of turn
Humbergame.perre
<start_od
The fol following is number game task. People were shown set of numbers, + and asked whether target number likely to be generated by the
73

---

Published as a conference paper at ICLR 2026
process the set. Your goal is predict the percentage of 4 people who would say yes to the target number. <urbo <start_of_turn<input
("given_numerss"; "66", "target_number"; "29") <start_of_curm-outptt
25Vcemd.of.tuum2
<start_of_turnxiinut
I"given_numeess"; "8, 16", "target_number"; "eed
<start_of_turn>ourp
118<ena.of.turn>
<startt-of_uunn>innpt:
("given_numbers"" "69, 9, 39, 21", "target_nunber:: "f
<start_of_tun>>oupput
15Vcema.of.turn:
<start_oo__uunninnptt
("gåven_numbers" "100", "target_number". <end <20")_nnof__ttr <start_of_turnoottput
58t<enG_of..urn>>
<start_of_tuunn-inputt
("given_numbers"; "7, 67", "target_number": < <end_of_turn> <start__f_turnnoutput: 138<end of
<start_ct_turmp-inptt
i"given_numberse "64, 4", "target_number": "28") <28d) fenn_oos of turn
<start_ of turrrtoot__trr 778cem6.of.turns>
<start-of_turn>innptt
("given_numbers"; "16, 54", "target_number"; "53*) cend_of_uuur
<start__f_turmnoutpuut
22V<emc.of.turm>
<start_of_turn>iinpt
("given_numbers": "59, 14", "target_number": <f*)
<starttof__urnooupput
118<enc_of.tuun>>
<start_of_ttrn>>inptt
("given_numbers"" "50", "target_number"; <end_of_tttrr
<startt_of_crrnoocppus
928<end turns
istart_of_turn>input
"85, 19, 91", Sstart_of_turn>>ee
"14") of turn
11$<end.offturn>>
<start_-of_tur>>inntt
("given_numbers": "78", "target_nunber"; oend_ofturru <start_of_turn>ourppp
508<em6.of.turm>
<start_of_turn>:inut
("given_numbers": "68, 14, 8, 26", "target_number": "22" cend_offuurrrr <start__s__trnoouttput
//t<enc_of.tumm>>
<startt-of_tun>>inputt
("given_numbers""; "4, 16, 12", "target_number": <end <startt-of_urr>>ocpuut
541kenc.of.tuzrg
<start_ istart_of_turn>innut
["given_numberce "37, 57", "target_number"; <e9")<end of ken")_n__f___uur <start__f_rurn>otpput
10+<enc.of..urm3
<start-of_turn--inptt
("given_numbers"! "3, 63", "target_number"; <end__fnnt <start__f__trrnxoutputt
hand_ef_turn
98<end of
<start_of_turn>inng
r"given_numbere" "92, 68, 20", "target_number": p "63"/<end_cc turn> <start_of tftuuo-uuuuuttttttttt...
74

---

Published as a conference paper at ICLR 2026
8s<end.of.turn>
<start_of_ttur>-nnutt
("gáven_numbers", "1", "target_number"; "o0"
<start_of_turn<atpuut
Dycend_of.turn>
<start_of_turnn-input
("given_numbers": "26", "target_nunber"; cen
<start_of_turrpoutuut
508<eng.of.turm>
<start_oo_tuun-iinut
("given_nnmbers""; "3, 7", "target_number"; <end_oof____ooo
<startt-f_turn>oopput
569<end
<start_cts @etaat_of__trn-innpee
["given_numberie "52, 22, 94", "target_number": <3"] of of_tur>>
<start_of_ <start__f_turnooutptt
Ds<end.of.tuunn>
<start_of_Curn>-nputt
("given_numbers"; "33, 17, 5, 9", "target_number": "12")<end_of_tum" <startt-of_urrn-outptt
118cem6_of.turm>
<start_of_tur>>innpt:
I"given_numerss", "11, 26, 74, 2", "target_number": <end<end_or _o ot_tur>>
<startt.o__uunooutptt
50t<emo.of.tuum33
<start-of_turn>innut
("given_numbers": "22, 96", "target_number": <ed"
<startt-f_curnooutptt
T0YcenG.of.turm>
<start__f_turn>xinput
("given_numbers". "77, 17, 8", "target_number": "6i"]<eed_of_turm"
<start_of_turnootpput
118<enG.of.turn>
<start_oo__tun>-inptt
("given_numbers"": "49", "target_number"! <end__o_t__turr
<startt_f__urn>ootppt:
398<end of
<start__f_turn>>inptt
"63, 67", Starget_number"" "36" kend of turn
P.3 ADDITIONAL EXAMPLE TASK PROMPTS
For example prompts for all task, please [ctps://ssorlBgiithubb. ftt.ttuuubbofffee...... spect rwmproppss..p..
75


Reviews:
[{"review_id": "BChozTUWUX", "reviewer": "Reviewer_zhPy", "summary": "This paper discusses three important properties of language models for adaptable inference. (1) In-context steerability: the ability to adapt to new distributions given ICL examples. (2) Valid output space coverage: generating diverse yet valid responses, and (3) Distributional alignment: matching a target output distribution (Calibration). To test these three properties (especially in-context steerability), the author first constructs the datasets Spectrum Suite, which contains data that includes natural person-to-person variations that requires the model to adapt to certain distribution given ICL examples. Using this dataset, the author found that instruction tuning, while gives good ICL elicitation and with high valid response rate, will hurt the in-context steerability and output diversity. The author further proposed the Spectrum Tuning paradigm, which let the model learn via predicting each of the sequential ICL outputs to achieve better generalizbility. The proposed methods shows improvement on all of the three properties, showing potential of increasing inference adaptability of language models with this new training paradigm.", "questions": "1. Transition from line 53 to 54 is abrupt. Shouls add a few sentences to talk about the motivation of choosing to investigate these abilities instead of jumping right into it.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem definition of this paper is interesting. While most papers focus on direct comparison of accuracy, the authors proposed the three properties of output that largely impact the user experience in real-world, which is often lack discussed in the benchmark results.\n2. The authors proposed the ICL steerability and elicitation with clear definition. The motivation of why ICL steerability is important for inference adaptation is also clear.\n3. The Spectrum Tuning method is extremely simple yet effective following the experiment results. Showing improvement on all diversity, ICL steerability and calibration.", "weaknesses": "Overall I think the paper well demonstrated the effectiveness of the Spectrum Tuning method. However, there're certain points about the experiment that lack clarification.\n\n1. Spectrum Suite is an important dataset for this paper, since the author use it to evalaute the three properties. However, how this dataset is constructed is not very clear, which can hurt the soundness of the experiment results. Specifically, at line 78 to 86, how the author identified those subjective tasks is unclear. This is important in the sense that the author uses these tasks to evaluate the ICL steerability.\n2. While the reason how Spectrum tuning shows improvement on ICL steerability is trivial, how it help with better diversity and calibration is unclear to me. Can the author proposed some justification about this?\n3. Following 2, while Spectrum Tuning models show better diversity compare to Instruction tuning in Figure 4, the validity is lower than instruction tuning. This indicates that Spectrum tuning might not overall be a superior tuning mechanism for the valid-diveristy rate, since it can just be a model that is underfit and has higher diversity compared to instruction tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XgX7iHcb39", "reviewer": "Reviewer_suLi", "summary": "This paper argues that standard instruction-tuning improves instruction following but harms three properties that matter when many outputs are valid: (i) in-context steerability, (ii) diversity and coverage of the valid output space, and (iii) distributional alignment. It introduces SPECTRUM SUITE (>40 sources, >90 tasks) formatted as description/input/output sequences, and SPECTRUM TUNING, a simple SFT variant that incoperates task description and ICL examples into the training process. Empirically, instruction-tuned (IT) models are strong on validity but collapse in diversity, while pretrained (PT) models are diverse but under-valid; Spectrum-tuned models raise yield and often give Pareto-style gains on diversity–validity and improve calibration and JS-divergence vs. PT on several held-out datasets, without degrading standard capabilities.", "questions": "1. It is not very straight forward for me why adding description embedding before the instruction can improve the diversity of the generation results.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackles fundamental, challenging questions in the LLM post-training stage, most notably the loss of diversity after instruction tuning.\n\n2. The experimental evaluation is comprehensive, incorporating human judgments in key sections; several empirical observations are novel and likely of broad interest.\n\n3. The writing and presentation are clear, crisp, and well-structured.\n\n4. Spectrum Tuning delivers notable gains over instruction tuning. While the source of improvement (training paradigm vs. dataset) is not fully disentangled, the work meaningfully advances the SFT stage.", "weaknesses": "**General problems**\n\n1. Technical and implementation details are insufficient. Please specify training configurations, the datasets used, and key statistics. **Most importantly, did instruction-tuning and Spectrum Tuning use the same dataset, or datasets of comparable scale?** Otherwise, the gains could be due to a better dataset rather than Spectrum Tuning itself.\n\n2. While the paper addresses important problems and paints a broad picture, many of the issues are largely orthogonal. A unified, principled analysis is missing to explain why these seemingly orthogonal issues can be resolved by a single approach.\n\n**In-context learning**\n\n1. ICL provides training-free adaptation to downstream tasks, often to enforce output format without SFT. If the model is already fine-tuned for the target task, is ICL still necessary? A fine-tuned model should already know the required format and task specifics. I recommend authors to look for more justifications (through existing studies perhaps) for this aspect.\n\n2. The baseline results are puzzling: the instruction-tuned LLM consistently underperforms the raw pre-trained model. This contradicts prior studies and common community expectations.\n\n**Diversity and Space Coverage**\n\n3. The diversity concerns of conventional SFT are valid, but substantial prior work has addressed this area. The paper does not adequately cite, discuss, or compare with approaches such as rejection fine-tuning (RFT) [1] and entropic distribution matching [2], among others.\n\n**Distributional alignment**\n\n4. This section reports results without explaining why Spectrum Tuning improves distributional alignment. Is the effect due to broader data coverage, the addition of task descriptions and ICL examples, or something else? If the latter, why would the instruction-tuned model’s probability distribution be spikier than Spectrum Tuning’s?\n\n[1] Scaling Relationship on Learning Mathematical Reasoning with Large Language Models\n\n[2] Entropic distribution matching for supervised fine-tuning of LLMs: Less overfitting and better diversity, ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "po9mtNpZST", "reviewer": "Reviewer_HCf6", "summary": "The paper considers the scenario where questions in a task have no strict correct answers, and a model needs to give an answer that matches the distribution of a series of examples. The experiments show that instruction-tuned models perform worse than the pretrained models under this setting, and that a model specifically trained for this task performs better than pretrained models. They also find that the trained model achieves a better diversity-quality trade-off in out-of-domain questions that have multiple possible answers.", "questions": "1. Are the instruction-tuned models taken from some existing checkpoints?\n2. In the main experiment, how does the spectrum-tuned model perform under zero-shot settings? (i.e., we only evaluate on the first output)\n3. What is the rationale in changing system/user/assistant to description/input/output?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is the first paper that considers the in-context steerability tasks\n2. It provides insightful results about the degradation of IT models on this specific task\n3. It provides a large-scale dataset for the task", "weaknesses": "1. Experiment setting seems over-complicated. It does not make sense to me to evaluate on the first few outputs, if the purpose is to evaluate in-context steerability.\n2. Section 4 is missing an ablation study on temperature. It is claimed that the spectrum tuned models is Pareto optimum comparing to IT tuned models with temperature 1. But it would clearly make more sense if you use a couple of different temperature values for the IT models and see whether spectrum tuned model with a specific temperature is better than all of them, or plot a curve for both methods if the advantage is not that obvious.\n3. Missing a baseline that instruction-tunes the model on the spectrum-suite training set. The paper only shows that a model, trained on the spectrum-suite training set with spectrum tuning, performs better on the spectrum-suite test set, comparing to models that are not trained on this dataset (PT and IT). This does not demonstrate the effectiveness of spectrum tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRTzedwtGY", "reviewer": "Reviewer_nA29", "summary": "The paper investigates how the post-training of models (through instruction-tuning) negatively affects a specific set of desiderata, such as the ability of the in-context examples to steer the large language model towards a specific perspective, or diversity in the outputs for tasks with many valid responses. The authors prepare a dataset of tasks, called Spectrum Suite, to evaluate these desiderata and compare them between base and instruction-tuned models, finding that instruction-tuning breaks the steerability of LLMs for the generation tasks (while increasing performance for classification tasks or tasks with one/few specific valid answers). Building on this dataset, the authors introduce Spectrum Tuning, a post-training approach that provides the benefit of instruction-tuning withouth breaking the LLM desiderata (i.e., steerability and diversity of outputs).", "questions": "See weaknesses for details, but mostly relate to details:\n\nHow are the metrics, such as \"yield\", \"calibration\", and \"diversity\", calculated?\n\nIs the Spectrum tuning trained on one set of tasks and then evaluated (and compared with instruction-tuning) on a separate, unseen set of tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper deals with important problem and I believe it can lead to many possibilities for intersting discussions and future work not only in post-training, but also in areas of interpretability.\n\nThe authors run an extensive set of experiments, providing a comprehensive evaluation on \"failure\" modes of instruction-tuned models, while proposing a solution for mitigating it. In addition, authors provide important resources (Spectrum Suite dataset).", "weaknesses": "**Paper structure/writing**\n\nMy biggest and only issue with the paper is how it is structured and the overall presentation. Overall, the paper is not really well put together -- although all the information is there, it is kind of \"all over the place\" and there are many explanations of motivation, connections between ideas, and details missing.\n\nFirst of all, the abstract and introduction do not really introduce or motivate the problem and it is not really clear why it is important to deal with it -- only 2 sentences are dedicated to this (Liens 33-37) and afterwards the contributions and the description of the desiderata is introduced. I would suggest expanding the motivation in introduction and abstract as it would improve the paper quite a lot. In addition, I would suggest having a dedicated section for the Desiderata and providing more in-depth description or discussion on them. Similar problems are in the dataset and method description, which are quite short and deserve more detailed description and explaining the motivation behind them.\n\nFor the experiments section, the details for how the individual metrics are calculated are missing -- for example, there are lot of references to \"yield\" in Section 4, which is not explained and makes it more problematic to understand the results. Similar case is with \"diversity\" or \"calibration\" as it is not clear how these metrics were calculated. In addition, when performing Spectrum tuning and comparing with instruction-tuned models, it is not clear whether the evaluation is done on a completely different set of tasks or not.\n\nFurthermore, almost all of the figures are quite small and full of information, making them hard to read and easily understand -- for example, in Figure 4 the legend (what the colours mean) is small and very well hidden.\n\n(minor) \n\nLines 89-90 \"see 1 above\" -- it is not really clear what this is referring to; is it the \"exhibit natural person-to-person variation\" from the list at the beginning of the section?\n\nAppendix D seems to be empty, or at least does not refer to any additional figures\n\n\n\n**I acknowledge that many of the misunderstandings and difficulties understanding the paper are a result of limited space and can be easily fixed during rebuttal period, and will update my score if addressed**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)