Don't Throw Away Your Pretrained Model
Authors: Shangbin Feng
Wenhao Yu
Yike Wang
Hongming Zhang
Yulia Tsvetkov
Dong Yu
Venue: ICLR 2026 Poster (2026)
Keywords: model collaboration
collaborative inference
Primary area: foundation or frontier models, including LLMs

Abstract: Alignment training has tradeoffs: it helps language models (LMs) gain in reasoning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models in the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose Switch Generation, where pretrained and aligned model versions take turns to ``speak'' in a response sequence. Specifically, we train a switcher LM by learning from outcomes of choosing different models to generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints to dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18 datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) Switch Generation further outperforms baselines by 12.9% on average. Further analysis reveals that Switch Generation discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by-products in expensive model training pipelines that are otherwise discarded.

Published as a conference paper at ICLR 2026
DON'T T THROW AWAY YOUR PRETRAINED MODEL
Shangbin Feng' Wenhao Yu² Yike Wang' Hongming Zhang² Yulia Tsvetkov! Dong Yu²
Unniversity Washington Teencett AI Scattle Lab shangbinecs washington.cc edu
ABSTRACT
Alignment training has tradeoffs: it helps language models (LMs) gain in rea- soning and instruction following but might lose out on skills such as creativity and calibration, where unaligned base models are better at. We aim to make the best of both worlds through model collaboration, where different models the training pipeline collaborate and complement each other. Since LM responses feature interleaving skills that favor different models, we propose SWITCH GEN- ERATION, where pretrained and aligned model versions take turns to "speak" in a response sequence. Specifically, we train a switcher LM by learning from out- comes of choosing different models generate the next segment across diverse queries and contexts. At inference time, the switcher LM guides different model checkpoints dynamically generate the next segment where their strengths are most needed. Extensive experiments with 8 model collaboration baselines and 18 datasets show that 1) model collaboration consistently outperforms individual models on 16 out of 18 tasks, and 2) SWITCH GENERATION further outperforms baselines by 12.9% on average. Further analysis reveals that SWITCH GENERA- TION discovers compositional skills to solve problems where individual models struggle and generalizes to unseen models and tasks, reusing and repurposing by- products in expensive model training pipelines that otherwise discarded."
INTRODUCTION
Alignment/RL. has become an integral part in language model (L.)) training, improving mod- els on skills such as reasoning and instruction following (Ouyang et al., 2022; Guo et al., 2025). However, it is not a Pareto-optimal strategy (Lin et al., 2024): aligned models have tradeoffs on skills such as creativity (West & Potts, 2025), calibration (Tian et al., 2023), and generation diversity (Sorensen et al., 2024b; Yue al., 2025; Yang & Holtzman, 2025), where unaligned base models are better at. How to make the best of both worlds essential for handling complex tasks that require composi- tional skills and developing AI systems that are flexible and adaptable diverse user needs and contexts.
① Instruction: In which film was actor Glenn Strange unaware he being
play Frenkenstein's monster until he reported to the make-up up room?
The actor Glenn Strange was known for playing the Frankanstein's monster in several films, but you are asking for a specific fllm when is documented that he was unaware ne was to play
Response:
Instruction Following
until arriving Glenn Strange also played the role of the monster several Uniierrsal Pictures' Frankenstein sequels, Including 'Son of Frankenstoinn and the original "Frankenstein: Thhreeorr, not known it Glenn Strange was unaware he was to play the Frankenstein's monster until reported to the make-up room. However, he may havebeen unaware that was to play
make- room.
Recalling Knowledge
Reasoning
"House of Frankonstoin" film, because he was credited as Monster" in these films was not specifically credited as Gienn Strange. The anawer is House of Frankoenstein.
Summarizaaion
Figure 1: Model responses not monolithie: they switch across diverse skills which favor dif- ferent model checkpoints in the training pipeline, thus we introduce model-guided collaborative in- ference to optimally use models with diverse skills for different segments of response generation.
To this end, resort to model collabora- tion (Feng et al., 2025a), where diverse model checkpoints (e.g. pretrained and aligned ver- sions of models') collaborate, compose, and complement each other. Since model responses
Code and data available in MoCo (Feng al., 2026) Pretrained models indicate models after autoregressive pretraining on mass corpora, finetuned models indicate models after instruction tuning, and aligned models indicate models after alignment and reinforeement learning. They are usually successive steps in model development.

---

Published as a conference paper at ICLR 2026
Learning the switcher
Query
What is the company with the largest market cap that doesn't directly sell things custamers?
diverging sogments
rollout
ktime
overage
graphics processing units IOPUs) and systems chip (9oC]
Ao of July 2025, the company with the largest mai kut that dosen't directty things customers (... not primarily direct-to-cone co-nnumme bustnees] NVDI
companies:
.70
0.68
Trice t
ramicondacter components to enterprise-lande clients, Including major technology firms, data center operators, and automotive manufacturers.
While NVVDDAS chips are integral to many consumer products, their care businssa designing and supplying these advanced
0.42
platforma specialized applications like artifciaal brithccilineelligrrr intellignce, data centers and professional visualzzaiion:
Candidete C
[ pretrained
finetuned
t aligned
switcher f(g.
> Using the switcher
Query 4 Please detail every property transaction in New York City from the year 2000 to the present.
resporse
der to obtain an approxcmate answer, hhvve used following information. have used average price per square foot for past years New York City. According dlnng to Zillow the everage price per square foot $1,200 Second, have used the papulation of hiw York City from
fat None, ++ 0
obtain approxmmate
City, According information.
the
have used average
order f1,,6
per square food Second, have populattin New City from the year the present However impossible to gather information. not detall every property
years
age price
populstion York from
order per squarn foot amprroxaiata
following information, have
average
/(g,
der to... the population York from year 2000 ... detsil every proparty
poputation New City from year 2000 the present. However, impossible gather of information. it feasible detail every property transaction New York from the year 2000 the present. The resson there are millions of property transactions New City. over the post two dww.deeaddee decades, and there are no complete reconde of property transactions.
Second, have years
city According
average price squuare
Figure 2: Overview of SWITCH GENERATION, where multiple model checkpoints in the training pipeline (e.g., pretrained, finetuned, and aligned LM checkpoints) are dynamically selected to gen- erate text segments in a sequence. (Up) We derive training data for the switcher LM f by rolling out which model would lead to the best average outcome for particular query and trace. (Down) At inference time, multiple models are guided by the trained switcher LM to generate text segments as part of response when their skills and strengths are most needed.
are not monolithic and feature a wide variety of skills favoring different models (Figure 1), we pro- pose SWITCH GENERATION, where different models in the training pipeline take turns to "speak" in response sequence. Specifically, we train a (small) switcher LM to decide which model should generate the next segment based on the query. trace (what has been generated thus far), and (model) candidates. For any (query, trace) pair, we let each model candidate generate one more segment, randomly sample continuations, and evaluate which candidate has led to the best average perfor- mance: this yields supervised fine-tuning data for the switcher LM, where it learns to predict the best model checkpoint for generating the next text segment on diverse (query, trace) pairs. During inference, the switcher LM dynamically selects the most suitable model checkpoint for each seg- ment, so the final response is generated as a sequence of turns where different models contribute under the switcher's guidance (Figure 2).
Extensive experiments with 8 model collaboration baselines (spanning API, text, logit, and weight- level collaboration) and 18 datasets (e.g. QA, reasoning, instruction following) demonstrate that:
Don't 't throw away your pretrained model: model collaboration approaches outperform all individ- ual models on 16 out of 18 datasets (close second on the other two).
SWITCH GENERATION presents a strong paradigm for collaborative inference: SWITCH GEN- FRATIOO outperforms all baselines on 13 datasets with average improvement of 12.9%.
Further analysis reveals that SWITCH GENERATION generalizes to unseen tasks and model set- tings, helps solve problems impossible for any of the models when used individually, identifies high-quality switching patterns, and texts generated through SWITCH GENERATION can be dis- tilled back into a single model for efficiency.
Our work put forward a new vision to reuse, recycle, and repurpose byproducts in expensive model training pipelines that have huge potential but are currently neglected and underappreciated.
2
METHODOLOGGY
We propose SWITCH GENERATION, collaborative inference algorithm where diverse model check- points in the training pipeline are dynamically selected to generate successive segments of the re- sponse. SWITCH GENERATION aims to dynamically leverage the complementary strengths and
2

---

Published as a conference paper at ICLR 2026
expertise of different model checkpoints (e.g., pretrained, finetuned, and aligned), especially for complex problems that require compositional skills. The core of SWITCH GENERATION deciding "who should speak at when", formally the Query-Trace CandddCan Problem (the QTC Problem):
t (pI, (I,,..., PP. R",
where q denotes the queryinsttruction t denotes the trace, i.c. what has been generated thus far, = fci,.... Cn denotes the pool of candidate language model checkpoints, and p; denotes the likelihood of choosing model c; to generate the next text segment. The QTC problem essentially tackles the research question: "Given the question and what has been said thus far, which model is best suited to speak next?"
This differs from existing routing paradigms (Ong et al., 2025; Frick et al., 2025) in that: 1) the trace f. * Ø; 2) each selected model c, only generates text segments instead of the full response; 3) the switcher f is applied multiple times instead of just once. This brings novel ways of combining model strengths, finer-grained control over model collaboration, and improved adaptability to diverse user requests.
Parameterizing the switcher We parametrize the switching strategy f as a (small) language model and encode its input q. t. and C with the following prompt:
Prompt 1: Switcher Prompt
query (model i begins) text generated by model (model i ends) (model j begins) text generated by model (model j ends) Which model should generate the next segment? Please respond with a number from 0 n-1. The answer is model
We provide the switcher f with meddidate-marken trace t using special token delimiters ((model begins/ends)), aiming to decipher what is needed next and who might be helpful by learning from the model-attributed generation history. The switcher f then predicts model ID, and we take the logits of 0 to n-l as [Pi, .Prl By making the switching strategy / compatible with natural language and language models, we seek leverage their language capabilities to aid in the QTC problem.
Learning the switcher Given any instruction q:
We randomly sample trace t (a partial response to the query) with random switching frandom Uniform(n). i.e., randomly choosing models to generate a segment after another. Trace t is capped a random threshold of 10% 90% of the maximum response length, aiming to capture switching behavior at different stages response completeness.
From the generated trace we take one divergent step: different models generate one more seg- ment following it: {t, t/le, (q. t.. tt, = c, (q, t) denotes string concatenation.
We sample & continuations for each t, with frandom- aiming to roll out diverse outcomes of choos- ing model c, at this particular (q. t). The utility for choosing c, is then:
1 k
8;
score(t,, Franstom q))
where score uses any evaluation metric corresponding q (accuracy, Fl-match, reward scores), Let arg max; 8, then model C, should be selected at this particular (q, t). This then yields { C, C) + cy }. supervised fine-tuning instance for training the switcher f (it should predict the model id g after "The answer model" " in Prompt 1). By sampling such SFT data points over diverse E Q, we obtain dataset for training the switcher LM
Using the switcher At inference time, the trained switcher f guides switching patterns among diverse model checkpoints for collaborative generation. While existing works might change models at every token (Shen et al., 2024; Fei et al., 2024), we propose to call the switcher per patch (a fixed set of tokens) it: 1) scales betters (Pagnoni et al., 2024), 2) preserves the continuity of thought for

---

Published as a conference paper at ICLR 2026
models instead of being interrupted every token, and 3) incurs much fewer times and thus much less cost of calling the switching strategy f-
We employ top-p (mucleus) sampling (Holtzman et al., 2020) to select model from the distribu- to t, C)) - c (instead of greedy selection), balancing utility and
tion [p1.... Pnl:: exploration in switching generation.
At first, given the query and no trace, we select model c(1) = top-p(/(s (((()),)))) Ø,C),, generate a patch of tokens e(1) (q), and append to trace ((2) c(
At the i-th step, we select model cl top-p t(1-1),))) ,)), generate patch cl (q]l(((-1)) and append to trace (*) (1-11 || (all((--)))
We continue until the generation ends or the maximum amount of tokens is reached. To sum up, SWITCH GENERATION employs diverse model checkpoints the training pipeline to collabora- tively generate, complement each other, and advance compositional intelligence.
EXPERIMENT SETTINGS
Models and Implementation by default employ the three models in the pretrained-finetueed. aligned pipeline of Tulu-v3 (Lambert et al., 2024) Ameta-llama/Llame
finetuned- 1-8B, allenai/Llama-3
1 1-Tulu-3-888) due to its transparency and experiment with dif- ferent model checkpoints, number of models or model settings in Section 5. We employ the aligned model (allenai/Llama-3.. -----u-----8B 3-8B) to initialize the switcher f. sample 10k switcher SFT instances for each task with 32, and train f for 5 epochs with 2e-4 learning rate and 32 batch size under two settings: switch-g(lobal), where one switcher is trained on the SFT data across all tasks; switch- (lask-specifie), where one switcher is trained on the SFT data for each task. At inference time, all methods generate 512 new tokens at max by default; for SWITCH GENERATION, we use the aligned model in the first and last patch, employ patch size of 50 tokens, and top-p sampling p = 0.7 by default.
Talu-3-8B-SFT, and allenai/Llama-3..
Baselines We compare SWITCH GENERATION with 11 baselines: the pretrained, finetuned, and aligned models employed individually, API-level collaboration (prompt-based routing (Feng et al., 2024a) and RouteLLM (Ong al., 2025)), text-level collaboration (collaborate (Si et al., 2023) and debate (Du et al., 2023)), logit-level collaboration (logit merge and proxy tuning (Liu et al., 2024)), and weight-level collaboration (greedy soup (Wortsman al., 2022) and dare-ties (Yadav et al., 2023; Yu et al., 2024)). These baselines cover a wide range of model collaboration protocols across diverse levels of information exchange.
Data and Evaluation We employ 18 datasets spanning 3 categories:
knowledge and factuality (WikiDYK (Zhang et al., 2025) and TruthfulQA (Lin et al., 2022)), creativity (poem (West & Potts, 2025) and GuessBench (Zhu al., 2025)), pluralism (Sorensen et al., 2024a), sycophancy (Cheng al., 2025), randomness, generation diversity (movie reviews (Wang et al., 2023b)), and uncertainty (AbstainQA (Feng et al., 2024b)), as supported by the findings of existing literature.
Datasets where having the base model might be helpful:
Datasets where having the base model might be worse: reasoning (gsm8k (Cobbe et al., 2021), BigBench-Hard (Suzgun al., 2023), and NLGraph (Wang et al., 2023a)), instruction following (Alpaca (Dubois et al., 2023)), and safcty (Coconot (Brahman et al., 2024)), since these skills are explicitly what alignment is for.
Datasets where the effect of base models is unclear: general QA (MMLU-pro (Wang et al., 2024), AGIEval (Zhong et al., 2024), and PopQA (Mallen et al., 2023)) and scientific literature (Sci- enceMeter (Wang et al., 2025a)).
These datasets cover a wide range of LM capabilities that favor different model checkpoints in the training pipeline. If SWITCH GENERATION improves category-1 tasks, slightly behind/on par on c2tegory-2 tasks, and on par/improves on category-3 tasks, it presents a promising collaboration strategy fuse the strengths of model checkpoints and enable them to complement each other.
4

---

Published as a conference paper at ICLR 2026
Table Performance of individual models and model collaboration methods. Green, red, and yellow denote category-1/2/3 tasks in Section 3. denotes the lower the better. Best in bold and second- best in underline. Model collaboration approaches outperform employing models individually on 16 out of 18 tasks. SWITCH GENERATION achieves the best performance on 13 tasks with 12.9% relative improvement over baselines on average.
WikiDYK
TruthfulQA
AbstainQA Phuralism Sycophancy GuessBench Numhers (L)
Movie
PRETRAINED
1.70
10.37
24.55
62.92
32.20
12.80
2.00
0.26
13.76
FINETUNED
3.27
30.63
49.45
49.44
58.90
13.00
4.40
0.90
3.82
ALIGNED
3,92
29.01
77.70
44.38
50.90
16.40
6.40
0.42
0.93
3.27
54,40
53.93
45.80
6.40
4,94
3,01
34.38
61.60
58.43
51.80
14,20
4.40
0.87
13.05
TEXT COLLAB
3.14
18.15
52.30
28.65
28.30
11.80
6.80
0.59
3.24
TEXT DERATE
3.53
16.86
$1.10
30.90
43,00
6.40
8.80
0.54
2.65
LOGIT MERGE
1.57
9.08
$3.80
12.92
23,90
14.80
1.20
0.22
10.75
PROXY TUNING
1.96
2.76
72.60
26.40
10.70
15,20
2.00
19.56
4.58
33.06
77.95
60.67
$7.80
13,20
13.91
2.61
23.99
47.90
52.81
37.50
12.40
1.60
0.31
10.60
WWITCH-GLOBAL
4,44
34,04
70.90
60.67
55.60
16,80
10.80
0.41
14.14
SWITCH-TASK
5.75
39.22
70.25
74.16
53.20
17.40
13.60
0.44
17.39
GSM8k
CocoNot
Alpaca
BBH
NLGruph
MMLU-pro
AGIEval
PopQA
Science
27.20
14.51
38.10
44.50
5.10
4.41
29.80
FINETUNED
37.20
64.00
52.43
26.70
38.33
13.10
11.94
26.10
56.30
ALIGNED
56.80
53.10
57.46
35.20
41.83
10.40
11.85
31.20
60.60
PROMPT ROUTE
48.10
43.60
42,90
43.40
44.83
5.10
11.68
29.90
51.80
ROUTELIM
48.10
57.60
43.97
45,90
48.67
10.50
12.32
31.30
59.00
TEXT COLLAB
34.60
7.10
15.74
46.90
40.90
52.35
39.10
44.67
8.30
15.05
29.60
52.70
LOGIT MERGE
38.30
16.70
28.46
39.00
25.67
1,50
7.01
21.60
10.50
PROXY TUNING
44.50
7.30
37.27
44.00
43.83
1.40
2.34
22.50
11.40
GREEDY SOUP
58.10
66.00
54.89
36.50
45.33
12,70
11.76
31.30
60.30
DARE TIES
22.90
19:40
40.01
30.30
42.33
6.50
7.87
25.10
46.50
49.50
70.90
49.06
$2.60
59.67
13.00
14.19
33.70
59.80
SWITCH-TASK
59.60
72.80
56.89
58.30
61.67
16.70
25.26
37.70
67.20
RESULTS
We present the performance of individual models and model collaboration methods in Table 1.
Don't throw away your pretrained model. Model collaboration among pretrained, finetuned, and aligned language models, baselines ours, outperforms using these models individually 16 of 18 tasks with 31.0% relative improvement on average. This indicates that checkpoints other than the aligned models diamonds in the rough, successfully complementing each other and contributing their unique strengths.
Table Ablation study of generation patch size and switch- ing strategy. Different tasks might favor different patch sizes while fine-tuning the switcher f consistently helpful.
Switch Generation offers a strong collaboration strategy. SWITCH all
GENERATION outperforms individual models and model collab- oration baselines on 13 datasets, with an average relative improvement of 12.9%. In addition to improving on cat-1 tasks, SWITCH GENERATION also gains 6.58 points on average across cat-2 and cat-3 tasks, where it originally uncertain whether having the base model in collabora- tion might be helpful. This indicates
Setting
TruthfulQA Pluralism GSMSk
BBH PopQA
39.22
$3.20
59.60
58.30
37.70
PATCH SIZE: 100
30.31
53.30
44.70
40.40
32.00
PATCH SIZE: 30
35.79
55.70
$2.20
53.50
33.80
PATCH SIZE: 20
32.58
56.80
47.80
48.10
31.80
PATCH SIZE: 10
30.96
54.50
51.40
48.70
32.80
RANDOM SWITCH
27.07
54,80
44.70
53.10
27.90
UNTUNED SWITCH
31.12
53.10
47.90
41.80
32.10
5

---

Published as a conference paper at ICLR 2026
that by collaborative inference with flexible switching strategy, approach adapts to diverse tasks through leveraging the strengths of candidate models and fusing their strengths.
Routing-based approaches are best for pretrained-aligned collaboration. descending order, bautingbbase,, weight-based, text-based, and bogit-based baselines achieve 31.15, 29.91, 26.32, and 18.97 points on average, indicating that routing-bac methods are best suited for the collaboration of aligned and unaligned models, since different tasks require different skills that favor varying models. SWITCH GENERATIO segment-level, so diverse model checkpoints could dynamically contribute in the problem-solving process when their skills are most needed.
further provides finer-grained and more flexible routing on the
5 ANALYSIS
Ablation study We conduct abla- tion study on two key design choices
Pretrained
Finetuned
Aligned
distil(AAligned) full ours
60.00
in SWITCH GENERATION: 1) gener- ation patch size: we by default gen- erate 50 tokens per model and call the switcher f and we additionally employ {10, 20, 30, 100} in this by default train the switcher f through supervised fine-tuning simulated switching outcomes and we addi- tionally employ random switching (rondom Uniform(n)) and un- tuned switching (directly employing the aligned model as f without fine- tuning). Results in Table demon- strate that different tasks might ben- efit from different switching granu- larity: by employing more frequent switching on Pluralism our approach further improves. Fine-tuning the switcher with our methodology is consistently effective out- performs random and untuned switching on all five tasks.
40.00
study; 2) switcher training:
20.00
0.00
WIKIDYK
TruthfulQA
GuessBench
BBH
NLGraph
PopQA
Figure Distilling the collaboration patterns of SWITCH GENERATION back into the aligned model. Distillation re- covers 58% of the collaboration gains with only one fourth of the inference cost.
Distillation back into a single model At in- ference time, SWITCH GENERATION loads and generates texts with n+1 LMS simultaneously. While this is fast with multiple GPUs and mul- tiprocessing, we propose to reduce inference costs by distilling the switching patterns back into a single model. The aligned model was pretrained and finetuned too, by dis- tillation we hope that it could recover the sub-
1.25
1.00
P-helpfulness
0.75
0.50
0.25
merged capabilities of strengths of its previous forms. For inputs in a dataset, we 1) gener- outputs with full SWITCH GENERATION, 2) fine-tune the aligned model on the generated outputs, and 3) evaluate the performance of the distilled aligned model when used individually. Results in Figure demonstrate that distillation successfully helps the aligned model pick up the submerged skills, recovering 57.5% of the gains of SWITCH GENERATION with only one fourth of the inference cost (one model only three models and switcher LM). This
0.00
-0.25
-0.60
-0.40
-0.20
0.00
0.20
P-performance
Figure 4: Correlation between the performance and helpfulness the pretrained model. While not being the best individual model, it is consis- tently helpful in the model collaboration system.

---

Published as a conference paper at ICLR 2026
Table 3: Performance when directly employing the trained switcher model for other model settings: PFA from another model family (qwen 2.5 7B). PA only (of Tulu-v3), PF and two versions of A (of Tulu-v3, one DPO and one RLVR), and three specialized LM experts (seperately fine-tuned versions of qwen 2.5 7b). The trained switcher consistently generalizes to these settings to varying extents.
Setting
Setting
Setting 3
Setting 4
Truthful
BBH
PopQA
Truthful
BBH
PopQA
Truthful
BBH
PopQA Truthful
BBH
PopQA
MODEL #1
26.09
69.50
24.10
10.37
38.10
15.30
10.37
38.10
15.30
40.19
48.40
20.90
MODEL.
53.00
54.80
25.60
29.01
35.20
31.20
30.63
26.70
26.10
53.00
54.80
25.60
MODEL
63.29
70.70
23.30
32.58
36.20
31.10
54.29
56,20
22.20
MODEL
/
29.01
35.20
31.20
/
OURS
66.64
73.70
27.60
32.58
45.60
34.60
35.82
48.70
31.70
55.27
57.60
26.90
cess sheds light on the broader potential of distilling multi-modellagent systems back into a single model/agent for inference-time efficiency.
Correlation between individual performance and helpfulness While the pretrained base model has many strengths, it consistently isn't the hest-performing individual model in Table 1. How- ever, this doesn't prevent it from being helpful in the collaboration and contributing its strengths when needed. We quantify this phenomenon with two metrics for each task: P-performance = P-max(P.F.A) P-helpfulness
where P. F, A indicate the performance of pretrained, finetuned, aligned models when employed individually and C(P, F, A) indicates the performance of their collaboration (through SWITCH GENERATION). Results in Figure 4 demon- strate that the vast majority of tasks fall into top-left quadrant: while the pretrained model isn't t the best when employed individually (P-performance < 0), SWITCH GENERATION leverages its strengths to gain in collaboration IP-helpfulness > 0). This highlights the broader potential that weak models are not useless: they rightfully diamonds in the rough and contribute their unique strengths when employed in the right model collaboration system.
(..(
Fax(/.FFA)
Generalizing to unseen models The trained switcher LM (in switch-global has learned from diverse tasks, contexts, and model collaboration patterns among the Tulu-v3 suite of models. We hypothesize that of could be employed off- the-shelf for switch generation with other model settings, in increasing difficulty and generalization gap:
Setting J: pretrained, finetuned, and aligned models in another model family, specifically Qwen2.5-7B (Yang et al., 2024).
Setting 2: one fewer model: only pretrained and aligned of Tulu-v3.
Setting 3: one more model: pretrained, finetuned, and two versions of aligned (DPO and RLVR) of Tulu-v3.
Seting 4: three specialized LM experts (Jiang et al., 2025) that are not the aligned version each other.
Table shows that the switcher consistently generalizes to these four settings, with an average relative improvement of 5.8%, 14.3%. 13.1% %, and 3.1%. We will release the switcher model ff as an artifact and encourage readers to employ for switch generation with their suite of models.
Table 4: The trained swicher model generalizes to unseen tasks, outperforming basclines on most tasks.
Generalizing unseen tasks We directly employ the trained switcher LM f (in switch-global) and com- pare it against two strong baselines on six additional tasks spanning the three task categories (Normad (Rao et al., 2025), human interests (Feng al., 2025b), MATH (Hendrycks
Nornad Interests MATH K-Cross
ARC
MedQA
PRETRAINED
29.85
43.47
27.89
7.00
16.13
20,30
FINETUNED
46.40
63.30
25.75
24.60
45.22
26.50
ALIGNED
48.70
66.95
28.74
20.80
46.76
28.10
TEXT DERATE
34.90
70.10
34.55
12.00
30.97
28.50
GREEDY SOUP
48.85
67.35
29.41
25.50
47.44
28.90
SWITCH- SWITCH-GLOBAL
50.65
69.78
25.70
48.38
31.50

---

Published as a conference paper at ICLR 2026
pretrained
finetuned
aligned
0.30
sequence
frequency
treatment effect
frequency
treatment
effect
0.365
treament affect
0.503
0.136
0.059
0.10
0.315
0,078
0.161
0.057
0.646
0.115
0.115
0.061
0.00
0.536
0.198
-0.10
9.611
0.12
0.14
0.16
0.18
0.20
frqquincy
Figure 5: Frequency and treatment effect of 2-length (left) and 3-length (middle) switching se- quences, and their correlation across three task categories for 3-1ength sequences (right). SWITCH GENERATION learns to identify helpful switching patterns and frequently leverages them.
al., 2021), Knowledge Crosswords (Ding al., 2024), ARC-challenge (Clark al., 2018), and MedQA (Jin et al., 2021)). Results Table 4 demonstrate that by learning from the switching patterns across diverse tasks, SWITCH GENERATION generalizes unseen tasks and outperforms baselines by 3.9% on average.
Good sequences By running SWITCH GENERATION across 18 tasks, we accumulate valuable traces of model collaboration and switching patterns. Within them exist many switching sequences (e.g. "PFA": pretrained generates first, followed by finetuned, followed by aligned): we could identify which of these sequences are good, we could 1) directly employ these switching patterns off-the-shelf without calling
Table SWITCH GENERATION solves 10.7% problems that no individual model did. Color shades denote outcomes that discover new skills, retain existing skills, no change, and might lose skills.
Truthful
AbstainQA
BBH
PFA / ours correct wrong correct wrong
correct wrong
all correct
3.6%
0.3%
15.7%
2.2%
7.4%
0.9%
> correct
25.3%
7.6%
50.6%
5.1%
37.1%
8.6%
wrong 10.4%
52.8%
7.9%
18.5%
13.8%
32.2%
the switcher LM for efficiency and/or 2) steer SWITCH GENERATION towards employing these
sequences more often. We define two metries for switching sequences:
Frequency: what percentage of responses was this sequence employed?
Treatment effect: performance when this sequence is employed minus when not employed.
We present results for unique 2-1ength and 3-1ength sequences as well as their correlation in Figure 5: the most helpful sequences (with the highest treatment effect) are also among the most frequent. This indicates that SWITCH GENERATION ATION learns to identify helpful switching patterns and more frequently leverage them for better collahoration.
New skills We hypothesize that the perfor- mance gains of SWITCH GENERATION might come from two aspects: 1) aggregate skills that one of the models already has. and 2) solving problems that none of the models could solve individually. We present the statistics between single-model and multi-model correctness in Table 5: SwITCH GENERATION successfully answers 10.7% of problems that none of the in- dividual models could, while only losing out on 8.2% of problems that model could individ- ually solve, netting a benefit of 2.5% through model collaboration.
Pretrain
knowledge
recall
instruction
following
Finetune
reasoning
creativity
feedback
Align
others
0%
25%
50%
75%
100%
Figure 6: Roles that each model plays in their gen- crated segments, averaged across all datasets.
8

---

Published as a conference paper at ICLR 2026
Model Roles Model responses often feature wide range of skills that favor different model stages (Figure 1); we investigate under SWITCH GENERATION, what are the roles of the pretrained, fine- tuned, and aligned langauge models in generated texts? Specifically, five skills with high frequency (knowledge recall, instruction following, reasoning, creativity, and feedback). We then employ L.LM-as-a-judge (Zheng al., 2023), specifically the GPT-40 model, to annotate each generated segment for one of the skills (or "others") given the query, trace, and the full generated sequence. We manually examine 100 LLM annotations and find an 86% agreement between human-LLM judgements (with the most difference in the "others" category). We average it across datasets and report the results in Figure 6: it shows that models are largely performing the skills that they are good at: for example, the pretrained model is most frequently used for knowledge recall, while the aligned model is most used for reasoning. This indicates that SWITCH GENERA- TION and the trained switcher f learns to leverage model strengths when their skills are most needed.
first identify suite of
6 RELATED WORK
The Tradeoffss of Alignment Alignment and reinforcement learning have become indispens- able part of language model training: they are credited for valuable skills in state-of-the-arr LMs such as reasoning (Guo et al., 2025), safety (Zhang et al., 2024), agentic applications (Ma et al., 2024), and more (Ouyang et al., 2022). An increasing line of research recognizes that alignment has tradeoffs (Lin et al., 2024), that the pretrained and unaligned base models might have advantages on skills such as creativity (West & Potts, 2025), uncertainty (Tian et al., 2023), pluralism (Feng et al., 2024c), knowledge (Wang et al., 2025a), or even reasoning itself (Yue al., 2025). However, we couldn't I directly employ the base model for these domains since they struggle to follow instructions and lack safety guardrails. We propose to make the best of both worlds by not throwing away your base model and instead leveraging model collaboration across diverse checkpoints in the training pipeline to fuse model strengths and complement each other.
Model Collaboration Advancing beyond training single, generalist language model, recent re- search is increasingly emphasizing modularity through model collaboration, where diverse (lan- guage) models collaborate, compose, and complement each other (Feng et al., 2025a). Model col- laboration approaches mainly vary by the level of information exchange: API-level methods such as routing (Ong al., 2025; Frick et al., 2025; Feng al., 2025c; Zheng al., 2025) and cascading (Chen et al., 2023; Gupta et al., 2024; Yue et al., 2024), text-level methods through collaboration
(Feng et al., 2024b; Guo al., 2024: Sun et al., 2024; Zhao et al., 2024a; 2025; Dang et al., 2025) or competition (Du et al., 2023; Liang et al., 2024; Zhao et al., 2024b), logit-level methods with logit fusion or contrast (Pei et al., 2023; Li et al., 2023; Mavromatis et al., 2024; Chuang et al., 2024; Mitchell et al., 2024; Liu et al., 2024: Huang et al., 2025), and leieht-level methods such as model merging (Yadav et al.. 2023; Yu et al., 2024; Huang et al.. 2024; Feng et al., 2025b; Zeng et al., 2025) and Mixture-of-Expers (Sukhbaatar et al., 2024 Diao et al., 2023; Yadav et al., 2024; Shi et al., 2025). Since model responses are often not monolithic, featuring a diverse set of skills that favor different model stages (Figure 1). we propose SWITCH GENERATION for the collaborative inference of pretrained, finetuned, and aligned LMs where they take turns to generate in a response sequence. SWITCH GENERATION is related to various model collaboration protocols (Fei et al., 2024; Shen et al., 2024; Wang et al., 2025b) while uniquely training a switcher LM as the switch- ing strategy, switching by the granularity of patches, and offers generalization to unseen models as switching candidates. Our work also highlights that we don't 't need to always train new models for collaboration: byproducts in existing model development lifecycles could be reused and repurposed for new potential.
7 CONCLUSION
We propose SWITCH GENERATION, an inference-time model collaboration strategy where multi- ple models in the training pipeline are dynamically selected to generate text in a single response. By training and employing a switcher LM, multiple models dynamically generate text segments and contribute their strengths when most needed. Extensive experiments demonstrate that SWITCH GENERATION outperforms each individual constituent models and eight model collaboration base- lines on 13 datasets by 12.9% on average. Further analysis reveals that SWITCH GENERATION generalizes to unseen models and tasks, as well as identifying and frequently employing helpful col- laboration patterns. Our work uniquely highlights the huge potential of reusing by-product models and checkpoints in current LM training pipelines that are otherwise discarded.
9

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Shangbin Feng would like to thank the support of the IBM PhD Fellowship, the Jane Street Graduate Research Fellowship, and the NVIDIA Graduate Fellowship.
REFERENCES
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi. Valentina Pyatkin, Abhi- lasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et The art of saying no: Contextual noncompliance in language models. Advances in Neural Information Processing Systems, 37:49706-49788, 2024.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2023.
Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. Social sycophancy: A broader understanding of I1m sycophancy. arXiv preprint arXiv:2505. 13995, 2025.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo,, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1410 2021.
Yufan Dang, Chen Qian, Xucheng Luo, Jingru Fan, Zihao Xie, Ruijie Shi, Weize Chen, Cheng Yang, Xiaoyin Che, Ye Tian, et al. Multi-agent collaboration via evolving orchestration. arXiv preprint 295.:92005.1951, 2025.
Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. dixture-off-domai---aapeee Decoupling and injecting domain knowledge to pre-trained language models' memories. In Pro- ceedings of the 6/ Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 5113-5129, 2023.
Wenxuan Ding, Shangbin Feng, Yuhan Liu. Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric knowledge reasoning with large language models. Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI. 2024, August 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. Forty-first International Conference on Machine Learning, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36:30039-30069, 2023.
Fei, Yasaman Razeghi, and Sameer Singh. Nudging: Inference- -ime alignment of lims via guided decoding, arXiv preprint arXiv:2410.09300. 2024.
Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling llms' knowledge gaps with plug-in in specialized language models. The Twelfth International Conference Learning Representations, 2024a.
10

---

Published as a conference paper at ICLR 2026
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying Ilm knowledge gaps via multi-llm collaboration. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14664-14690 2024b.
Shangbin Feng, Taylor Sorensen, Yuhan Liu, Jillian Fisher, Chan Young Park, Yejin Choi, and Yulia Tsvetkov. Modular pluralism: Pluralistic alignment via multi-llm collaboration. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 4151-4171, 2024c.
Shangbin Feng, Wenxuan Ding, Alisa Liu, Zifeng Wang, Weijia Shi, Yike Wang. Zejiang Shen, Xiaochuang Han, Hunter Lang, Chen- Yu Lee, et al. When one Ilm drools, multi-llm collaboration rules. arXiv preprint arXiv:2502.04555 2025a.
Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Collabora- tive search to adapt IIm experts via swarm intelligence. In Forty-second International Conference Machine Learning, 2025b.
Shangbin Feng. Yuyang Bai, Ziyuan Yang, Yike Wang, Zhaoxuan Tan, Jiajic Yan, Zhenyu Lei, Wenxuan Ding, Weijia Shi, Haojin Wang, et al. Moco: A one-stop shop for model collaboration research. arXiv preprint arXiv:2601.21257, 2026.
Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for Ilm selections. In The Thirteenth International Conference on Learning Representationss 2025c.
Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopou- los, and Ion Stoica. Prompt-to-leaderin leaderboard.. arXiv preprint arXiv:2502. /4855, 2025.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi. Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's MergeKit: toolkit for merging large language models. In Franck Dernoncourt, Daniel Preojiuc-Pietre and Anastasia Shimorina (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing: Industry Track, November 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprini arXiv:2501. 12944, 2025.
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty-Thing International Joint Conference on Artificial Intel- ligence, pp. 8048-8057, 2024.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Ef- ficient cross-task generalization via dynamic lora composition. In First Conference on Language Modeling, 2024.
Chengsong Huang, Langlin Huang, and Jiaxin Huang. Divide, reweight, and conquer: A logit arithmetic approach for in-context learning. In Workshop on Reasoning and Planning for Large Language Models, 2025.
11

---

Published as a conference paper at ICLR 2026
Yuru Jiang, Wenxuan Ding, Shangbin Feng, Greg Durrett, and Yulia Tsvetkov. Sparta align- ment: Collectively aligning multiple language models through combat. arXiv preprint
W7X11:2500604721, 2025.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- case does this patient have? large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.1512 2024.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimiza- tion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12286-12333 12312, 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang. Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17889-17904, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Lingnistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 580-606, 2024.
Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. In First Conference on Language Modeling, 2024.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 2019.
Hao Ma, Tianyi Hu, Zhiqiang Pu, Liu Boyin, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevoly- ing with the other you: Fine-tuning IIm with sequential cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 37:15997-15525, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 0802-9822, 2023.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of lmss: Model fusion at test-time via perplexity optimization. In First Conference on Language Modeling, 2024.
Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. An emulator for fine-tuning large language models using small language models. In The Twelfth International Conference on Learning Representations, 2024.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wa, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning route llms from preference data. In The Thirteenth International Conference on Leurning Representations, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
12

---

Published as a conference paper at ICLR 2026
Artidoro Pagnoni, Ram Pasumuru. Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871, 2024.
Jonathan Pei, Kevin Yang, and Dan Klein. Preadd: Prefix-adaptive decoding for controlled text generation. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 10018- 10037, 2023.
Abhinav Sukumar Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Nor- mAd: framework for measuring the cultural adaptability of large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), April 2025.
Zejiang Shen, Hunter Lang, Bailin Wang. Yoon Kim, and David Sontag. Learning to decode col- laboratively with multiple language models. In P Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12974-12990, 2024.
Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, et al. Flexolmo: Open language models for flexible data use. arXiv preprint arXiv:2507.07024 2025.
Chenglei Si, Weijia Shi, Chen Zhao, Luke Zettlemoyer, and Jordan Boyd-Grates Getting more of mixture of language model reasoning experts. In Findings the Association for Computational Linguistics: EMNLP 2023, pp. 8234-8249, 2023.
Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19937-19947 2024a.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye. Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: A roadmap to pluralistic alignment. In Forty-first International Conference on Machine Learning, 2024b.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Roziere, Jacob Kahn, Shang- Wen Li, Wen-tau Yih, Jason E Weston, et al. Branch-train-mix: Mixing expert ilms into mixture-of-experts 1lm. In First Conference Language Modeling, 2024.
Hao Sun, Jiayi Wu, Hengyi Cai, Xiaochi Wei, Yue Feng, Bo Wang, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. Adaswitch: Adaptive switching between small and large agents for effective cloud-local collaborative learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 8052-8062, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether thain-of-thought thought can solve them. In Findings of the Association for Computational Lin- guistics: ACL 2023, pp. 13003-1351, 2023.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-544) 2023.
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36:30840-30861, 2023a.
Heng Wang. Wenqian Zhang, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Qinghua Zheng, and Min- nan Luo. Detecting spoilers in movie reviews with external movie knowledge and user networks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 16035-16050, 2023b.
13

---

Published as a conference paper at ICLR 2026
Yike Wang, Shangbin Feng, Yulia Tsvetkov, and Hannaneh Hajishirzi. Sciencemeter: Tracking scientific knowledge updates in language models. arXiv preprint arXiv:2505.24332,, 2025a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 2024.
Ziyao Wang, Muneeza Azmat, Ang Li. Raya Horesh, and Mikhail Yurochkin. Speculate, then col- laborate: Fusing knowledge of language models during decoding. Forty-second International Conference on Machine Learning, 2025b.
Peter West and Christopher Potts. Base models beat aligned models randomness and creativity. arXiv preprint arXiv:2505.00047 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lope Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties- merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recy- cling and routing among specialized experts for collaborative learning. Transactions on Machine Learning Research, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren. Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 200::2067..6.,,, 2024.
Chenghao Yang and Ari Holtzman. How alignment shrinks the generative horizon. arXiv preprint arXiv:2506. 1 17871, 2025.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Ab- sorbing abilities from homologous models as a free lunch. In Forty-fisst International Conference on Machine Learning, 2024.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representutions, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13833 2025.
Runjia Zeng, James Chenhao Liang, Cheng Han, Zhiwen Cao, Jiahao Liu, Xiaojun Quan, Qifan Wang, Tong Geng, and Dongfang Liu. Probabilistic token alignment for large language model fusion. arXiv preprini 2rXiv:2509.17277 2025.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang. Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'i don't 't know'. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 7106-7132, 2024.
14

---

Published as a conference paper at ICLR 2026
Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, and Jingbo Shang. Bidirectional lms are better knowledge memorizers? benchmark for real-world knowledge injection. arXiv preprint arXiv:2505.12306, 2025.
Justin Zhao, Flor Miriam
and Amanda Cercas Curry. Language model council: Democratically benchmarking foundation models on highly subjective tasks. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 12395-12450, 2025.
Kun Zhao, Bohao Yang, Chen Tang, Chenghua Lin, and Liang Zhan. Slide: A framework integrat- ing small and large language models for open-domain dialogues evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 15421-15435, 2024a.
Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Com- peteai: Understanding the competition dynamics of large language model-based agents. Forty- first International Conference on Machine Learning, 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-4662, 2023.
Wenhao Zheng. Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric Xing, Hongyi Wang, and Huaxiu Yao. Citer: Collaborative inference for efficient large language model decoding with token-level routing. arXiv preprint arXiv:2502.01976 2025.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299- 2314, 2024.
Zifeng Zhu, Shangbin Feng, Herun Wan, Ningnan Wang, Minnan Luo, and Yulia Tsvetkov. Guess- bench: Sensemaking multimodal creativity in the wild. arXiv preprint arXiv:2506. 00800, 2025,
LIMITATIONS
begin
middle
end
SWITCH GENERATION jointly employs multi- ple model checkpoints in the training pipeline for collaborative inference, which incurs ex- tra cost (compared to just using the aligned version) in exchange for compositional model strengths. The extra cost could be mitigated several fronts: 1) by employing multiple GPUs and multiprocessing for parallel text generation over batches of instructions, the throughput is much higher compared to using single model; 2) by calling the switching strategy every patch (instead of every token), the switching over- head is significantly reduced and the user could also configure the patch size customize the cost; 3) by distilling the collaboration patterns in SWITCH GENER ATION back into the aligned model (Figure 3), we recover part of the performance gains while cutting inference costs back to single model.
80.00%
60.00%
40.00%
20.00%
0.00%
pretrained
finetuned
aligned
Figure 7: Frequency of pretrained, finetuned, and aligned models being used in the be- gin/middle/end of the sequence.
We observe that switching by patches works better than tokens, so employ fixed-size patches in SWITCH GENERATION. We also observe that different tasks might need different amounts of generated tokens; thus, the optimal patch size might also change across tasks and contexts. We treat as a hyperparameter for now: future work could look into flexible and dynamic adjustments of patch sizes and switching frequency.
15

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide extensive experiment details such as hyperparameter settings, dataset statistics, and more in Section 3 and Appendix B. We will release the training and inference code, switcher LMs, and experiment logs upon acceptance.
ETHICS STATEMENT
SWITCH GENERATION is model collaboration protocol across multiple language models, so it is susceptible to malicious contributions: for example, the alignment datastore compromised and the aligned model is malicious, when used in collaboration, the system would also be seriously im- pacted. Safety in model collaboration systems is a critical future research question, and its findings would have great impacts SWITCH GENERATION.
As the base (and finetuned) versions of models are mostly not safett--lignnd, possible that having them in collaboration might override the safety guardrails of aligned language models. This might have implications for red teaming and adversarial language modeling.
A ANALYSIS (CONT.)
Model Locations We investigate whether pretrained. finetuned, and aligned language models might used more frequently in the beginning (first one-third), middle (one-third to two-thirds), and the end (two-thirds to end) patches in SWITCH GENERATION. Results in Figure demon- strate that the aligned model is more frequently employed in the beginning and the end while the pretrained/finetuned model more employed in the middle, suggesting that the middle of the sponse is more suited for exploration, while the beginning/end requires instruction following and summarization that favors the aligned model.
Switching Frequency How often does the switcher f decide that it's time to change to an- other model? We plot the switching frequency and collaboration helpfulness (P-helpfulness, Figure 4) in Figure 8: it is demonstrated that the switching frequency is consistently high, indi- cating that the models are actively used in col- laboration. There isn't a consistent conclusion about whether more or less switching is better for performance.
switch frequency (%) P-helpfulnees
1.25
1.00
0.75
0.50
0.25
0.00
-025
wildlyk initilugga
poem abatauqaa
pluratiom
sycophancy
gentk
coconct
nipace
bahh
nigraph
minti pro
agieval
sciencertetr
poppo
Qualitative Analysis We present examples where the pretrained, finetuned, and aligned models did not generate a good response in- dividually, while SWITCH GENERATION was successful in generating good response through collaboration in Tables 7 to 9. shows that SWITCH GENERATION has better deliberation and more extensive "reasoning" and explanation to reach a more well-rounded response. This also suggests that while the pretrained and finetuned models might not be following instructions when used individually, if the aligned model provides good context to work on, they will be helpful.
Figure 8: Switching frequency and P-helpfulness across tasks.
Another LM Family We additionally test SWITCH GENERATION another model family, specifically the base and aligned versions of Qwen 2.5 7B. Results Table 10 show consistent improvements over single models and baselines.
Additional Ensemble Settings We compare SWITCH GENERATION with four additional model ensemble settings: stacking (models sequentially refine the generation of the previous model), weighted voting (weighted by dev set performance). MoE (with top-1 expert activation), and simple ensemble of checkpoints. Results in Table 11 show consistent improvements over these baselines.
16

---

Published as a conference paper at ICLR 2026
Pushing the Limits on Two Datasets We find that two factors, smaller patch sizes and more SFT data for the switcher LM, are helpful for the two datasets Table where SWITCH GENERATION lagged behind baselines. Table 12 shows that these changes narrow the gap with baselines.
Runtime We present the training/inference runtime with 1, 2, and 4 GPUs in Table 13.
Token-Level Switching We compare with two ways of per-token switching: using the switcher LM for every single token or using token probability as confidence measure and deciding which tokens to switch. Figure 14 demonstrates that switching by patch is more effective.
New Skills, extended We extend the analysis in Figure more datasets while also comparing against baselines, on how much problem previously not solvable with single models now solvable with SWITCH GENERATION: Results Figure 15 show that SWITCH GENERATION better at solving problems that are challenging for single models.
Less Switcher SFT data We try scaling down switcher training by reducing the switcher SFT data to 2.5k, 5k, and 7.5k from our 10k default. Table 16 shows that even though performance minorly drops with fewer SFT data for switcher training, they still better than using the three models individually.
More Model Settings We run SWITCH GENERATION with different model architectures and dif- ferent model sizes: Tables 17 and 18 show consistent improvements.
Longer Generations We set maximum generation of SWITCH GENERATION from 512 to 1024 and 2048. Table 19 shows that SWITCH GENERATION could effectively leverage the increased generation budget.
Smaller Switchers We experiment with smaller switcher LMs. Table effective and outperform using any model individually.
shows that they are
Refinement We compare with refinement setting: taking the response of model and ask the next model to refine it. Table 21 shows that this leads to performance gains, but still underperforms SWITCH GENERATION.
B EXPERIMENT DETAILS
Dataset Details We employ 18+6 diverse datasets spanning multiple LM capability ar- eas for evaluation in this work. All datasets are evaluated with zero-shot prompting. Syco- phancy uses the original prompt in Cheng et al. (2025) and GPT-4o for evaluation: only a re- sponse does not fall into any of the sycophan- tic categories we give a score of 1, otherwise AbstainQA employs abstain accuracy (Feng et al., 2024b) as the evaluation metric. Sci- enceMeter is employed as an NLI task where the passage either supports or refutes the claim. CocoNot (Brahman et al., 2024) uses the regex in the original paper to judge contextual non- compliance. We employ the multiple-choice setting of TruthfulQA. We employ GPT-4o to generate a description of images in GuessBench (Zhu al.. 2025) to transform it into language task. Movie reviews are generated ten times for each movie, five times with the IMDB summary
Size
Dataset
Source
dev
test
Sycophancy AbstainQA
(Cheng al., 2025)
1000
1000
(Feng et al., 2024b)
178
178
Normad
(Rao et al., 2025)
500
2000
ScienceMeter
(Wang et al., 2025a)
1000
1000
MATH
(Hendrycks
2021)
956
956
Human Interests
(Feng et al.,, 2025b)
400
400
AGIEva
(Zhong al., 2024)
1156
CocoNot
(Brahman al., 2024)
1000
1000
TruthfulQ**
(Lin et al., 2022)
200
617
WikiDYK
(Zhang al., 2025)
6849
765
MMLU-pro*
(Wang et al., 2024)
70
1000
BBI
(Suzgun et al., 2023)
1000
1000
PopQA***
(Mallen et al., 2023)
1000
1000
K-Crosswords
(Ding et 2024)
200
1000
GuessBennch
(Zhu et al., 2025)
250
250
Movics
(Wang al., 2023b)
200
200
GSM8K
(Cobbe al., 2021)
200
1000
Alpaca
(Dubois al., 2023)
10000
1000
(Wang 20233)
600
600
H *** indicate the improvement of SWITCH
Table 6: Statistics of employed datasets.
GENERATION against baselines for this dataset is
statistically significant with < 0.1, p < 0.05,

---

Published as a conference paper at ICLR 2026
and five times without, employing RoBERTa- base (Liu et al., 2019) for embeddings, and cal- culating the average pairwise distance. NL- Graph (Wang et al., 2023a) are evaluated with 50% connectivity and 50% shortest path prob- lems. Statistics and statistical significance tests presented Table 6.
Hyperparameter Details We descibe main hyperparameter configurations in Section 3. We run grid search for training the switcher with epoch E {1,2,3, 4, 5, 6, 7, and learning rate I (le 3, 5e 4, 2e 4, le 5e - 5). We then select the switcher f that leads to the best performance on the dev set for evaluation on the test set.
Pretrained, finetuned, and aligned models used individually with lop p 0.7 and 512 maximum generated tokens (same as SWITCH GENER ATION); PROMPT ROUTE employs the aligned model and the following prompt select one of the three models for inference: "You are given an instruction and three candidate models: a pretrained model, a supervised fine-tuned model, and an aligned model. Please select a model that might best fulfill the instruction. Instruc- tion: [instruction] You are given an instruction and three candidate models: a pretrained model, supervised fine-tumed model, and an aligned model. Please select model that might best fulfilt the instruction. Response:"; ROUTELLM employs the "causal LLM classifier" approach in Ong et al. (2025) with the aligned model; TEXT COLLABORATE employs the prompts Feng et al. (2024b); TEXT DEBATE employs the prompts in Du et al. (2023); LOGIT MERGE and PROXY TUNING both employs the implementation
Baseline Details
Liu et al. (2024), with the former approach assigns a weight of I/3 every model and the second (best (best worst)); GREEDY SOUP and DARE-TIES employ the implementation at MergeKit (Goddard et al., 2024).
18

---

Published as a conference paper at ICLR 2026
Instruction
What location in the West Bank has been associated with the temptation of Christ since least the 4th century?
Pretrained
The Church of the Holy Sepulchre.
Finetuned
Jericho
Aligned
The location in the West Bank associated with the temptation of Christ is tradi- tionally identified as the "Place of the Sermon" or "Petrine Monastery" on the top of the Mount of Olives. It is believed that this is where Jesus was tempted by the Devil, according the Gospels of Mathew, Mark, and Luke. The tradition holds that the actual place where the temptation took place was on the nearby rocky terrain known as the Garden of Gethsemane, located just below the Mount of Olives. The Mount of Olives is a rocky hill east of Jerusalem, and the Garden of Gethsemane is where Jesus is traditionally believed to have prayed before his arrest. X
Switch Generation
The location in the West Bank associated with the temptation of Christ since least the 4th century is traditionally considered also known as the Quarantine. According to Christian tradition, it is believed that Jesus was 40 days in the desert, tempted hy the devil, and fasted. This mountain is located in the Judacan Desert, hetween the Dead Sea and Jerusalem. The mountain is approximately 2,700 feet above sea level, and is a popular pilgrimage site for Christians. The site marked by chapel, which dates back the 4th century and was built to commemorate the event. The chapel has undergone various renovations and modifications over the centuries, but the basic structure remains symbol of this significant event in Christian belief. It's important to note that the Mount of Temptation within the West Bank, area that has been point of contention in the ongoing Sraeli-Palestinia conflict. Access and control over religious sites such as the Mount of Temptation have been a source of tension and conffict between the two groups
the Mount of Temptation.
Table 7: Example one of individual model and SWITCH GENERATION responses.
19

---

Published as a conference paper at ICLR 2026
Instruction
Passage 1.At the 2007 American Geophysical Union's 's s.meeting in Acapulco. Mexico, some two dozen..cientists presented multiple studies arguing that a.comet or asteroid exploded above or on the northern.ice cap almost 13,000 years ago showering debris.across the North American continent and caus- ing. tng.teeerrtures plunge for the next millennium. its idea explains multiple.ohservationss not only the climate cooling and the the.disappearance
team argues that
the Clovis hunters. but also see.naar-immllaneuus extine- tion of the continent's 's bogg..mamuals...e all will be convinced. Several lead- ing hypothesess already explain cach of these three events...A change in ocean cir- culation is generally thought to.have brought about the onset of the millennium- long.cooling, which is known the Younger Dryas This.cooling might, in turn, have caused the Cloviss to disappear. And, they had not previ- been killed by disease or hunted to extinction, the big, sig.prehistoric beasts may also have been doomed by thincc in climate..Tho new evidence comes in the form of. geochemical analysis of sedimentary layers at archhhooll cal sites across North America- 99 of. Clovis. Certain features of the lay- ers, say the.team, suggest that they contain dehris formed by an.extratterestraall impact. These include spherules ol.glass and carbon, and amounts of the el- ement.iriiium said to be too high to have originated Earth. In addition. the rocks contain black layers carbonized material, which the team says are the.remains of wildfires that swept actoss the continent.fter the iasact. 2. (...) The team argues that when the quality. and accuracy of the dating-which was based on radiocaron and other techniques
saamined.closelyy only three of the 29 sites actually fall within,the time frame of the Younger Dryas onset, about 12,800 years ago; the rest were probably either carlier.or later by hun- dreds (and in one case, thousands) yf.years.. "Thh supposed Younger Dryas im- pact fails on. theoretical and empirical grounds," says.Meltzer, who adds that the popular appeal of the.hypothesis is probably due the way that it provides. "simple explanations for complex problems." " Thus.. "giant chunks of space debris clobbering the planet.and wiping out life on Earth has undeni- ably broad.appeal, Meltzer says, whereas "no one in Hollywood makes movies" about more nuanced explanations,such as Clovis points disappearing because aarly.Americans turned other forms of stone tool.technology as the large mammals they were hunting.wen went extinct as a result of the changing climate or.hunting pressure.. impact proponents appear unmoved by new study. "We still stand fully hehind the sompact.hypothesii a.confluence of dates," says Richard Firestone, a.nuclear chemist at the Lawrence Berkeley Vational.Laboratoo process," " he contends, adding that the. of Clovis artifacts and mammoth bones just. the claimed iridium, nanodiamond, and.mattetic sphere de- posits is a more reliable indicator.that an extraterrestrial event was responsible for their. disappiisappcaaacce The authors of both passages characterize the impact hy- pothesis A) unsupported by reliable evidence. B) interesting but difficult to conclusively evaluate. C) more appealing to the public than specialists. D) controversial in the scientific community. Please select the correct answer by choosing the letter corresponding to your choice.
which is based on more than
in California. "Radiocarbon dating is a.perilous
Pretrained
3 A B CD7. 7. Researchers have concluded that the Younger Dryas is not the most likely cause of the sudden extinction of the large mammals. A) true B) false C) impossible to conclude D) not yet known Answer: 1 X
Finetuned
'hhe correct answer C) more appealing to the public than to specialists. x
Aligned
The correct answer B) interesting but difficult to conclusively evaluate.
Switch Generation
Both passages characterize the impact hypothesis as controversial within the sci- entific community. Passage I mentions that not all will be convinced by the idea and several leading hypotheses already explain each of the three events: the cli- mate cooling. the disappearance of the Clovis hunters, and the extinction of large mammals. Passage 2 directly states that the Younger Dryas impact hypothesis has been sharply contested by some specialists in relevant fields, who either did not detect such evidence or argued that the deposits had other causes than cos- mic impact The correct answer is D. D) controversial in the scientific community. The correct answer is D) controversial in the scientific community. /
Table 8: Example two of individual model and SWITCH GENERATION responses.
20

---

Published as a conference paper at ICLR 2026
Instruction
What is the sum of the odd integers from 11 through 39, inclusive?
Pretrained
25
What is the sum of the odd numbers from 11 through 49, inclusive? Response:
275
What is the sum of the odd numbers from 11 through 49, inclusive? Response:
275
What is the sum of the odd numbers from 11 through 49, inclusive? Response: 275
Finetuned
The sum of the odd integers from 11 through 39, inclusive, is 250.
Aligned
To solve the problem of finding the sum of the odd integers from through 39, inclusive, follow these steps:
1. Identify the range of numbers to be summed: 11 through 39.
2. Determine which numbers in this range odd. Odd numbers are integers
that not divisible by
3. Sum identified odd numbers.
Let's identify the odd numbers in the range 11 through 39.
11 is odd 13 is odd 15 is odd 17 is odd
19 is odd 21 is odd 23 is odd
25 is even (not odd) 27 is odd 29 is odd 31 is odd 33 odd 35 is even
(not odd) 37 is odd
Now, sum these odd numbers:
17
33 37.
Adding these numbers together:
11 24 24 = 39 39 17= 56 56 - + 23 119
17 146 146 29 175 175 + +3 206 206 +333 = 239 239 + 37 276
So, the sum of the add integers from 11 through 39, inclusive, 276.
The answer is: 276 X
Switch Generation
To find the sum the odd integers from 11 through 39, inclusive, we need to first identify all the odd numbers in this range. Odd numbers are those that are not divisible by 2.
First, list the numbers from 11 to 39, inclusive. Then, identify which of these numbers are odd:
11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33 35, 37, 39
The sum of the odd integers from 11 through 39, inclusive, is 375. We found this sum by adding the numbers in the list, follows:
11 + 13 15 17 19 21 +3
29
35 37 = 375
Table 9: Example three of individual model and SWITCH GENERATION responses.
Setting
PopQA
BBH
CocoNot
qwen 2.5 7B base
21.40
40.20
43.80
qwen 2.5 7B aligned
36.60
48.30
53.20
route 11m
35.90
46.10
42.30
text debate
33.80
53.90
34,20
greedy soup switch generation (ours)
38.10
49.30
45.10
42.40
59.10
62.80
Table 10: Results with another model family, the base and aligned versions of qwen 2.5 7B.
Setting
PopQA
BBH
AGIEval
stacking
29.50
36.90
14.76
weighted voting
27.30
36.40
13.54
MoE
32.80
42.20
17.53
checkpoint ensemble
22.10
31.70
10.94
switch generation (ours)
37.70
58.30
25.26
Table 11: Results with additional model ensemble settings.
21

---

Published as a conference paper at ICLR 2026
Setting
Poem
Pluralism
pretrained
24.55
32.20
finetuned
49.45
58.90
aligned
77.70
50.90
ours, original
70.25
53.20
ours, patch=15
72.15
54,10
ours, patch=10
73.60
$5.80
ours, SFT, patch=10
73.95
55.60
ours, SFT, patch=10
74.55
56.40
Table 12: Pushing the limits datasets with smaller patch sizes and more switcher training data.
Setting
Training
Inference
GPU
20h 1m 48s
4h 2m 5s
2 GPUs
9h 20m 3s
1h 38m 25s
4 GPUs
4h 32m 28s
42m 17s
Table 13: Runtime with 1, 2, and 4 GPUs with default settings and the BBH dataset.
PopQA
BBH
AGIEval
switch every token
31,50
48.10
20.33
contidence-bused switch
34.20
52.30
21.54
switch generation (ours)
37.70
58.30
25.26
Table 14: Results with switching every token.
Method
TranhfulQ
AbstainQAA
BRH
PopQA
AGiEvaa
CocoNot
lest dehote
3.40%
4.80%
7.20%
1.70%
0.95%
0.30%
greedy soup
5.10%
6.30%
5.40%
2.30%
3.63%
5.80%
switch generation (ours)
10.40%
7,90%
13,009
8.20%
6.92%
11.90%
Table 15: Extended results for new skills in Figure 5.
Setting
PopQA
BBH
AGIEval
pretrained
15.30
38.10
4.41
finetuned
26.10
26.70
11.94
aligned
31.20
35.20
11.85
ours, 2.5k
33.00
48.90
21.54
ours, 5k
35.40
52.80
23.27
ours, 7.5k ours, 10k (default)
35,80
54.20
24.22
37.70
58.30
25.26
Table 16: Scaling down the SFT data for switcher training.
Setting
PopQA
BBH
CacoNot
tulu v3 8b base
15.30
38.10
11,90
qwen 2.5 aligned
36.60
48. 30
53.20
generation (ours)
Table 17: Collaboration of different model architectures.
PopQA
BBH
CocoNot
15.30
gemma 3 27b aligned
38.90
51.20
53,60
switch generation (ours)
44.20
56.20
59.80
Table 18: Collaboration of different model sizes.
22

---

Published as a conference paper at ICLR 2026
BBH
NLGraph
512 max tokens 58.30
61.67
1024 max new tokens 59.60
63.50
2048 max new tokens 61.20
64.83
Table 19: Longer generations with SWITCH GENERATION.
Setting
PopQA
BBH
CocoNot
pretrained
15.30
38.10
11.90
finetuned
26.10
26.70
64.00
aligned
31.20
35.20
53.10
qwen 2.5 1.5b
42.10
50.90
68.20
gemma 2 2b
40.70
51.20
68.90
tulu-v3 8b (default)
44.20
56.20
72.80
Table 20: SWITCH GENERATION with smaller switcher LMs.
Setting
PopQA
BBII
AGIEval
base + aligned
30.40
34.90
13.84
aligned + base
29.70
34.20
12.89
base - finetuned + aligned
29.50
36.90
14.76
aligned + finetuned + base
31.20
41.20
15.74
switch generation (ours)
37.70
58.30
25.26
Table 21: Refining model responses with another model.
23


Reviews:
[{"review_id": "0TnT6lZrgv", "reviewer": "Reviewer_yuQM", "summary": "This paper introduces Switch Generation, an inference-time model-collaboration method that trains a small “switcher” LM 𝑓 to decide which model checkpoint (pretrained, finetuned, aligned) should produce the next patch of tokens in a response. During switcher training the authors generate traces by randomly switching among candidate models, roll out 𝑘 continuations for each candidate, score those continuations with task-appropriate metrics, and turn the argmax choice into supervised labels for 𝑓. At inference time 𝑓 is queried every patch and selection is sampled with top-p. Experiments use three-stage Tulu-v3 checkpoints across 18 diverse datasets (QA, reasoning, creativity, safety, etc.), compare to 11 collaboration baselines, ablate patch size and switcher training, show generalization to unseen tasks and model pools, and demonstrate that distilling SWITCH GENERATION outputs back into a single aligned model recovers a substantial fraction of gains.", "questions": "See  Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a clear, novel operationalization of the who speaks when problem that is conceptually simple and practically appealing: reuse existing checkpoints rather than discarding them. The method bridges routing and multi-agent text-level collaboration in a way that is easy to implement (small switcher LM + rollouts) and is evaluated comprehensively: many real tasks, multiple baseline families, ablations (patch size, untuned/random switcher), and analyses. The empirical gains are substantial and well aligned with the paper’s motivation. The distillation result is particularly useful for deployment trade-offs.", "weaknesses": "1. Experiments only use Tulu-v3. The authors can validate SWITCH GENERATION using model families beyond Tulu-v3 to show the approach is not specific to one model.\n2. Because SWITCH GENERATION is conceptually similar to ensembles/routing, include direct comparisons to established ensemble methods (e.g. stacking, weighted voting, MOE and simple ensembling of checkpoints) and clarify where switch generation outperforms or trades off against those baselines.\n3. For tasks where some metrics are worse than the original single-model baselines, please analyze why SWITCH GENERATION degrades performance on those metrics and whether tuning patch size, switcher regularization, or constrained candidate sets can recover or explain the losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T4GbcjAX6I", "reviewer": "Reviewer_ddHQ", "summary": "This paper proposes the SWITCH GENERATION method to address the alignment tax problem in LLM alignment training. The core idea is that each model checkpoint has its advantages at different stages of the training process (pretrained/fine-tuned/aligned). Aligned models have strong inference capabilities, but pre-trained models are better at tasks such as creativity and calibration. The authors train a small switcher LM that dynamically selects which model to use to generate different text segments during generation (segment-level switching).\n\nExperiments validated the method on 18 datasets, outperforming single models on 16/18 tasks, with an average improvement of 12.9% compared to 8 baselines. The method can generalize to unseen model combinations and tasks, and inference costs can be reduced through distillation (although only 57.5% of the gain is recovered).\n\nOverall, this is an interesting task with a clear motivation, but the main problems are high computational costs and weak theoretical foundation.", "questions": "1. **What are the specific inference time, memory usage, and throughput?** This is crucial for evaluating usability. How does it perform with a single GPU/multiple GPUs?\n\n2. Can the patch size be adjusted dynamically? Currently, we need to optimize for each task, which isn't very practical, is it?\n\n3. Why use 8B as the switcher? Have you tried 1B? What was the bottleneck?\n\n4. Can the training cost of 10k×32 rollouts be reduced? Have you tried importance sampling?\n\n5. Where is the complete comparison of token-level switching? Table 2 only shows patch size, not token-level results.\n\n6. Of the 10.7% \"new skills\" in Table 5, how much is a combination effect and how much is randomness? Can they be separated?\n\n7. **How to ensure the security of the base model?** How to prevent harmful content? Can security constraints be added during switcher training?\n\n8. Do long sequences (>512 tokens) still work?\n\n9. Can models of different sizes (1B+7B+70B) collaborate? Wouldn't that be more practical?\n\n10. In practical applications, how do you trade off performance and cost? Is a single model sufficient after distillation, or is multiple models necessary?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Important Problem, Strong Motivation:** Alignment tax is a real problem, supported by Table 1 and related literature. Pre-trained models do perform better on tasks such as WikiDYK and creativity. The idea of \"not discarding pre-trained models\" is quite meaningful given the high cost of training .\n\n2. **Innovative Method Design:** Segment-level switching offers a new granularity option; using LM as a switcher leverages its language understanding capabilities to address the timing of switching, a reasonable design; model-labeled traces help the switcher learn the strengths of each model; and rollout evaluates the average performance of each choice.\n\n3. **The experimental design is reasonable and comprehensive:** The 18 datasets are representative and cover different types (knowledge/reasoning/creativity/security); the 8 baselines cover different collaboration methods (API/text/logit/weight level); ablation experiments (patch size, training strategy) were conducted; the supplementary materials include statistical significance tests; and the generalization ability was also verified.\n\n4. **Experimental results are good:** Significant improvements were observed on most tasks, outperforming single models on tasks 16 and 18, with an average improvement of 12.9% compared to the baseline.\n\n5. **Strong generalization ability:** It can generalize to different model families (Qwen2.5), different numbers of models (2/4), specialized experts, and unseen tasks.\n\n6. **Clear Writing:** Figure 1 effectively illustrates the motivation (the pretrained model does indeed perform better on some tasks), and Figure 2 fully demonstrates the methodology. Table 1 uses color to distinguish task types in the main results , facilitating understanding. The case studies (Tables 7-9) are compelling.\n\n7. **High reproducibility:** We promise open-source code, data, and the switcher model. Appendix B contains detailed experimental information.", "weaknesses": "1. **High computational cost is the biggest problem.**\nInference requires n+1 models in memory (4 times the memory), and the switcher needs to be called every 50 tokens. Distillation reduces costs but only restores 57.5% of the gain. More importantly, the paper does not provide actual runtime and throughput comparisons, which are crucial for practical deployment. Hyperparameters are sensitive, and the patch size needs to be optimized for the specific task.\n\n2. **Weak theoretical foundation**\nWhy is segment-level analysis superior to token-level or response-level analysis? There is a lack of theoretical explanation; the ablation in Table 2 is insufficient.\nWhy is tracing helpful? The R² in Figure 5 is 0.017, which shows almost no correlation.\n- The definition of \"new skills\" (10.7%) in Table 5 is not rigorous enough and may be partly due to randomness.\nHow does the Switcher learn to recognize different skills? Figure 6 is only descriptive.\n- The lack of in-depth analysis of these issues limits further improvement of the methodology.\n\n3. **The difference between this method and routing methods is not significant enough.**\nThe essential difference lies mainly in the granularity and the use of traces; it is not a revolutionary breakthrough.\n\n4. **Soundness-related issues**\nTraining the Switcher requires 10k samples × 32 rollouts (320k calls); sample efficiency was not discussed.\n- Patch sizes vary greatly across different tasks (Table 2 shows 10-100), but there is no research on adaptive selection.\n- The rule that the beginning and end must use aligned models is hard-coded and not elegant enough.\n\n5. **Insufficient experimental analysis**\n- Table 1 in the main text does not indicate statistical significance (although the supplementary materials do).\n- Some datasets choose to be based on hypotheses rather than empirical evidence.\n- Different evaluation indicators are not uniform\n- On which tasks is switch generation inferior to single-model or baseline approaches? Why? These analyses are missing.\n\n6. **Security Issues**\nThe Ethics Statement mentions that a misaligned base model may circumvent safety guardrails, but it lacks empirical assessment of the risks associated with harmful content generation. This is a serious concern.\n\n7. **Generalization is still questionable.**\n- Table 3 shows a significant decrease in generalization performance (setting 1: 5.8% → setting 4: 3.1%).\nCan models with different architectures collaborate? For example, can GPT and LLaMA be combined ?\nHow did it perform with long text (>512 tokens)? Not evaluated.\n\n8. **Minor Writing Issues**\nThe definitions of pretrained/finetuned/aligned in footnote 1 appeared relatively late. The R² of Figure 5 is 0.017, indicating extremely weak correlation; the statement of \"positive correlation\" in the text is overly optimistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qo45biVDLr", "reviewer": "Reviewer_x8fX", "summary": "This paper proposes SWITCH GENERATION, a collaborative inference framework that dynamically combines multiple model checkpoints (e.g., pretrained and aligned LMs) during response generation. Recognizing that alignment improves instruction following but harms creativity and calibration, the method uses a trained “switcher” LM to select, at each segment of the output, the model best suited for the current context. Evaluated on 18 tasks, it consistently outperforms individual models and existing collaboration baselines, demonstrating strong generalization, compositional reasoning, and efficient reuse of models typically discarded in training pipelines.", "questions": "(1)Why not compare against a “refinement” baseline?\nThe paper should include a baseline where a strong pretrained (or larger) model processes or refines the output of an aligned/SFT model (e.g., by editing, re-ranking, or validating segments). This would help clarify whether the gains come from dynamic switching itself or simply from leveraging base-model capabilities in a post-hoc manner.\n\n\n(2)Would the approach generalize to heterogeneous model scales or architectures?\nThe current evaluation assumes homogeneous model families and sizes. It remains unclear whether SWITCHGENERATION would work effectively when combining, say, a 70B base model with a 7B SFT model—or models from different families (e.g., Llama + Mistral). Have the authors tested or considered such settings?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1)Novel dynamic collaboration mechanism: Introduces fine-grained, trace-aware switching where models “take turns” generating response segments—more flexible than static ensembling or single-model inference.\n\n(2)Effective and generalizable switcher design: The switcher LM is trained via supervised fine-tuning on rollout-based utility signals and generalizes well to unseen tasks and model combinations.", "weaknesses": "(1)The core idea of dynamically switching between models is not entirely new—it resembles prior work in other areas such as cloud-edge collaboration (e.g., ADASWITCH [1]) and LLM evaluation frameworks (e.g., Slide [2]). The paper does not sufficiently differentiate itself or provide deep mechanistic insights into why and when switching between base and aligned models works.\n\n(2)The evaluation only uses same-sized (8B) checkpoints from the same model family (Tulu-v3). This fails to reflect practical scenarios where a large pretrained base model collaborates with a much smaller instruction-tuned model—a more common and resource-efficient deployment pattern.\n\n(3)The paper does not compare against a simple but strong alternative: letting a capable base model refine or post-process the output of an aligned/SFT model. Such a baseline would better isolate the benefit of dynamic segment-level switching versus static post-hoc correction.\n\nReferences:\n[1]ADASWITCH: Adaptive Switching between Small and Large Agents for Effective Cloud-Local Collaborative Learning\n[2]Slide: A framework integrating small and large language models for open-domain dialogues evaluation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9PuxNts3Sm", "reviewer": "Reviewer_WS2y", "summary": "The paper proposes to use a trained model selector to select from an aligned model and its pretrained version. The selection happens every few tokens, so the final response combines the capability of both models. The method improves over any single model, and is capable to answer some questions that both models cannot. Ablation study shows that the method can also be generalized to different models (instead of different versions of a single model).", "questions": "1. In the main experiment, has any of the baseline methods finetuned on these tasks (i.e., is the comparison with switch-g fair)? Has any of the baseline methods finetuned specifically for each of these tasks (i.e., is the comparison with switch-t fair)?\n2. You mentioned that the switcher is a \"small\" model in section 2, but it's actually the same size of the pretrained and aligned models (8B), is there a reason for this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel and provides valuable insights.\n2. The paper provides thorough analysis of the results.", "weaknesses": "1. Some parts of the writing are unclear, e.g., in Table 3, we don't know the name of each model.\n2. The size of the model selector seems to be the same as the pretrained and aligned models, which raises efficiency concerns. It would be nice to see either the result of a smaller selector model or an efficiency analysis.\n3. Some experiment settings are unclear, and they might impact the fairness of the evaluation. See question 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)