EMFuse: Energy-based Model Fusion for Decision Making
Authors: Kejie He
Yi-Chen Li
Yang Yu
Venue: ICLR 2026 Poster (2026)
Keywords: Model Fusion
Energy-Based Model
Decision Making
Primary area: reinforcement learning

Abstract: Model fusion has emerged as a promising research direction, offering a resource-efficient paradigm that leverages existing pre-trained models to circumvent the need for training from scratch. In this work, we investigate the fusion of models specifically adapted for decision-making tasks. This challenge divides into two distinct, yet related subproblems: the direct fusion of models that act as policy and the fusion of dynamics models that subsequently induce a policy. We suggest that these seemingly divergent subproblems can be unified through the lens of energy-based models (EBMs), which parameterizes a conditional distribution via an energy function where lower energy implies higher probability. Our framework, \textbf{EMFuse}, provides this convergence by leveraging the concept of energy as a common currency for fusion. For direct fusion of policies, such as those in language models, the output distribution is commonly softmax (Boltzmann), which essentially defines the negative logarithmic probability as an energy function. For dynamics models, existing works often train a set of models on the same dataset to obtain robust uncertainty estimation; such an ensemble approach leads to an exponential explosion in computational complexity when it comes to dynamics fusion across multiple sets of models. To overcome this, we introduce the Any-step Dynamics Energy-based Transition Model (ADETM), a novel architecture that performs efficient single-model-per-dataset uncertainty estimation with its energy-based backbone, thereby avoiding this computational explosion. Our EMFuse framework surpasses other baselines by 0.34\% to 6.63\% on single/cross domain discrete decision-making benchmarks, and achieved an extra 2.3 to 7.4 normalized points on average in D4RL MuJoCo continuous-control scenarios. Our code is available at https://github.com/LAMDA-RL/EMFuse.

Published as a conference paper at ICLR 2026
EMFUSE: ENERGY-BASED MODEL FUSION FOR DECI- SION MAKING
Kejie ; Yi-Chen Li!;
Yang Yu
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China & School of Artificial Intelligence, Nanjing University, Nanjing, China fheki, liycaelamda........m.aa.aa.amiia amdia n ju. edu. en
yuyõ @andaaa nju.. edu.cn
ABSTRACT
Model fusion has emerged promising research direction, offering resource- efficient paradigm that leverages existing pre-trained models to circumvent the need for training from scratch. this work, we investigate the fusion of models specifically adapted for decision-making tasks. This challenge divides into two distinct, yet related subproblems: the direct fusion of models that act as policy and the fusion of dynamics models that subsequently induce a policy. We sug- gest that these seemingly divergent subproblems can be unified through the lens of energy- models (EBMs), which parameterize conditional distribution via an energy function where lower energy implies higher probability. Our frame- work, EMFuse, provides this convergence by leveraging the concept of energy as a common currency for fusion. For direct fusion of policies, such as those in language models, the output distribution is commonly softmax (Boltzmann), which essentially defines the negative logarithmic probability as an energy func- tion. For dynamics models, existing works often train of models on the same dataset to obtain robust uncertainty estimation; such an ensemble approach leads to an exponential explosion in computational complexity when comes to dy- namics fusion across multiple sets of models. To overcome this, we introduce the Any-step Dynamics Energy-based Transition Model (ADETM), a novel ar- chitecture that performs efficient single-moddelperr-aaaases uncertainty estimation with its energy-based backbone, thereby avoiding this computational explosion. Our EMFuse framework surpasses other baselines by 0.34% to 6.63% on sin- gle/cross domain discrete deciisionmmaiing benchmarks, and achieved an extra 2.3 to 7.4 normalized points average in D4RL MuJoCo continuous control scenar- hwt.gittaa....om..aeeeoeaee LAMDAA RL. EMFuse
ios. Our code is available at
INTRODUCTION
Modern deep learning ecosystems are increasingly populated by highly specialized models that power tasks from text generation to complex problem-solving (Altol 2024 et Directly training new monolithic models is expensive, while discarding existing expertise wasteful. Model fusion promises a resource-efficient alternative by combining pre-trained experts into stronger sys- tem (Wortsman et al. 2022, Matena Raffel 2022; Hu al. |02 Lu al., 2024). However, while model fusion has seen rapid growth, its application to the specialized domain of decision- making remains comparatively underexplored. Recognizing the immense potential in this area, our work focuses specifically on fussio for decision making. a critical frontier for creating more capable and adaptable intelligent agents from fusion (Levine et al. 2020 Moerland et al. 2023).
At high level, the behavior of a decision making agent is governed by either a directly learned policy or a policy derived from a learned dynamics model. This fundamental architectural choice presents two distinct points of intervention for model fusion: (1) Direct policy fusion: This approach
"Equal Contribution
1 "orresponning Author

---

Published as a conference paper at ICLR 2026
involves combining the output distributions of multiple policies the point of decision, potentially deriving a single, more robust action by taking all distributions into consideration. (2) Dynamics
fusion: In model-based RL (MBRL), dynamics models learned from offline logs (Levine et al.
2020) enable further policy training (Chua et al.] 2018: Janner et al. 2019; Yu et al.) 2020] Hafner al. 2020). The dynamies fusion approach acts dynamics model (their predictive understanding of the environment's 's dynamics). The goal is to construct a single, more comprehensive and reliable simulation of the transitions by unifying each model's 's knowledge of the world. A superior policy could be learned under this unification, which provides a richer and more accurate basis for decision-makes
foundational level by merging the agent's
These seemingly divergent subproblems can be unified. We take the perspective of Energy- Based Models (EBMs), where distributions are represented through energies whose additive composition corresponds to multiplicative densities (LeCun et al. 2006 Hinton 2002). In this view, both policy outputs and dynamies likelihoods can be written as ply x) = exp( (xr, (r). and fusion becomes energy sum Efuse >; E, This makes policy fusion and dynamics fusion two instances of the same principle, differing only by the choice of ar, y and the sampler.
Our framework, EMFuue, applies enegyy-additive fusion to two key settings: (1) direct policy fusion and (2) dynamics model fusion. This approach enables efficient and effective inference without re- quiring additional training of the full models. We observe that the fused energy distribution remains close to each constituent policy on its specific domain, as measured by small KL-divergence. This faithfulness makes the fused energy powerful indicator for selecting the best policy for given context, a principle we formalize in our EMSelect algorithm.
Most existing dynamics model learning work trains an ensemble of models for uncertainty estima- tion YYu et al. 2020). However, fusing ensembles of dynamics models quickly becomes intractable due exponential blow-up. Inspired by ADMPO (Lin et al., 2025) that enables uncertainty es- timation on one model by its ability to predict multiple next state with flexible action sequence input, we therefore introduce the Any-step Dynamics Energy-based Transition Model (ADETM), an energy-based dynamics model that performs pogle-model--er-daae retaining multi-step context. Additional state and action encoders are added make ADETM avail- able to the energy-based context. This setup avoids the combinatorial cost of cross-ensemble fusion.
uncertainty estimation while
Our main contributions are listed as follows:
EMF use: We formalize policy and dynamics fusion under the same energy framework Euse -; E, connecting classical PoE (Hinton 2002) decision- decision-making settings.
EMSelect: An EMFuse-based policy selection framework that achieved additional 1.18% 1.31 % gain top of EMFuse.
ADETM for tractable dynamics fusion: A single mongle-model-pee model-per-datasee energy- world model with sny-step context that circumvents cross-ensemble explosion while preserving uncertainty-aware behavior.
Empirical gains: On single/cross-domaii discrete decision-making benchmarks, EMFuse improves accuracy by 0.34 1344%-663%% % and on D4RL MuJoCo continuous control (Fu et al. 2020 it adds +2.3 to +7.4 normalized points on average over other fusion baselines.
2 PRELIMINARIES
2.1 TOWARDS ENERGY-BASED MODELS FOR DECISION MAKING
Markoy Decision Process
We consider discounted MDP M A,r., with an offline [ss a, r, s')} collected by unknown behavior policy Th (Puterman 1994 Sutton & Barto 2018 Levine et al. 2020). Offline training faces a support gap: test-time states/actions may lie outside the empirical support of DD causing distribution shift and value overestimation. This makes calibrated uncertainty and support awareness central to algorithm design.
dataset D
Behavior modeling: explicit vs. implicit One axis models or constrains by the behavior distri- bution. The Explicif approaches fit an estimate a 8) and use them prior regularizer for
2

---

Published as a conference paper at ICLR 2026
policy learning (Fujimoto et al. 2019
Kostrikov et al. 2022) Implicit approaches shape learning
objectives to bias solutions toward the dataset support, e.g., conservative value learning and pes- simistic objectives (Kumar et al., 2020
Levine et al. 2020). Both aim to remain out-of-distribution (OoD))robust; they differ in how support is represented (density vs. objective penalties).
Dynamics Models and Uncertainty Estimation. A second axis learns a model of the dynamics and plans or trains policies inside it. Classical likelihood models parameterize Pa(s' s, a, with Gaussian or mixture families; latent/variational world models compress trajectories into learned state abstractions (e.g.. World Models, PlaNet, Dreamer) (Ha & Schmidhuber 2018 Hafner et al.) 2019 2020). Recent work also explores diffusion//core-buee based generative modeling for sequential control (Janerr et al., 2022). Likelihood models require calibration under aff-distribution conditions; consequently, robust uncertainty estimation becomes key.
Epistemic uncertainty is commonly estimated through bootstrapped or independently initialized model ensembles; aleatoric uncertainty is captured by predictive dispersion (Kendall Gal. 2017] Chua et al.) 2018). ADMPO (Lin et al., 2025) trains single recurrent net variable-length tran- sitions, achieving ensemble-like robustness without model repetition, further simplifying the frame- work. Ensembles not only improve robustness, they also provide composable uncertainty signals that will align naturally with an energy-basee -assed view.
Energy Based Models (EBMs) EBMs represent distributions via unnormalized energies, (o(x) x exp(
Eg (x)), trained with contrastive or score-based criteria (Hinton 2002 Hyvärinen,
2005 LeCun et al., 2006). Energies add linearly when combining independent experts, correspond- ing to multiplicative composition of their unnormalized densities. Specializing to dynamics, Energy- based Transition Models (ETMs) (Chen al. 2024) learn transition energy Eg Ea a, s') whose negative exponent defines a next-state distribution,
po(s' 88 a) x exp(- a, a,,'));
enabling support-aware modeling and contrastive learning of transitions. Behavior priors also admit an energy view, E5 a) = - log (a 88. Given expert energies (Ei), polluct-of-Experes fusion corresponds to energy sum
Eruse (x) = Σ Nv Ex),,
Litaso Pruse (x) x exp( - (z)).
(2)
which can combine multiple transition experts (ensembles, domains) and behavior priors in single, support-conscious objective (Hinton 2002, LeCun al., (2066). This additive structure underpins the tractable fusion rules we use later for both policy and dynamics.
Policy learning ETMs or fused ETMs can be used to generate rollouts for policy learning der pessimissm/regulrzzztioo (Rubinstein & Kroese 2004 Chua et al. 2018 Feinberg et al.) 2018 Buckman et al. 2018; Kumar et al. 2020). This closes the loop in offline MBRL: behavior pri- ors constrain actions, world models predict consequences with quantified uncertainty, and additive energies provide a principled path to compose experts while remaining support-aware.
2.2 BOLTZMANN OUTPUT AS ENERGIES
Autoregressive policies, such as those in modern LLMs, can be viewed directly through an energy- based lens. The key prerequisite for fusing such policies that they must operate over shared, finite vocabulary V. a condition typically met by using common tokenizer (Vaswani et al., 2017] Brown et al. 2020 Hoffmann et al. 2022 Touvron et 2023). This shared support makes their output distributions directly comparable and fusible.
At each step I, the model maps a context Zgg to a vector of logits E RRM These logits define a normalized next-token policy via the softmax function with temperature T:
(x)(=
(o ( = moxt ( Eyev (xx=xxv)))))
Trained via maximum likelihood, this is the canonical estimator for conditional token probabilities [Bishop] 2006)] Goodffllow et al, 2016). Critically, it is equivalent to a Boltzmann(softman dis- tribution with energy Fo(x =y/)// This direct equivalence- where pogprobaasilities
3

---

Published as a conference paper at ICLR 2026
scaled negative energies is the cornerstone that allows to apply energy-addiin fusion framework to LLM policies.
3 ENERGY-BASED MODEL FUSION FOR DECISION MAKING
In this section, we will mainly discuss applications of our Energy- -ased Model Fusion for Decision Making (EMFuse) framework to direct policy fusion and dynamics fusion. A subsequent applica- tion of EMFuse in policy selection (EMSelect) will be mentioned and the supporting architecture that enables dynamics training (ADETM) will be introduced.
The property of energy additivity forms the core of our fusion framework (Hinton 2002). which we now define formally. Let (2r, y) ) be the energy of the policy model / dynamic model that defines normalized conditionals Pı(y x) = exp( E, (z,,))) (Z,)) on shared support. For
nonnegative weights X with E; X = 1, define the fused energy
Eruse (2, 1)
Σ A V.
(4)
Then the fused distribution is
exp( Etuse (I)
Pfuse (y x)) Zruse (1))
II p:(y x)*;
(5)
i... logarithmic opinion pool (geometric mixture, or LogOP), which is the unique minimizer
of the weighted reverse-KL. KL projection arg min, E, (g(l(g (Heskes) 1998 Genest & Zidek, 1986). This rule is application-agnostee policies (e.g. LLMs; via Eq. and (ii) dynamics models (e.g. ETMs; Eq. 1
subsequent subsections instantiate E, for (1) energy- based
3.1 DIRECT POLICY FUSION
By taking energy- based perspective, many recent diciisionmmaking models can be written as the conditional distribution: Pe x) = exp(- (z, /)) Whenever experts expose such normalized policies on shared support, EMFuse fuses them by adding energies (equivalently, multiplying their distributions): Efuse = Σ, E > Ptuse x) x II, x) z)^;, which is precisely the logarithmic opinion pool (LogOP) optimizer of weighted reverse-KL projection
and coincides with a Product-of-Experts (PoE) in probability space (Heskes 1998 Genest & Zidek
1986 Hinton 2002 LeCun et 2006).
We extend this to real case-study by taking modern LLMs as an energy-based policy example. Contemporary LLMs are decoder-only Transformers trained with next-token prediction (Vaswani
ct al. 2017: Brown et al., 2020 Holfmann et al., 2022, Touvron et al. 2023). By equation 3 at time f, each expert M; induces energies E, (Zige y) (U))/T over the shared vocabulary V, so
(: Zgg) are (negative) energies up to normalizer.
training makes Eq. the canonical estimator of conditional token probabilities [Bishop] 2006 Goodfellow et al. 2016). Equivalently, this is a Boltzmann distribution with energy (o y) 2e (T)/T, so log Po is (negative) energy up to a normalization constant exxstatt-xxcclly the form required for endrgy-additive fusion.
Let experts {M, share (or be mapped to) the same tokenizer, hence the same support V. Each produces p: (y xg()) softmax(z:/T).. EMFuse defines the fused policy Pruse the LogOP
solution
Pruse (: [g = arg min
X KL(g])p:( (gge)) Pruse (y) xgge) or II P:(w x,
GEA(V)
(6)
with nonnegative weights X; summing Writing p; exp(-E,) yields energy additivity Efuse = Σ; A,E, i.c., multiplicative densities with additive energies (Hinton 2002 LeCun et al.
2006
Check Algorithm for details, experiments w.r.t. LIM settings will be discussed in
4

---

Published as a conference paper at ICLR 2026
Algorithm IEMFuse-Policy (Case study on LLM)
1: Input: context ggg experts Mini per-expert temperatures {Ty}:
weights {A:} (A, 20, E; A =);; optional decoding (temperature/oop-kkoopp)
2: For i = 1..mt compute logits 2; Mo.logils( log-probs & logsoftmax(z//7)
3: Fuse in log-space: (uuee < >i= i= 7, 4
4: Normalize: Pfuse f- exp exp(efuse - logsumexp( (ftsse)
5: Decode: sample/argmax Yt ~ Ptuse( zg(): append Yt to context.
6: Output: fused next-token distribution (and the generated token if decoding).
The following discussions should be noticed: fusion operates over an identical support V; this avoids vocabulary mismatch and makes Eq. well-effiee.. In practice, this condition can be satis- fied either by sharing tokenizer by mapping heterogeneous vocabularies into common space. Recent work on vocabulary mapping and tokenizer transplantation (Sennrich et al., 2016; Kudo & Richardson, 2018; Mavromatis et al., 2024; Xu al., 2024; Goddard & Neto, 2025) shows that such projections can be implemented in a training-free or lightly fine-tuned way. While a detailed engi- neering treatment is beyond our scope, EMFuse is designed to operate downstream of these mapping layers: as long as the mapping preserves semantic monotonicity (i.c., avoids severe semantic drift), the PoE structure in Eq. remains valid and empirically robust. Crucially, the additive energy struc- ture (Eruse = Σ; E,) provides inherent resilience to mapping artifacts. If vocabulary projection introduces uncertainty (scale distortion), it naturally manifests as higher-entropy (flatter) energy surface. In Product-of-Experts these flattened distributions are automatically down-weighted in favor of the sharper, more confident energies from native experts, effectively filtering out map- ping noise without manual intervention. For weights, X; can be static (uniform/held-out tuned) or context-adaptive (e.g., entropy-ave aware): mild per-expert temperatures т; adjust sharpness without changing the EMFuse form (Guo et al. 2017]] Hinton et al. (015)).
In practice, we are applying uniform weights, and ablation studies on catropy-based weight showed statistically insignificant results, which will be discussed in the Appendix $ Apply decoding heuristics (temperature, top-k, top-p) post-process Pfuse but do not alter its normalized form (Holtz- man al. 2020). We observed minimal, statistically insignificant gains from adding Laplace smoothing of per-expert distributions (to prevent multiplicative collapse when policy assigns zero probability viable token); details and ablations in Appendix SE.T From the LogOP perspec- tive, EMFuse acts as a conservative consensus or "AND-gate" over experts: tokens that any expert assigns very low probability are exponentially suppressed in the fused distribution. As a result, single miscalibrated expert cannot destabilize the decision rule. This stability guard is verified in the distribution faithfulness experiment in Table and further increases our EMSelect design in the following discussion.
3.2 DIRECT POLICY FUSION AN ALTERNATIVE
Empirically, when ex- perts are spenainspeciillzed. this consensus tends lie close (in KL) the expert whose domain dominates the current context. This suggests using EMFuse as reference to decide which expert should act each decoding step, which the heart of our EMSelect algorithm.
EMFuse provides a consensus distribution by additive energies (Eq, 45
Two-expert derivation. Consider two experts and with token policies p.(- agg)) and py(: | Jg)). Instantiate EMFuse on this pair with weights [0, and 1 (defaults to a = f):
Pras(: (gg)) x pa(- (gge) ps(: xgge)=
Select the expert whose policy closer (smaller KL) to this pairwise fused reference:
choose iff KL(pies px)
KL(pie) ||p;).
Since (T.(plgg) (logp log 4). the entropy term cancels and the decision reduces to E Epig, [:] 2 Epeas Dogp;)
i.e., pick the expert with higher expected log- liekkleeioodd under the pairwise EMFuse reference.
5

---

Published as a conference paper at ICLR 2026
EMSelect (tournament over experts). For n 2, apply the two-expert selector sequentially in a lightweight tournament without any initial seeding: (i) fix deterministic order over experts (c.g., their indices), (ii) set the incumbent the first expert in that order, (iii) compare the incumbent against the next expert using the two-expert rule, (iv) keep the winner as the new incumbent, and (v) continue until all experts have heen compared once.
Algorithm 2 EMSelect: KL-guided per-step policy choice from EMFuse
Input: context zgg experts Mini pairwise weight E [0, 1] (default a f))
+ 2: Expert log-probs: for all i, compute & + lo (- zgg)) (via Iog -oggsaa)
3: Initialize incumbent: let the comparison order be (1, .2.....)........ n); set 1" <-
4: for j 2 to do
5: Pairwise EMFuse: li +- a. + (1-a) Piraj t softmax((f-0))
6: Score experts: S(i*) 4 E, Pi (v) 4. (v);
S()) + Cg Pr (1)))r)))))))) (
7: Advance winner: 8 arg maxkeli" J} S(k)
8: end for
9: Decode: use P:- (- sige) to sample/argmax yr append context.
10: Output: selected expert i" and per-step policy pi..
EMSelect complements EMFuse. EMFuse aggregates all experts into a conservative consensus, while EMSelect commits each decoding step the expert that best preserves that consensus lo- cally. In domains with strong specialization and mild calibration differences, this targeted commit- ment yields the additional accuracy reported in our benchmarks. Theoretical guarantee. Let y1:T denote generated token sequence of length T. For each expert i, let Pi(- (gg)) be its conditional distribution given context d<z (which includes prompt Zp and history y<t)), and let Ptuse (xse) be the EMFuse consensus. EMSelect induces a per-step distribution Psei xzge) = Pi, (- zg() by selecting an index i, at each step, and we denote the resulting sequence distributions by (sal()1:)) and Piuse (VI.T). For conciseness, define the worst-case local expert-to-fuse divergence
mmax (zge) max KL xgg)) Péuse (- =g())
Appendix E.4 shows, via the KL chain rule for autoregressive policies, that A maxx (x=()].
T
KL Pluse)
E Eycerpanl
Empirically, all expert distributions remain very close to the fusion mean (per-step divergences sat- isfy KL < 0.09, see Table 2), so the upper bound in Eq. 7 tight in setting. Thus, EMSelect mathematically "leashed" to the conservative EMFuse geometry: it can exploit sharper experts locally, yet any sequence- level deviation is constrained by the shared consensus.
3.3 DYNAMICS MODEL FUSION
Recall the general EMFuse rule (Eqs. 15: expert energies add and their (unnormalized) densi- ties multiply. Specializing to Energy-based Transition Models (ETMs; Eq. D. of expert transitions {E,(s, a, s.) ! defines the fused transition
Eruss (s)
Σ A a's,
Pruse
s))a) x exp( Eruse (s, a, s s')),
(8)
with nonnegative weights A; (we use A; = by default), assuming shared state/action spaces and possibly different training datasets/domains per expert. In continuous state spaces, normalization is unnecessary for ranking or sampling procedures that operate in energy/log-space.
ADETM: sty-step ETM enabling single-model uncertainty. We emplo an Any-step Dynamics Energy-based Transition Model (ADETM) to retain ETM's training recipe while equipping each expert with robust, ensemble-like uncertainty at single-model cost. This is inspired by the ADMPO [Li et al. 2025) framework. Concretely, ADETM wraps the ETM energy network with (i) an ML.P state encoder and (ii) multi-heaa-attention action-sequence encoder operating fixed

---

Published as a conference paper at ICLR 2026
Algorithm 3 EMMFuse-Dnaamic (one step)
1: Input: current state 8;; action or; expert ETM energies {E:].= fusion weights = ni
2: Per-expert energies: compute (s,, (t, for all (on candidate s' or as an energy field).
3: Fuse in energy space: Ejsse (st, Qt, ) f Σi=1 A Ei(s, a,,
i==
4: Sample/score next state: draw 8+41 Pfuse (- St, az) via Langevin dynamies on Etsee (as in
ETM), or take a MAP estimate St+1 = arg ming' (fuse(St, at, s if sampling is not used.
5: Output: 81+1 (optionally, diagnostics derived from Efuse).
State
Kevisel Energe beoee Thanstion Nock
Srque Actios Seggarce.
Irpet Staee
Ssae Ineeder (Mil))
Beloeiie Staee
Fredetel Stae
Tload Lengtt Soqarnce (yri))
2
sall
Food Lan feg
Seiondee Aaton
Podlingg
Adtin
Villi 14)
Lengh Soy
feect Action Segazxee Bquu Stiee
Pond Lnogt Sugarce Manta (on
M--1
Freducted Nec: State
FeR
Tesrehie
Cakelate
pul!
Fue
Zue
Au!f
na!!
Palline
Void Leaph Sog (4.222)
Figure I: Any-step Dynamics Energy-based Transition Model for dynamics fusion 3.3 step next-state prediction; bottom: stacked-input variant for parallel multi-step prediction.
Top: single-
history window & with a valid-length mask. The encoders produce joint embedding [he ha] that conditions the ETM energy Eg(s, a.-k,, s'). This preserves the ETM module and its training dynamics (contrastive / InfoNCE objective, Langevin sampling for training and diagnostics), while allowing ADETM to exploit short-horizon action context without duplicating models. Figure illustrates the architecture; full encoder details and hyperparameters are deferred to the appendix.
Uncertainty without ensembles via stacked histories. ADETM yields practical history- sensitivity regularizer by stacking feasible history slices from a FIFO queue of the last k state-action pairs and measuring dispersion across the resulting next-state predictions. Let the queue contain (8:-4, 04-4,,.)))...................... .).... (8t, az). For the same target step 1+1, construct up to k valid history slices
(8-44, 4, 0t 4:-4:) 8411: (81-3, 0t 4x)3)) 8
(st, az) + +8++1
(k)
Define the uncertainty score the sample variance (or mean squured dispersion) across predictions,
k Σ m==
k
un ( a.)
= K 2+1
stt1 1|
8et1
k
2+1
m==1
This dispersion behaves analogously to ensemble disagreement but requires only one trained ADETM per expert, effectively solving the exponential explosion when fusing ensembled mod- els. Consequently, EMFuse rollouts with ADETM scale with the number of experts rather than the ensemble size, yielding a lightweight dynamics module in terms of parameters, FLOPs, and roll- out latency; we provide quantitative analysis in Appendix B.5 and ablate the resulting uncertainty signal against resk-neutral and local-sensitivity baselines in Appendix B.6 The fused transition (s s, a) and ADETM's uncertainty are then utilized within an offline RL loop to generate model rollouts and train a policy; we defer training protocols and all evaluations to the Experiments section and the appendix.
7

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EVALUATION OVERVIEW
We evaluate EMFuse on two complementary LLM families and on model-based offline con- trol. Family L (Llama, Touvron et al. (2023) is used assess mechanistic fidelity via length- controlled preference ($C.) and token-level KL to domain experts ($C.)). Family Q (Qwen Yang et al. (024);; OpenCompase Contributors (2023) is used assess task accuracy fi- sanee/mathennttcc/mmciccaa
under the standard OpenCompass protocol ($C.2). This design high-
lights model-aannstte benefits (accuracy under external evaluator; Family Q) while establishing
distributional faithfulness under controlled tokenization//raiiing (Family L.).
4.2 RESEARCH QUESTIONS
RQ1 (Policy fusion effectiveness). Does EMFuse improve decision performance over parameter-
space and regularized merging baselines? Metrics: OpenCompass accuracy (Family Q: Table 20 AlpacaEval length-controlled win rate (Family L,; Tables 16 177.
RQ2 (Faithful distribution). Does EMFuse preserve each expert's 's distribution its domain? under teacher forcing (Family L; Table 18
Metric: token-level DKL, (1Bexpere ||
RQ3 (Selection fusion). Does EMSelect (KL-guided tournament) provide additional gains over EMFuse? Metrics: same as RQI (deltas over EMFuse; see $3.2 and aggregated gains reported).
RQ4 (Dynamics fusion). Does EMFuse with ADETM improve offline RL returns versus dynamics- level baselines? Metric: D4RL normalized return (IQM) with BCa 95% CIs (Table
4.3 MODELS, DATA, AND TASKS
Family L (Llama). We train 1B SFT experts on language (harmless/helpful) and subject (agricul- tree/medicatiio/pplloooppy splits; 8B counterparts follow FLOPs-matched budget (Eq. [0). Full training and generation settings are in $B.2 Family Q (Qwen). We SFT mixed-subject experts dinancemmattennaiissmeeecee
and specific-subject (within finance, different categories) $B.1
Evaluation is done with OpenCompass defaults C.2). Dynamics. ADETM ($3.3 is trained on D4RL-v2; rollout and SAC details are in (B.4
4.4 BASELINES
We compare three representative training-free free baselines spanning the main fusion axes:
Uniform Model Soup (Wortsman al. 2022): data-less, parameter-space merger widely adopted in practice; it requires no access to the original training data and provides reference point for weight-space averaging.
RegMean (Nguyen et al. 2025): training-free (and data-less) regularized weight merging method that aligns statistics without task data; this family is commonly identified as a canonical training-free approach in recent overviews.
PackLLM (Mavromatis et al. 2024): training-free, logit-space policy fusion method inference time. PackLLM's pairwise packing informed our EMSelect tournament design ($..)) This baseline exclusive the policy fusion.
Together these baselines test EMFuse against: (i) parameter-space averaging (Soup), (ii) trainless weight alignment (RegMean), and (iii) output/logit-space fusion (PackLLM).
4.5 EVALUATION PROTOCOLS
Family L (mechanistic fidelity). We use AlpacaEval with length control to mitigate verbosity bias (SC.3 and measure token-level DKL from each BBxxpet EMFuse and its 8B counterpart under a shared tokenizer ($C.4). isolating distributional faithfulness from capacity. Family Q (task accuracy). We report OpenCompass accuracy on fianne/mathemmiissmeiiccciioo using framework
8

---

Published as a conference paper at ICLR 2026
Dynamics. We report D4RL normalized returns IQM with BCa confidence intervals under identical SAC/ADETM settings.
defaults ($C.2)
Reporting. RL results use IQM with BCa 95% Cls over = seeds. LLM results follow the same protocol: AlpacaEval uses paired bootstrap over prompts; OpenCompass uses its default resampling.
4.6 RESULTS SUMMARY AND VALIDITY CONSIDERATIONS
RQ1/RQ3 - Policy fusion effectiveness. On OpenCompass, EMFuse improves over parameter-
space and logit-space training- training-free baselines on both aggregates reported in Table [ On the subject- mix, EMFuse attains 63.49 T.22
-1.23+ outperforming Soup (60.88) and RegMean (60.31), and narrowly
1.40 versus 88.27 (Pack- LLM), 83.51 (Soup), and 82.58 (RegMean). Task-level details are in Appendix Tables 20 and [1 Complementary preference evaluations on Family I. (AlpacaEval with length control) show EM- Fuse strongly outperforming Soup/RegMean and remaining competitive with larger 8B variants on some subject splits; see Appendix Tables [6 for per-dataset win rates and Cls. Appendix B.6 further disentangles the role of ADETM's temporal-consisteney uncertainty signal by comparing against a risk-neutral variant and local-sensitivity (action-noise) variant under the same rollout and SAC configuration. Using EMFuse as a reference for per-step selection further improves accuracy. From Table I, EMSelect gains + 1.31 points on the subject-mix and +1118 the finance-suic over EMFuse. Per-task breakdowns in Appendix Tables 20 and 21 show improvements concentrated in finance (e.g., FPB +2007, LendingClub + 1.42) and medication (e.g., MedQAM +2.68), with small regressions on some math sets (c.g., MGSMZ -2.40), yielding positive aggregate deltas.
exceeding PackLLM (63.15). On the finance-suite, EMFuse reaches 89.21 +0001
Table 1: OpenCompass (Family Q) - average accuracy. Two aggregates are shown: subject-mix (fnance/nahhematiimmddccccoo and finance-suil Numbers are averages with 95% CIs (2)) See Appendix Tables 20 and 21 for per-task breakdowns as tables are too large to fit in this section.
Aggregate
Soup
RegMean
PackLLM
EMFuse (Ours)
EMSelect (Ours)
Subject-min mix
60.88+1 25
60.31 1 1.25
1.24 63.15
63.49 1.23
64.80 15
Fo.3 31
1:32
23
Finance-suite
83.51
0.79
82.58 0.85
88.27
89,21 0.01
90.39
0..07
Table 2: Distributional faithfulness (Family L). Average token-level Kullback-Leible divergence from each IB domain expert (IBexp) to EMFuse (EM-F) and 8B experts on five evaluation domains (columns), smaller is better, with 95% confidence intervals. Domain descriptions: "harmless" and "helpful" refer to RtHF-style safety and alignment sets; "agriculture", "medication", and "philoso- phy" refer subject-specific QA and reasoning benchmarks. Evaluation details in Appendix $C.4
Language- based domains
Subject-sppciffe domains medication
Compurison Pairs
harmless
helpful
agriculture
philosophy
BBexp + EM-F
0.0391 0.0016
0.0801 +0 0035
0.0485 0000077
0.0481 +00001
0.0459
0.0000
00334
0.94225
0.00903
BBexp + 8B
0.5063 0.5063+000166
0.0190
0.3075-0.0121
0.5880 -00117
0.2827+0
0.0081
0079
0.2041 0.0022
RQ2 - Faithful distribution on experts' home domains. Table 2 shows that the token-level DKL (1Bexpert .) from each 1B domain expert to EMFuse consistently small (==0.04-0.00 0.00), and markedly below the divergence to its 8B counterpart (z00.20-..55 This supports that EMFuse preserves domain-specific token probabilities more faithfully than capacity scaling.
RQ4 Dynamics fusion for offtine control. Table 3 reports D4RL IQM returns: EMFuse av- erages 50. across
exceeding our data-free (Soup) and training-free (RegMean) basclines, and slightly surpassing the Mixed-data oracle baseline (47.8) on average. These findings support the conclusion that EMFuse, together with ADETM, provides tractable policy training framework that not only sidesteps the exponential overhead of naive ensembling but also preserves or enhances the performance of domain-mater matched experts across control tasks.
Scope and caveats. (1) Evaluator heterogeneity. Family Q uses accuracy- OpenCompass, while Family L. uses preference-baset Alpacalival (mitigated via length control); therefore avoid

---

Published as a conference paper at ICLR 2026
Table 3: Offline-RL performance on the MuJoCo medium quality benchmark (5 seeds, evaluation over the last 100 steps). Higher IQM return is better. Bootstrap CI provided on the right-hand side.
EMFuse (Ours)
Mixed (Oracle)
Environment
RegMean
Soup
Hopper
49.03
0.666
46.34
+4
47.33
47.33*
49.35 49.3511.13
+2.73
0.47
2.08
3.01
1.06
Walker2d
59.53 6.87
+ 7.13
52.24
8.11
12:29
46.52
-9.33
+22 14
51.64 +6.4+-1.224
26.64
HallCheetah
41.83 1.98
1.06
32.80
0..33
34.36
2.33
42.48
+0.53
2.89
3.92
1..77
AVERAGE
50.1
43.8
42.7
47.8
cross-family numeric comparisons and interpret each within its protocol. (2) Tokenizer/control of confounds. KI. analyses are restricted Family I. ensure shared tokenization; extending KI. to other families is left for future work due to compute limits. (3) Uncertainty in RL. Offline-RL results show wide CIs in some environments; we emphasize aggregate IQM and provide per-env CIs (Table 3) to avoid over-interpreting single tasks. (4) Selection trade-offs. EMSelect's gains are not uniform across tasks (notably some math sets); we attribute this local selection being most beneficial when expert specialization is strong and calibration is comparable, consistent with per-task deltas in Appendix Tables 20 and 21 (5) Scope of baselines. We focus on widely used triiiin-frre mergers (Soup/RegMean) and a logit-space method (PackI.LM) to match our operating regime; parameter- use uniform fusion weights by default; entropy-based reweighting yielded statistically insignificant differences, and Laplace smoothing produced negligible gains. Check Appendix $E.1
or data-hungry mergers are out of scope. (6) Weight-related ablations. We
5 CONCLUSION
In this work, we introduced EMFuse, a principled framework that unifies model fusion for decision- making under the lens of energy-based models. By treating energy common currency, ap- proach recasts the seemingly disparate problems of direct policy fusion and dynamics model fusion instances of single, elegant principle: the additive composition of energies.
Our core contributions are synergistic. The EMFuse framework provides the theoretical foundation, while the ADETM architecture makes this theory practical for modern offline reinforcement learning by circumventing the intractable computational cost of fusing model ensembles. Furthermore, our investigation reveals fundamental trade-off between two distinct fusion philosophies: consensus versus commitment. EMFuse, as a logarithmic opinion pool, generates conservative consensus distribution, hedging against the failure of any single expert. In contrast, EMSelect implements higher-variance variance commitment strategy, making winner-take-all decision at each step. Our empirical results underscore this trade-off; while EMSelect excels when context aligns perfectly with one expert's specialty, the consensus approach of EMFuse proves more robust on complex reasoning tasks where a synthesis diverse perspectives is superior to a single, potentially flawed, viewpoint.
setting due computational constraints, we view the challenge aligning heterogeneous tokenizers as a tractable engineering problem rather than fundamental barrier, especially given recent successes in vocabulary mapping (Mavromatis) eet al. 2024) Xu et al., 20244] Goddard Neto (225).. Integrating these techniques will unlock the full potential of applying FMFuuse diverse, ofe-the-sheff models. Overall, EMFuse provides solid and extensible foundation for collaborative AI, demonstrating that the simple addition of energies is a powerful and versatile tool for decision-makee makinn...
While our current study was scoped to
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Yangtze River Delta Science and Technology Innovation Com- munity Joint Research Program 2024CSIZN00302, the National Science Foundation of China and the Natural Science Foundation of Jiangsu (BK20243039). The authors thank anonymous reviewers for their helpful discussions and suggestions for improving the article.
ETHICS STATEMENT
This paper proposes efficient model fusion strategy and conforms with the ICLR Code of Ethics in every respect.
REPRODUCIBILITY STATEMENT
This paper provides all the information needed to reproduce the main results in the appendix.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice, 2022. URL https:/7
2200 13264
V. Alto. Building LLM Powered Applications: Create intelligent apps and agents with large lan- guage models. Packt Publishing, 2024. ISBN 9781835462638. URL https://books::
google.cmm co. hp/bookk/id-222 odtosssiidsiooooooooo TECAAQQAA]
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline re- inforcement learning with diversified q-ensemble, 2021. URL htt.psss//axii..rrg///////
2110.0153 01588
Miller Janny Ariza-Garzón, Mario Sanz-Cuerrero, Javier Arroyo Gallardo, and Lending Club. Lend- ing club loan dataset for granting models. Zenodo, 2024. URL htttps:://d ooogiiorg/000 [ 5281/ 11295916 get.
Derived subset of the public LendingClub data with binary tar-
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac HatfieldDoddss, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL httpt::t////// /ttps:///rxiiii .rg. abs 2204 05862
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
Tom B. Brown, Benjamin Mann, Nick Ryder, and et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee, and Marc G. Belle- mare. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems, 2018.
Ruifeng Chen, Chengxing Jia, Zefang Huang, Tian-Shuo Liu, Xu-Hui Liu, and Yang Yu. Offline transition modeling via contrastive energy learning. In Forty-first International Conference Machine Learning, 2024. URI htt.tttt: /wwpeoeeeeiieeeeeeeeeeeeeeee eww net forum? i d=dqpg
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement leam- ing in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
11

---

Published as a conference paper at ICLR 2026
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. atttp:://githue www.thub.co//oeeee compass/ opencompass, http://waa....aapaaaaaooaoooaoaa 2023.
Nicholas Kluge Corrêa, James William Santos, Camila Galvão, Marcelo Pasetti, Dieine Schiavon,
Faizah Naqvi, Robayet Hossain, and Nythamar De Oliveira. Crossing the principle-practice gap in ai ethics with ethical problem-solving. Al and Ethics, 2024, doi: 10. 000000076436811024-0004444 /t11/443366--022-00---------
URL https: springerrinnge comc coom/article//in
Joe Davison, Yacine Jernite, Clara Ma, Lysandre Debut, J. Plu, Sam Shleifer, Julien Chaumond, Thomas Wolf, Clement Delangue, Victor Sanh, Pierric Cistac, Teven Le Scao, Alexander M. Rush, Sylvain Gugger, Canwen Xu, Tim Rault, Quentin Lhoest, Mariama Drame, Patrick von Platen, Anthony Moi, Morgan Funtowicz, and Rémi Louf. Huggingface's transformers: State-of- the-art natural language processing, arXiv.org, 2019.
DeepSeek-AL. DeepSeek Model License Agreement, Version 1.0. https://github. htt.th..bccooo/cc..o deepseek ai aaeaaiieeeeeeeeeeeeeeeeeee -e/.kk-v//bleeeee wt/plob/maiinnieeeetpp..e hww.l//mmmn/nInEs...... MODEL, October 2023. Applies to all VeepSeek-V checkpoints and derivatives. Copyright @ 2023 DeepSeek-Al.
DeepSeek-AL. DeepSeck-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. /tttss://ttttttaa
May 2024. Initial pub- lic release 6 May 2024. Code under MIT; model weights governed by the DeepSeek Model Li- cense Agreement.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404. 04475, 2024.
Vladlen Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. In International Conference on Machine Learning, 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 202.00..........000...
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019. (Introduces BCQ / batch- constrained Q-learning).
Christian Genest and James V. Zidek. Combining probability distributions: A critique and anno- tated bibliography. Statistical Science, 1(1):113-135, 1986.
Charles Goddard and Fernando Fernandes Neto. Training-free tokenizer transplantation via orthog- onal matching pursuit, 2025. URL
06607
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, 2017.
David Ha and Jürgen Schmidhuber. World models. Neural Information Processing Systems Workshop, 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https:
01290
Danijar Hafner, Timothy P. Lillicrap. Mohammad Norouzi, and Jimmy Ba. Learning latent dynamics for planning from pixels. In International Conference on Learning Representations, 2019.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2026
Tom Heskes. Selecting weighting factors in logarithmic opinion pools. In Advances in Neural Information Processing Systems, 1998.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning Workshop, 2015.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8)11771-1800, 2002.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
/ //axxiv.ogg/ab///111.
09685
Aapo Hyvärinen. Estimation of non nom-normllized statistical models by score matching. In Journal of Machine Learning Research, volume 6, pp. 695-709, 2005.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Michael Janner, Yilun Li, Sergey Levine, and Chelsea Finn. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What dis- case does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL http:::ttt//sttttt http:/tpss//......ee//aaaa////// 2009. 13081
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances Neural Information Processing Systems, 2017.
Kisan Vaani. Agriculture-ga https://hugging agriculture qa english harpllisshonlyy// 2024.
co cc//eataatts/Kissaaannl
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-
learning. International Conference on Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu-Jie Huang. A tutorial on energy-based learning. Predicting Structured Data, 2006.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 000420005000,,,,, 2020.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of pnstruction-followee models. /ittps://qitiin
hatp/alpaca. _vvll 2023.
Haoxin Lin, Yu-Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia. Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step dynamics model improves future predictions for online and of- fline reinforcement learning. In The 13th International Conference Leurning Representations (ICLR (IC, Singapore, 2025.
Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang, Merge, ensemble, and cooperate! survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.066 06089, 2024.
13

---

Published as a conference paper at ICLR 2026
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 2022.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ilms: Model fusion at test-time via perplexity optimization. arXiv preprint arXiv:240411153,, 2024.
Meta. Llama 3.1 8b, 2024a. URL http:ps:/uugghhaeeeeee////////s2 Accessed: 2025-09-24.
1
8
Meta. Llama 3.2 1b, 2024b. URL http:://aarggaafa..e.////t22aa-a Accessed: 2025-09-24.
2
1B
Meta Platforms, Inc.
LLAMA 3.1 COMMUNITY LICENSE AGREEMENT.
https:
//buggingface, co cc/meta--llmme Llama- 7311-00--Instreet swtruuct/loommaaaa
LICENSE July 2024a. Version release date: 23 July 2024. Copyright @ Meta Platforms, Inc.
All reserved.
Meta Platforms, Inc.
https: //hugginnfface www./o/ooomaaaaaaaeeeeaeeeaaaaae www.aaaa...........a 1B-1BB/lob/nnn LICENSE txt September 2024b. Version release date: 25 September 2024. Copyright @ Meta Platforms, Inc. All rights reserved.
LLAMA 3.2 COMMUNITY LICENSE AGREEMENT.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024.
Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based rein- forcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):1-118, 2023. doi: 10.1561/2200000088,
The-Hai Nguyen, Dang Huu-Tien, Takeshi Suzuki, and Le-Minh Nguyen. Regmean++: Enhancing effectiveness and generalization of regression mean for model merging, 2025. URL
https:
/ //aaxii..rr/aaa//2555... 03121
Alexander Nikulin. sac-n-jax: Single-file SAC-N implementation in jax. https://g
comm/ sotowwhhsac--ee j-aac---jax 2023. GitHub repository, commit a 4044b3a a (21 May 2023), cessed 24 Sep 2025, Code under MIT, referenced in code implementation.
Long Ouyang, Jeff Wu, Xu Jiang, and et al. Training language models to follow instructions with human feedback. In Advances Neural Information Processing Systems, 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa : A large-scale multi-subject multi-choice dataset for medical domain question answering. 2022. URL https:
//rxxiv.org/bbs/20033 14371
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wi-
ley, 1994.
Reuven Rubinstein and Dirk Kroese. The Cross-Entropy Method. Springer, 2004.
Max Savery, Asma Ben Abacha, Soumya Gayen, and Dina Gemner-Fushman Question-driven summarization of answers consumer health questions. Scientific Data, 7(1):322, 2020.
Ankur Sinha and Tanmay Khandait. Impact of news on the commodity market: Dataset and results, 2020. URL https: / httpiaivivioo. org/ abs/2009. 04202
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction (2nd Edition). MIT Press, 2018
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL ht /tt.tttp:///ee..n qwwnlm.
github.i./blan io/blog http///llg////.../.g// twen2.57
14

---

Published as a conference paper at ICLR 2026
TheFinAL
Cra-lendingclub.
https://hugingfacc wt/ddaaasetttteeee
https:
cra- cra- lendingelub) 2024. Hugging Face dataset; accessed 2025-09-24.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, and et al. LLaMA: Open and efficient foundation language models. arXiv preprint urXiv2230..3997,
2023. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. Attention all you need. In Advances in Neural Information Processing Systems, 2017.
Mitchell Wortsman, Gabriel Ilharco, Samir Gadre, and et al. Model soups: Averaging weights of multiple munetuned models improves accuracy without extra training. In International Conference Machine Learning. 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024. URL https: hatp:://ppeneeeeew..eefftto
BAakY1hNKS
Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. Bridging the gap between different vocabularies for Ilm ensemble, 2024.
wwwpiivwwgg/ab..e. wttp:orbbabba....../// 09492
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, and et al. Mopo: Model-based offline policy optimiza- tion. In Advances in Neural Information Processing Systems, 2020.
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
In compliance with the ICLR 2026 Code of Ethics, we disclose the use of Large Language Models (LLMs) in this work. Our usage detailed as follows:
1. LLMs as Research Subjects: The core of our experimental evaluation involved using various large language models (e.g., Llama and Qwen families) to generate outputs for benchmarking purposes. All performance metrics and subsequent analyses presented in Section 4 are derived from the outputs these models.
2. LLMs as an Assisting Tool: A large language model was used for minor stylistic improve- ments and to assist in generating the LaTeX code for some of the tables presented in the manuscript.
The authors have meticulously reviewed and verified all content, including any text or code assisted by an LLM, to ensure its correctness and factual accuracy. The authors take full responsibility for the entire content of this submission.
B POLICY TRAINING DETAILS
B. TRAINING THE QWEN FAMILY OF EXPERTS (THE Q FAMILY)
We instantiate ALL Q experts from Qwen 2.5 7B Yang al. 2024 Team 2024) using supervised fine-tuning (SFT) with LoRA (Hu et al. 2021). Following a uniform recipe, we apply instruction- style formatting (answer-only loss), cosine LR with warm-up, and LoRA all linear submodules. For each adapter we train multiple runs (varying seed and a small LoRA grid) and select the best per- forming checkpoint by validation loss and downstream OpenCompass score training length, we use the largest epoch budget per group: 4 epochs for Subject-Mix and 10 epochs for the Finance-suite. This setup aims explore cross domain (Subject-Mix) and within domain (Finance-suite) fusion results, with each Post-SFT policy relatively strong in their own expertise, as can see in Table 20 and 21 All training regarding the Qwen family was conducted on a single A100 node containing 4 GPUs.
To standardize
dapters and training sets. Table 4 lists the exact adapter names (as shown in the results tables) and their paired training sets. Evaluation benchmarks are summarized separately in Table 5
Table 4:
Q-family adapters and their training sets (Qwen 2.5 base). Best checkpoint per adapter selected from multiple runs.
Group
Adapter name
Training set(s)
Finance Mathemati
CRA LendingClub Arizaauuubbiaaaeeee Garzón al.
2024)
Subject-Mix
Orea Math Word Problems (Mitra et al.
2024
Medi cation
MedQA + MedMCQA (Jin et al. 2020 Pal el al. 2022)
FPB Headlines
Financial PhraseBank (Malo et al. 2014)
Finance-suite
Financial news headlines (inha & Khandail 2020
LendingCing
LendingClub (hheFinAl 2024 Anza-Garzon al.
2024
Evaluation benchmarks (OpenCompass). decoding and normalization rules (cf. macro-averaging within domains. Benchmarks per domain are:
We evaluate with OpenCompass defaults under fixed
Decoding and selection. OpenCompass decoding parameters left framework defaults for comparability. For each adapter we keep the checkpoint with the lowest validation loss; ties are broken by the higher domain score under the corresponding OpenCompass evaluator.
16

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation benchmarks used for the Q family.
Domain
Benchmark (full name)
abbry.
Financial Opinion Mining & MA--Sentment Analysis
FiQASA
Finance (Subject-Mix)
LendingClub approval / credit risk
LendingClub
Grade School Math (8K)
GSM8K
Mathematics (Subject-Mix)
Multilingual GSM (English)
MGSME
Multilingual GSM (Chinese)
MGSMZ
MedQA (Mainland/CNN/ZH)
McdQAM
MedQA (Taiwan/CN/ZH)
MedQAT
Medication (Subject-Mix)
MedQA (USMLE/US/EN)
MedQAU
MedMCQA (Medical MCQ benchmark)
medmeqa
Financial PhraseBank (AllAgree subset)
FPB
Finance-suite
Financial news headline sentiment
Headline
LendingClub approval / credit risk
LendingClub
Table Canonical SFT hyper-parameters for Subject-Mix adapters (Qwen 2.5 7B).
Parameter
Value
Base model
Owen2.5-77
Epochs (num_train hamm:rriineeocheee n_spochs) Per-device train hatch size
Gradient accumulation steps
Learning rate
2 x 10-3
Weight decay
Max grad norm
Warmup ratio
0.03
LR scheduler
cosine
LoRA rank (lora.z*
64
LoRA alpha (lora_alpha)
128
LoRA dropout
0
LoRA target modules
all linear layers
Context length (max.length)
8192
Loss masking
answer tokens only
Table 7: Canonical SFT aaper-parraeters for Finance-suite adapters (Qwen 2.5 7B).
Parameter
Value
Base model
Qwen2. 5-7B
Epochs (num_tz foum_train... sepcchs
Per-device train batch size
2
Gradient accumulation steps
4
Learning rate
10
Weight decay
0
Max grad norm
1
Warmup ratio
0.03
LR scheduler
cosine
LoRA rank (1ora_r)
LoRA alpha (1ora a ppa)
64
LoRA dropout
0
LoRA target modules
all linear layers
Context length (max.length)
4096
Loss masking
answer tokens only
17

---

Published as a conference paper at ICLR 2026
B.2 TRAINING THE LLAMA FAMILY OF EXPERTS (THE L FAMILY)
We instantiate all L experts from the Llama 3.x line using supervised fine-tuning (SFT) on shared tokenizers that all policies operate over identical vocabulary V (cf. $2.2). Unless other- wise noted, we perform fill-purumeter SFT (no LoRA), adopt answer-only loss masking, and use standard trans www.eeeeeeeeeppeeeeeeeee
training (B2.2)). For robustness, we train multiple runs (varying seed and modest optimizer/schedule settings) and select the best checkpoint by validation loss and downstream metrics wongth-controlled AlpacaEval win rate; KL. to domain experts; see $C.3 (C.4). To match compute across capacities, the 8B expert training is FLOPS-capped to the aggregate of the IB experts as in Eq. 10 (within 1%).
FLOPs(IB.) s FLOPs(8B),
(10)
Experts and training sets. We organize L experts into groups-Language Quality (helpful- ness/harmlessness) and Subject Knowledge ates each expert, its base model, and the corresponding training set(s).
Table enumer-
Table 8: L-family experts and their training sets. The 8B experts trained under FLOPs budget matched to the aggregate 1B experts (Eq. 10
Group
Expert
Base model
Training set(s)
BBase
Llama 3.2-IB
Language
Bhanntess
Llama 3.2-1B
RLHF harmlessssss (Bai et al.
2022
Ouyang al. 2022
Bhelpful
Llama 3.2-1B
RLHF-helpful (Ba et al
2022 Ouyang of al.
2022
Beos
Ilama 3.2-1B
B aggicature
Llama 3.2-1B
Agriculture-QA Kisan 2024
Subject
B mediicaiion
Llama 3.2-1B
Consumer Health QA (Savery al.
2020
philosophy
Llama 3.2-IB
Ethical Problem-Solvine (Correa
2024)
RLHIF-hamless (Bai et al.
2022
Ouyang et al.
2022
8B Bangugg
Llama 3.1-8B
ReIlF-hllpful (Ba et al.
2022
Ouyang al.
2022
Aggregate
Agriculture-OU (Kisan Vaani
2024
Llama 3.1-8B
Consumer Health QA (Savery
2020
Ethical Problem-Solvee (Correa 2024)
Canonical SFT hyper-parameters. To keep this section parallel to the Qwen presentation, we summarize the SFT settings used for 1B and FLOPs-capped 8B Llama experts in two compact tables (Meta 2224a b). These consolidate the Trainer/DeepSpeed configuration into the key choices that most affect optimization and throughput. (Full details remain in (B2.22)
Table 9: Canonical SFT hyper-parameter for Llama 1B experts.
Parameter
Value
Base model
1ilama-3.2-11
Epochs (num_t wwanmtrain_eee craics)pochs) Per-device train hatch size
3
4
Gradient accumulation steps
8
Max sequence length
256
Optimizer
AdamW
I.R scheduler
linear (WarmupDecayl.e) equiv.)
Precision
fp16
Loss masking
tokens only
DeepSpeed
ZeRO-3 (off by default IB)
Evaluation protocol and checkpoint selection. For L, we evaluate with AlpacaEval under length control ( C.3) and compute token-level DKL. to domain experts ($C.4) Best checkpoints per expert
18

---

Published as a conference paper at ICLR 2026
Table 10: Canonical SFT hapee-parameters for Llama 8B experts (FLOPs-capped to Eq. I0
Parameter
Value
Base model
Llama-2 3.1---3.188 1-8B
Epochs wommmtrain.epochhe. Per-device train batch size
(stop early FLOPs target)
Gradient accumulation steps
Max sequence length
256
Optimizer
AdamW (31=0.9, 02=0.99 c=le-8
LR scheduler
linear (WarmupDecayl.F equiv.)
Precision
fp16
DeepSpeed
ZeRO-3 with CPU offload (params & optimizer) answer tokens only
Loss masking
chosen by (i) lowest validation loss and (ii) highest downstream score these evaluators (ties broken by the latter). Decoding uses uniform configuration across models (Table II EOS or max new tokens terminates generation.
Licenses. All Llama checkpoints are retrieved via Hugging Face (Davison et al. 2019) review the associated Meta licenses before use (Meta Platforms, Inc. 2024b
Please
B.2.1
LLM EXPERTS AND TRAINING SETS
Table 8 specifies base models and datasets for all L experts.
B.2.2 TRAINING SETUP AND FLOPS PARITY
We train with wesssoooCallbacae to enforce the budget Σ; FLOPs(IB.) ss FLOPs(8B) (Eq. 10). Hard- ware (typical): 1B experts 4090 node (2 GPUs); SB experts on 4090 node GPUs).
Traines and DeepSpeed ZeRO-3. The 8B experts
B.2.3 GENERATION CONFIGS
We implement Algorithm|| by computing per-expert log- probabilitiees (log softmax), summing in log-space with weights AG normalizing, then decoding. Each expert runs on a dedicated GPU; the fusion step executes single device.
Table II: LLM generation configuration.
Config
Value
max.new.tokens
200
temperature
1.0
top-p
0.95
We terminate EOS or max_new www.ens.ttkos,, applying the same decoding heuristics to the fused distribution as to single models (cf. (2.2).
19

---

Published as a conference paper at ICLR 2026
B.3 ADETM ARCHITECTURE AND TRAINING
We adopt the Energy-basee Transition Model (ETM) formalism (Chen al. 2024) and use the Any- step Dynamics Energy-based Transition Model (ADETM: cf. $3.3) to encode short action histories single-mode cost. Figure 2 shows the single-step and stacked multi-step variants.
PL-
Irper Stase
Thansties Modkl
Irpet Seggance
lxpa Staie
Enibedded State
Next State
Lomgla Sequence (1=-5)
Enaade: (MLP))
Itand
feg
Padcing
Valid Lough Seq, [4]
Input Vilit Leogo
Insodar (Atestion)
lastt Action Sequerer Input
Fitel Lengh Softerie Manix 3r
nul!
Predicwd Next Stins
nuf?
SE
Act
STM
Calestate Uncerotery. erc
nud!
mull
nulf
mul
Pesking
Wild Lesgth Seg (4222.)
Figure 2: ADETM variants: top single-step sext-state prediction; bottom stacked history for parallel multi-step predictions. Mirrors Fig. iin the main text.
State encoder. Given s E
hg = (Wss + b2) e Rd.
with two fully connected layers W1 E egda W2 E.ddd ReLU activations, and LayerNorm.
Action-sequence encoder. Let A E BB xdu be padded action sequences with true lengths l E (1,..., Immx) )
Algorithm 4 ActionEncoder(AA, A, E)
I: X Linear -ody (A)
|| projection
2: X + X + PosEmbb( [mmxx - 1)
3: maskij [ 2 4
|/ key-padding mask
4: Y < MHA(X; mask)
5: Y < LN(Y + +X)
6: Z + FFN(Y)
7: Z <- LN(Z +Y)
8: return ha & Ejch, Zij
The ADETM energy network conditions on [hs || ha].
Environments and hyper-parameters. We use D4RL-v2 (Fu et al. 2020) (Apache 2.0) across Hopper, Walker2d, and HalfCheet ah, with randonimeddumllepper splits. Each split yields a fixed offline buffer; unless otherwise noted, training uses single RTX 4090. ADETM defaults:
20

---

Published as a conference paper at ICLR 2026
Table 12: ADETM training defaults. Entries above the midrule follow ETM (Chen 2024);
embedding hww.:hidddennidii. dims applies to both state and action encoders.
Config
Default
etm 1r
x 10
etn.hidden.dimm
[200, 200, 200] relu
etmaactivation
etm.withh.ewward
true
1.0
www.unnnenativ..eappllel. etm.num.negati
16
etm_loss_type
infounce
stm.add.grad..eema.ty
true
sem.grradd.eeaally.marrriii
etm.langevin.itez
50
eepoch
500
etm stm.batch.size
1024
256
B.4 POLICY LEARNING (OFFLINE SAC IN WORLD MODELS)
We adopt Soft Actor-Crroit Critic (SAC) with two Q-networks (ensemble) Haarnoja et al. 2018 An et al.,
2021 Nikulin 2023). Hyper-parameters
held fixed across environments; Table 13 Rollouts use
ADETN with a short FIFO history (up to the maximum action action-sequence length; cf. Fig. D At step t, we backtrack up to min Lmax, and compute dispersion across stacked predictions obtain the penalty signal (as in Eq. 9 9n the main text).
Table 13: Offline SAC hyper-parameters for world-model experiments.
Config
Default
actor.lr
1 x 10
critic.lr
3 10-4
critic.nums
ganma
0.99
tau
0.005
alpha
0.2
alpha.1r
1 x 10-4
rollout.freq
1,000
callourtbatch.size
5,000
rollout.length
15
penalty.coef
p
model_retain.epochn
5
real,ratic
0.5
epoch
2,500
step.per.epoch
1,000
batch.size
256
eval.episodes
10
penalty.type
ensemble.std
We set p=0.3 for Hopper and p=0.7 for Hal aai Seeds: [1,2,3,4.5] A fixed budget of 2,500 steps is used; checkpoints are saved every 10 steps. Evaluation on the true *-medium environments
10 episodes per checkpoint. We report normalized return with inter- quartile mean (IQM) and BCa bootstrap 95% CIs across 5 seeds following Agarwal et al. 2022 ADETM hyper-parameter match Table 12 rollout.
21

---

Published as a conference paper at ICLR 2026
B.5 COMPUTATIONAL PROFILE OF ADETM
To make the computational footprint of ADETM explicit, we construct a controlled rollout-only simulation tailored to the default configuration used in our world-model experiments. Throughout the paper, ADETM uses a maximum valid history length of k=5, so that a single ADETM evalu- ation can condition on up five recent state-action sequence pairs when forming its uncertainty signal. In the notation of the main text, ensemble-base world models with M environments and N dynamics models per ensemble require O(M x N) dynamics evaluations per uncertainty-ave aawaee rollout step, whereas ADETM replaces per-environment ensembles with a single transition network per environment, which scales as O(M) ($3.3), To study this scaling under realistic architectures, we instantiate an ensemble configuration with - dynamics models per environment, matching the default ADETM history length k=5.
Both ADETM and the N=5 ensemble reuse the same nunsittionnetwork architecture in real runs. but gradient computation and parameter updates are disabled so that we measure only inference-tine time costs of the dynamics module. In practical ensemble, these N dynamics models would typically be initialized with different random seeds and/or trained on resampled data; here we deliberately keep architectures and weights identical across the N ensemble members that any differences arise purely from the number of dynamics evaluations per step rather than representa- tional capacity. Each configuration is plugged into the same world-model-based SAC rollout code path as in $B.4 on the D4RL-v2 medium tasks, with identical history length and hyperparameters but with gradients turned off. The goal of this subsection is solely to characterize computational scaling of the dynamics module; all return metrics for our method are reported separately in the main Experiments section.
Table 14 reports (i) the number of trainable parameters in the dynamics module, (ii) the estimated floating-poot point operations (FLOPs) per epoch of this rollout-only procedure (using the same counting scheme as (B.4), and (iii) the measured wall-clock rollout latency in milliseconds per environment step on one 4090 node.
Table 14:
Computational profile of ADETM versus an ensemble N=5 dynamics models per environment on D4RL-v2 medium tasks in rollout-only simulation. FLOPs are per epoch of the world-model-basee SAC loop; latency is wall-clock milliseconds per rollout step.
Enviromment
Dynamics setting
Params (M)
Epoch FLOPs (T)
Latency (ms/step)
Hopper
ADETM (k=5)
0.69
12.32
569
Ensemble (N=5 models)
3.47
44.35
642
Walker2d
ADETM (k=5)
0.70
12.43
589
Ensemble (N=5 models)
3,48
44.62
669
HalfCheetah
ADETM (k==)
0.70
12.43
591
Ensemble (N=5 models)
3.48
44.62
673
Across all three tasks, ADETM reduces the number of dynamics parameters by factor of 5x (about 1.69-0.70M vs. 3.47-3.48M) and the epoch-level dynamics FLOPs by roughly 3.6x (about 12.3- 12.4T vs. 44.3-44.6T), while also providing consistent reduction in rollout latency (approximately 10-1.. Because this simulation reuses the trained transition architecture and tensor shapes from our world-model runs and differs only in the number of dynamics evaluations per step, these results concretely instantiate the (MM) vs. O(M N) argument $ and $3.
22

---

Published as a conference paper at ICLR 2026
B.6 ABLATION ON ADETM UNCERTAINTY SIGNALS
To assess whether ADETM's performance gains stem from its temporal-consistency uncertainty signal rather than from generic variance heuristic, we compare ADETM against two baselines that share the same dynamics model but modify the uncertainty term used in the offline RL penalty.
Baseline A:
Risk-neutral (Zero Penalty). This baseline utilizes the full ADETM dynamics model for rollout generation but eliminates the pessimism term from the reward function. The penalized reward F(s, a) is computed as
F(s.a) f(s, a) us(s.a). a),
(11)
Le., the penalty coefficient is explicitly set to Open 0. All other syper-parameten (rollout length, model architecture, offline SAC settings) remain identical to the main EMFuse experiment. This baseline establishes risk-neutral floor and tests whether uncertainty-aware pessimism is necessary in the offline setting.
Baseline B: Local sensitivity (Action noise). This baseline replaces ADETM's temporal- consistency uncertainty with a standard local sensitivity metric derived from action-space pertur- bations. Instead of measuring variance across history lengths, it measures the variance of the pre- dicted next state s' under Gaussian action noise. For single model So and input tuple (s,a), the uncertainty is estimated as
()) Var ((So(s,a+ +c)) ())))))
4 ~~100,07
(12)
We set the perturbation scale to o = 0.05 (a 5% perturbation on normalized actions in [-1, 1). and the ensemble size to K 5, matching ADETM's maximum history window length Lmax so that the computational budget per rollout step is identical. The penalty coefficient Open is kept
identical to the main experiment (0.3 for Hopper and 0.7 for Walker2d/HalfChettahh) isolating the quality of the uncertainty signal as the only variable.
Results. Table 13 reports the Interquartile Mean (IQM) normalized returns with BCa bootstrap 95% confidence intervals across D4RL MuJoCo medium tasks. ADETM's temporal-consistens signal ("EMFuse (Ours)") outperforms both the risk-neutral and local-sensitivity baselines on all environments. Notably, the "Action-Noise" baseline (local sensitivity) performs substantially worse than the resk-neutral "Zero-Penalty" bascline on HalfCheetah, indicating that a naive sensitivity- based penalty is miscalibrated and can be actively harmful. This supports temporal consistency non-trivial and superior proxy for epistemic uncertainty.
Table 15: Ablation on dynamics uncertainty signals (IQM normalized return with 95% BCa confi- dence intervals) on D4RL-v2 MuJoCo medium tasks.
Method
Uncertainty source
Hopper
Walker2d
HalfCheetah
EMFuse (Ours)
Temporal consistency
49.03+0.72
77.53+7.13
41.83++1.0
0.47
6.87
2.01
Zero Penalty
None neutral)
40.25 1.50
51.76
+1.12
7.94 7.94+17.45
17:22
5.99
Action Noise
Local sensitivity
37.22 -224
45.05 99.18
1.72 -2.6
23

---

Published as a conference paper at ICLR 2026
C EVALUATION
C.I BASELINES
We compare three representative training- free baselines spanning the main fusion axes:
Uniform Model Soup (Wortsman al. 2022): a data-less, parameter-space merger
widely adopted in practice; it requires no access to the original training data and provides a reference point for weight-space averaging.
MeaMean (guyen et al. 2025); training-free (and data-less) regularized weight merging method that aligns statistics without task data; this family is commonly identified as canonical training- approach in recent overviews.
PackLLM (Mavromatis et al. 2024); training-free, logit-space policy fusion method at inference time. PackLLM's pairwise packing informed our EMSelect tournament design
3.2
C.2 CONFIG DRIVEN EVALUATION BY OPENCOMPASS
OpenCompass is open-source, configuration-ccntric evaluation framework for large language models that standardizes prompt templates, post-processing, and metrics across diverse datasets and tasks (Contributors 2023). It supports both local checkpoints (e.g., via vLLM/rann rorerss))) and hosted APIs through a unified model interface, enabling apples-to-apples comparisons under fixed decoding and normalization rules. This becomes more helpful when we can create customized wrapper to delegate existing models, therefore, simplify the model evaluation pipeline to great ex- tent. Although good number of datasets have already been supported, we add customized dataset entries to further complement our evaluation framework. On evaluation, each dataset is paired with its built-in prompt template and label normalizer, metrics are the framework defaults (classification accuracy for multiple-choice and exact-match style tasks). We aggregate results at the dataset level and then macro-average within each domain, matching the protocol reported in Tables 20 and 21 We bookkeep the configs (model adapters, dataset lists, decoding parameters, evaluators) for repro- ducibility. Consistent with the main text, OpenCompass scores are only compared within the Q family to avoid cross-framework tokenizer confounds.
C.3 LENGTI-CCOTTROLLED WIN RATE BY ALPACAEVAL
To assess domain expertise without relying on noisy absolute metrics, we use AlpacaEval (Li al. 2023 Dubois et al., 2024). This length- controlled preference evaluation is strongly correlated with human judgments (Spearman up to 0.97, Pearson up to 0.93 with alpaca_eval.gpt remaining cost-effective (Dubois et al.) 2023). We pre-generate responses over each test set to avoid adaptivity and then run pairwise comparisons with fixed rubric; the harmlessness prompt template is shown below.
while
<|in_start /sssstesse
You are a helpful costruction-follovii assistant. < <lim_endll
/sser
Select the output (a) or (b) that best matches the given instruction while still maintaining harmlessness and keep language usage free from cuss words.
(omitted for brevity; identical to main text template) < <iim_end|>
We use DeepSeck- V2 as the judge model (DeepSeek-Al 2024) via the official API respecting its license terms (DelpSeck-Al 2023).
http://ppp..deeeeek.ckkkk
24

---

Published as a conference paper at ICLR 2026
C.4 KL DIVERGENCE TO DOMAIN EXPERTS
We measure DKL. from each BBexperise distribution to (i) its corresponding 8B expert and (ii) EM- Fuse (pair group), loken-by-token under teacher forcing. averaged per prompt and then across the dataset (see Table 18 in the appendix). FLOPs are controlled as in Eq. 10 within 1%.
Given shared tokenizer for each context agee we compute
P(y) Q(1)
DRI(P (Q)
P(y) yEV
log
where P = softmax(21R) and Q is either softmax(Zsn) the fused Pruse from Algorithm | After scoring, we append the ground-truth next token and proceed to 1+1 until EOS.
D ALL EXPERIMENT RESULTS
Table 16: Win rate (%) of EMFuse against other methods on language datasets without additional training. 95% Cls shown as -i- When evaluating 1 Bexxprtisee> the dataset's 's matching domain expert
+
used (e.g., HH-RLHF harmless > Bharmless)
1Bbase
1 Ieexertisee
RegMean
Soup
8 SBtanguage
harmless
78.55
52.48 +1.87
78.5
78.54+1.00
62.61
52. 28. +1.66 31.34 1.69 1.81
helpful
52.48 1.73
+..82
35.23 1.94
+1.80
1.96
51.32" 1.96
1.81
47.67 2.00
1.85
Average 65.52
43.86
64.93
55.14
41.81
Table 17: Win rate (%) of EMFuse against other methods on subject datasets without additional training. 95% CIs shown -7-
1B
1B 1Bexpertise
RegMean
Soup
8Bsubject
agriculture
87.21
-0.30
48.27 L.62 1.62
1.g0
86. +0.50
53.33
+9.76
17.57 1.28
-1.31
philosophy
medication
88.30 0.41
84.03
H6.02
-0-64
34.65
28. 10.78
+0.07
-077
83. 63
88.22 0.41
+0.62
0.87
52.56
51.48
+0.69
+0.91
62.31
36.14
H0.39
+0.87
0.30
-0.63
Average
86.51
37.01
86.19
52.46
48.67
Table 18: Distributional faithfulness (Family L). Average token-level Kullback-Leibler diver- gence from each IB domain expert (1Besp) to EMFuse (EM-F) and 8B experts on five evaluation domains (columns), smaller is better, with 95% confidence intervals. Domain descriptions: "harm- less" and "helpful" refer to RLHF-style safety and alignment sets; "agriculture", "medication". and "philosophy" refer subject-specific QA and reasoning benchmarks. Evaluation details in Ap- pendix CC.4
Language- domains helpful
Subject-specing domains medication
Compurison
harmless
agriculture
philosophy
1 BBexp + EM-F
0.0391 0. 0015
0.0801 + 0035
0.0485 0000027
0.0481
+0000144
0.0459 -U.0---U000 0.00
0.0014
0034
0025
000000 0014
0.0005
Besp + 8B
0.5063 40.0011
0.0190
0.3075 0.0121
+000124
0.5880
+0.0110
0.0177
0.2827
+00007
0.0081
0.2041
+0.0022
0.0022
25

---

Published as a conference paper at ICLR 2026
Table 19: Offline-RL. performance on the MuloCo medium quality benchmark (5 seeds, evaluation over the last 100 steps). Higher IQM return is better. Bootstrap CI provided on the right-hand side.
EMFuse (Ours)
Mixed (Oraele)
Environment
RegMean
Soup
Hopper
49. 49.03 +..++0.00
46.341a.4
47.33*3.10
47.33
49.35
2.66
Walker2d
59.53 6.87
52.21
46.52 19.31
51.64
16.24
25.64
HallCheetah
41.83 1.98
32.80
2.89
34.36 -3.2
42. 12.48 1.37
AVERAGE
50.1
43.8
42.7
47.8
Table 20: Performance across finance, mathematics, and medication benchmarks. Higher is better; bootstrap CI 95% is shown.
Task
raw
SFT
fusing 3 SFT models
Qwen 2.5 (7B)
EM-F
EM-S
Fin
Math
Med
Soup
RegM
Pack
(Ours)
(Ours)
78.67
97.99
76.59
80.71
81.31
79.86
96.58
96.80
97.73
LendingClub
11.51
10.47
1.00
11.43
11.47
10.00
10.00
-1 -1.59
-0 na
-1.64
-1.55
-1.32
-0.75
-0.65
43.83
56.17
37.45
34.04
42.98
43.40
45.11
42.55
53.19
FIQASA
16.39
16.19
6.84
10.27
9.89
6.40
16.90
16.99
1.228
0:15
6.88
79.98
83.24
84.00
81.12
83.40
82.79
84.99
84.69
83.62
GSM8K
+2.08
+1,92
-1.85
+2-02
+1.91
+1.91
+1.82
+1.81
+1,90
24
-2
-2
-2:11
03
-2 03
-2
78.40
82.00
78.80
80.80
80.80
80.40
82.40
83.20
82.00
MGSME
-5 94
-5.33
-5.94
58.80
53.20
54.00
42.80
58.80
59.20
54.00
56.00
53.60
MGSMZ
+5,92
+60.0
-8.07
+-6.20
+-5922
+5.01
+6.07
+6.02
4-0.88
66.19
44.19
56.60
52.77
71.28
60.01
55.93
60.07
63.05
65.73
MedQAM
+1.67
+1.00
-1-67
+1.40
+1 63
+1.65
+1.63
+1.60
+1.57
15
84
677
65
63
47.
57.54
59.02
60.08
56.05
55, 27
56.76
57.68
58.24
MedQAT
-2.99
-2 B0
-2.58
-
-2
-2.00
-2.99
2
37.23
44.46
37.78
49.18
43.91
42.81
47.76
47.29
48.00
MedQAU
+2.70
+2.74
-2.70
+2--4
+2-74
-2.74
+22-7
+2:75
+2
-2-11
-2.74
-2.71
-2.
-2.73
-2.73
-2-74
48.72
45.71
36.84
41.02
40.62
43.13
40.66
40.14
41.05
medmeqa
-1.51
-1.51
-1.45
-1.48
-1.50
-1.47
-1.48
-1.88
57.44
64.10
57.47
60.11
60.88
60.31
63. 15
63.49
64.80
Average
-1.27
-1.25
-122
-1.25
-1.23
-1.25
Table 21: Finance-suite performance (leendingClub, FPB, Headline). Higher is better; bootstrap CI
95% is shown.
Task
raw Qwen 2.5 (7B)
+ SFT (finance)
fusing 3 SFT models
EM-F
EM-S
FPB
Head
Lend
Soup
RegM
(Ours)
(Ours)
81.85
98.81
85.78
86.62
93.15
91.96
92.71
95.94
98.01
FPB
-1.38
-1.85
-0.84
-1.50
-1.11
-1.19
-1.14
-0.00
-0.68
72.12
71.76
79.33
73.08
75.17
75.02
76.30
75.40
75.45
Headline
+1.87
+1.88
-1.67
+1.90
+1.79
+1.50
+1.75
+1.79
+1.79
-1
-1 us
78.67
78.56
77.26
97.99
82.20
80.75
95.80
96.28
97.70
LendingClub
11.51
11.61
11:45
10.06
10.00
-1.59
-1.59
-1.82
-1.49
-1.53
-0.82
-0.64
77.65
83.04
80.79
85,90
83.51
82.58
88.2 $8.27
89.21
90.39
Average
-0.81
-2.92
-1.12
-0.78
-0.85
-0.04
-0.01
-0.07
26

---

Published as a conference paper at ICLR 2026
E ADDITIONAL ANALYSES
This appendix collects several supporting analyses around EMFuse and EMSelect. First, we doe- ument two lightweight modifications used in policy-fusion experiments: (i) Laplace (add-o) smoothing of token probabilities to guard against zero support for individual experts, and (ii) step- wise entropy-based reweighting of experts. Both variants were evaluated using the same OpenCom- pass protocol and decoding settings as the main results (cf. Appendix. C.2;; in tasks and domains, none yielded statistically significant improvements over the baseline of uniform-weit weeoht EMFuse, with confidence intervals that largely overlapping. Second, we present a simple continuous-learnng extension of EMFuse, where experts are updated toward the fused consensus under regularized ob- jective, together with short empirical summary illustrating how this improves returns in follow-up setting. Finally, we provide KL-bbaed analysis of EMSelect, using the KL. chain rule to show that the sequence distribution induced by EMSelect remains bounded relative to the EMFuse consensus. For completeness, we give precise definitions and ablation summaries for each component in the following subsections.
E.1 LAPLACE SMOOTHING FOR EMFUSE POLICY FUSION
Definition. Given an expert's next-token distribution p(- (gg)) over a vocabulary of size V, we form smoothed distribution
(1- + UU
U(y) =
(13)
We apply this independently to each expert at every decoding step and then perform LogOP/PoE fusion the smoothed log-probabilities:
PPo xzge) = softmax Σ de log log zg))).
(14)
Smoothing guarantees strictly positive support on all tokens and mitigates numerical fragility when an expert assigns (near-)zero mass off its top-k. In practice we keep a small to preserve the experts calibration.
Motivation.
Observation. On the finance, mathematics, and medication suites, add-o had mixed, magnitude-
small effects: its CIs largely overlapped the baseline EMFuse (Tables 22 23 unsmoothed EMFuse in the main experiments.
We therefore retain
E.2 ENTROPY-BASED EXPERT WEIGHTING
Definition.
Let (T(- Ige)) be expert i's next-token distribution with Shannon entropy H;(!)
Ey Tz(y x log ( x.)). We define step-wise fusion weights
(t) x exp( - 3 (t),,
Σ (t)
(15)
and use these weights to fuse the experts* log-probabilities before decoding:
tuse (f) Σ (i) log xz:))
(16)
Tisse (t) exp (einee logsumexp(Camee (t))
(17)
Here sharpens the preference for lower-entropy (more confident) experts.
Motivation. Per-step entropy modulates experts by simple, calibration-agnonn proxy of confi- dence, without additional training or task labels. Computationally. the overhead negligible relative to computing per-expert logits.
Observation. Entropy weighting occasionally nudged scores upward on some finance benchmarks but produced minor regressions elsewhere; the average effects were not statistically significant under our bootstrap CIs (Tables 22 23). Consequently, we use uniform X; in the main text and report these variants ablations.
27

---

Published as a conference paper at ICLR 2026
Table 22: Ablation on EMFuse across finance, mathematics, and medication benchmarks. Higher is better; 95% bootstrap CIs shown under each score.
EM-Fuse (baseline)
Laplace Smoothing
Entropy- Entropy-based
Task
Weighting
96.80
94.28
97.81
LendingClub
10:00
10:81
10.49
42.55
40.43
43.83
FIQASA
16.30
16.38
66.39
0.16
0.06
6.15
84.69
84.46
84.38
GSMSK
11.84
11.80
|1.50
-2.05
-2-06
-2.06
83,20
82.80
82.40
MGSME
44.12
14.17
44.22
56.00
55.20
58.40
MGSMZ
+6.02
+6.04
+5.99
-6.20
-0.20
-6.19
63.05
61.15
61.88
MedQAM
+1.60
+1.62
+1.61
-1.63
-1.04
-1-64
57.68
56.55
56.97
MedQAT
+2.55
+2.58
+2.58
47.29
16.66
45.25
MedQAU
+2-75
+2.75
+2.74
-2.73
72
-2.12 72
40.14
40.40
40.40
medmeqa
11.49
11.50
63.49
62.44
63.48
Average
+1.29
+1.29
+1.23
Table 23: Ablation on EMFuse for the finance suite (FPB, Headline, LendingClub). Higher is better; 95% CIs shown.
EM-Fuse (baseline)
Laplace Smoothing
Entropy- based
Task
Weighting
95.94
95.94
98.19
FPB
10.73
10.73
10.47
75.40
75.40
74.83
Headline
11:79
11.79
11.51
1.96
1.56
1.20
96.
96.25
97.88
LendingClub
+0-06
+0-65
89.21
89.20
90.30
Average
+1.40
40.71
+0.67
28

---

Published as a conference paper at ICLR 2026
E.3 CONTINUOUS LEARNING UNDER EMFUSE
EMFuse is presented as a training-free fusion rule over a fixed family of experts Mı,..., Mn, but the same energy-additive structure also naturally supports lightweight continuuusleaannng variant. Let P.(- x) denote the normalized policy of expert M, on the shared vocabulary and define the EMFuse consensus
Piuse x) & II: xx)
i=1
with nonnegative weights x; that sum to one. When new trajectories D arrive, we update only small adapter parameters 1mm on top of the frozen experts by minimizing
C(01:n) = (E log (y x) + Oreg
KL (L [| [1)))
(18)
where po; is the updated policy of expert M, and Oreg 2 is regularization coefficient. The first term fits the fused policy to fresh data, while the second term keeps each expert softly aligned with the evolving EMFuse consensus without changing the inference-time sampler fusion rule. Op- tional robustness tweaks such as adding a small Laplace smoothing constant to logits are treated purely minor regularizer and are essential to the learning objective (as discussed in the appendix). In successive experiments following the main EMFuse study, instantiating this objec- tive with lightweight adapters led consistent additional gains: language-model experts obtained modest but stable improvements in length-controlles win rates, and policies trained under fused dy- namics achieved higher normalized returns than the purely training-free fusion baseline, indicating that EMFuse can be embedded into continual-learme pipeline when desired. Representative gains on language benchmarks are summarized in Table 24
Table 24: Representative improvement from continuous-leamingg extension of EMFuse on the Fam- ily L. language tasks. Domain descriptions: "harmless" and "helpful" refer to RLHF-style safety and alignment sets. Entries report EMFuse win rates (%) after a single adapter-traiiig stage, averaged over seeds under the same OpenCompass protocol as Table [6] with 95% BCa confidence intervals shown as superssriptss/ubssriiss. The "Average" row is the mean over the two language tasks. The last row (A Average) shows the corresponding improvement over the training- freee fused policy, in percentage points.
Bass
1Bxxpetise
RegMean
Soup
8Bianguage
harmless helpful
78.44
54.03+2.01
79.94
1.50
63.18+1.80
+1.44+1.87
1.43
1:83
64.76 1.84
49.07
63.57 1.87
1.80
62.23 1.91
1.86
45.
1.84
1.88
Average
71.60
51.55
71.76
62.71
49.30
A Average
+6.09
+7.70
+6.83
+7.57
+7.49
29

---

Published as a conference paper at ICLR 2026
E.4 SEQUENCE-LEVEL KL BOUND FOR EMSELECT
We formalize the relationship between EMSelect and EMFuse by analyzing the sequence-level di- vergence between their induced autoregressive policies. We reuse the notation from the main pa- per. For a fixed prompt To. let V1.7 = (V1--.., M/)) denote the generated token sequence and Tst == (ro, g<ee) the conditioning context step Each expert {I...., nz defines an autore- gressive policy
Pi age). (B mo) = II
t=1
EMFuse defines the LogOP/PoR fused policy
T
PFuse (VI.T (o))
II
Piuse (ye xgee),
Piuse (ye (gge) oc II
P:(ue xsa)"
1=1
2=1
and EMSelect induces another autoregressive policy
(eel 2o)
II Psct(Ut xgg)) 1=1
Puel(y xzge)= = (re( xgt),
where (x( is the index i" of the winning expert selected by the tournament mechanism described Algorithm of our main paper. Lemma (KL chain rule for autoregressive policies), Let
T
T
((hrr) = II (e Vee),
p(yi.)) = II
Pe Vee))
be two autoregressive distributions on VT, Then
KL(q
Ey Byergg KL mEe) ll po( ( v=ti))
(19)
Proof (by induction on T). For T = 1 we have
(v1) KL(g) II), (v1)
KL(allp) = Σ 21 (21) log
91
so equation 19 holds. Assume it holds for all sequence lengths up to T - 1. Write
((yI.)) = g(veT)er(yr veT), p(yiT) p(v-r)pr(m BeT).
Then
((1:T)
KL(gllp) = Σ ((viT) log
p(y1T)
Mi.r
Σ ((ver)) log ((+TT)
2r (vr yeT) pr(3T y<rr
Σ (((rr)) ( log
p(yxT)
W.T
Σ ((ver) log ((v<T)
ar(ur (eTT) log qr (yT BET)
Σ Ea(ver) Σ
p(yeT)
pr pr(yr Der)
Mer
yr
KL(g(-er) [[x(-=r))
KL ( [((==r)) + Eyerrny KL KL V<r) p ver))]
By the induction hypothesis applied to the length- 1) marginals g(-xT) and p(-er), we obtain
T-1
KL(g(=<r\\pp(-<n) = Σ
E B
KL vea) pl 3=)))
30

---

Published as a conference paper at ICLR 2026
Combining the two displays yields equation 19 for length T, which completes the induction. Sequence-level bounds between EMSelect and EMFuse. Apply Lemma equation with
q Psel( zo) and Ptuse( (- mo), and write
P (gge) = Psel æg(), Pluse (- (gge)= = Ptuse( (- (Zge).
We obtain
(Taa (0) Pfuse (- ro))
Σ t=1
E e [T.(Psct,e( (iga) Pruse, [ (20)
For each step & and context Zgt, define the expert-fuse KLS
(#gg))) (T.(p: (rge) Prusel (rge));
Amin (ree) min A (x (*=(())))))))))))))) = max (rrge)))
By construction, (- (gg)) coincides with one of the experts (pil. xgt)) pointwise in Zşt we have
Therefore,
(zg() & (P (gg)) |l Pluse, (: xg1)) I A A (ISL).
(21)
Taking expectations in equation 21 with respect to Vet ~ Paet and summing over t, equation 20 yields the sequence-level envelope
Eyer rupnll
Σ min
s [psell 20) Pluse (- z00) E Puperrpen
Σ A
(I
(22)
In partieular, if we have per-step uniform bounds Amax (x
C2 for all Tig,, then
T C..
(p (D) || Płuse (oo))
S
(23)
In words. the divergence between EMSelect and EMFuse at the sequence level is bounded by the cumulative envelope of expert-to-fuse divergences along rollouts generated by EMSelect. Empir- ically, our distributional faithfulness measurements ($ C.4 Table (8) show that the expert-to-fuse divergences remain small (typically < 0.09), the sequence-level divergence must also remain small. This confirms that EMSelect is mathematically leashed to the EMFuse consensus, ruling out uncontrolled temporal drift even though EMSelect commits at each step to single expert.
31


Reviews:
[{"review_id": "Kqepp1Dyqw", "reviewer": "Reviewer_sqFs", "summary": "This paper investigates a challenge about resource-efficient model fusion in decision-making tasks. By considering both fusion of models to act as policy and fusion of dynamic models to induce a policy simultaneously, this paper introduces a framework called EMFuse and an architecture called ADETM to tackle the challenges. The EMFuse shows superior performance in multiple benchmarks including subject-mix, finance-suite and D4RL, compared to three modern training-free baselines.", "questions": "Please see my weaknesses above. Additionally, I might miss something in related work, I'm curious how do those evaluation labels defined and the values calculated in table 2, i.e., harmless, helpful, agriculture medication, philosophy?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. Visual presentations are clear.\n- The motivation is clearly described and the designed framework and algorithms seem clearly aligned with the investigated problems\n- Though I’m not an expert in the energy-efficient domain, the proposed methods seem interesting and well tackled the problem.\n- The proposed approaches achieve superior performance in various benchmarks, and in terms of various evaluation metrics  \n- Limitations are thoroughly discussed", "weaknesses": "I don't have a lot background knowledge in energy-based models, the proposed approaches in general seem sound to me and work well in the benchmarks. But given one of the claimed motivations is the heavy computational complexity for existing dynamic models, I'm expecting if further analysis on related evaluations compared to existing dynamic models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yNcfwokK4F", "reviewer": "Reviewer_K6PE", "summary": "This paper proposes an energy-based decision model fusion framework, EMFuse. It unifies policy fusion (e.g., for LLM) and dynamic model fusion (e.g., in offline reinforcement learning) through  additive energy composition. The authors also propose EMSelect, a KL-logic-based expert selection method, and ADETM, a single-model architecture that estimates uncertainty without ensemble requirements. Experiments on LLM and D4RL benchmarks demonstrate that EMFuse significantly improves performance compared to training-free baseline methods such as Model Soup and RegMean.", "questions": "1. This paper assumes that multiple expert distributions can be fused through energy addition. However, if the experts differ significantly or even conflict, could the product of energies lead to overly sharp or degenerate fused distributions?\n2. EMSelect performs per-step expert selection using KL divergence but lacks temporal consistency constraints. If the selected expert changes at every step, could this cause unstable behavior or disrupt contextual coherence during decision making?\n3. The authors claim that ADETM has an “any-step” property that enhances modeling capability, but in practice it seems to mainly extend the history window. Does ADETM truly offer a representational advantage over existing autoregressive or recurrent ETMs, or is it more of a structural extension rather than a conceptual innovation?\n4. The paper always uses uniform fusion weights λ, but different experts may have varying confidence or training quality. Is this fixed weighting scheme reasonable, or would it be better to consider dynamic or uncertainty-based weighting strategies?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a well-structured and theoretically consistent framework called EMFuse, which unifies policy fusion and dynamics model fusion under the lens of energy-based modeling.\n2. By leveraging the additive property of energy functions, the paper provides an intuitive explanation of model fusion and naturally connects it to classical formulations like LogOP and PoE.\n3. The  proposed ADETM model avoids the high computational cost of ensemble methods while achieving single-model uncertainty estimation, which makes it quite practical.\n4. The experiments cover both large language model tasks and offline reinforcement learning benchmarks, showing the framework’s versatility and effectiveness across different decision-making scenarios.", "weaknesses": "1. The theoretical discussion in this paper is relatively shallow. Although the authors explain the relationship between EMFuse and LogOP/PoE, they do not provide in-depth mathematical analysis or systematic ablations on fusion stability, the influence of λ, or the effect of uncertainty modeling.\n2. The main idea is conceptually close to existing energy-addition or PoE/LogOP-based fusion methods.\n3. The paper does not provide sufficient analysis of the independent roles of EMFuse, EMSelect, and ADETM. More detailed ablation studies would help clarify how each module contributes to the overall performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G0tm0dSQ4Y", "reviewer": "Reviewer_YqY1", "summary": "This paper proposes a unified framework called EMFuse, aimed at solving the problem of model fusion in decision-making. The authors argue that whether it's directly fusing policy models or fusing dynamics models, both can be uniformly addressed through the lens of Energy-Based Models. To address the computational explosion problem caused by using ensembles when fusing dynamics models, the paper introduces an architecture called ADETM. This architecture leverages variable-length action histories within a single model to estimate uncertainty, thereby avoiding the need to ensemble multiple models.Furthermore, the paper proposes the EMSelect algorithm as an alternative to EMFuse. Instead of generating a consensus distribution, it uses a pairwise EMFuse as a reference at each decision step to select the single expert model with the minimum KL divergence from this reference distribution.In the experiments, the authors evaluated EMFuse and EMSelect on LLM tasks and D4RL MuJoCo control tasks. The results show that this method outperforms several existing training-free fusion baselines.", "questions": "1. Can the authors elaborate further on the scenarios in which the \"consensus\" strategy of EMFuse would be superior to the \"selection\" strategy of EMSelect?\n2. Given that the entropy-based weight adjustment was not effective, have the authors considered other dynamic weighting schemes?\n3. Beyond citing existing work on vocabulary mapping, could the authors discuss the potential negative impacts on the EMFuse framework's performance when forcibly aligning the representation spaces of different models? After mapping, can the \"shape\" and \"scale\" of the energy functions still retain their original semantics to ensure the effectiveness of the fusion?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper provides a conceptually unified framework, treating the two seemingly different problems of policy fusion and dynamics model fusion as an additive combination of energy functions through the EBM perspective.\n2. The authors identify the \"combinatorial explosion\" problem that traditional ensemble methods introduce when fusing dynamics models. They propose the ADETM architecture to address this. By borrowing ideas from ADMPO and combining them with EBMs, ADETM achieves effective uncertainty estimation using only a single model.\n3. The paper's experiments cover two distinct decision-making domains: LLM-based discrete decisions and D4RL-based continuous control. It compares the proposed methods against several representative training-free fusion methods like Model Soup, RegMean, and PackLLM.\n4.  By comparing EMFuse and EMSelect, the paper reveals that EMFuse seeks a conservative, robust consensus solution, whereas EMSelect chooses an \"optimal\" expert at each step. This discussion deepens the understanding of the nature of model fusion.", "weaknesses": "1. The experimental results show that EMSelect outperforms EMFuse in multiple benchmarks. This raises a key question: If \"selecting an optimal expert at each step\" is more effective than \"fusing all experts,\" does this, to some extent, weaken the \"consensus\" strategy advocated by EMFuse? The paper attributes this to a trade-off but fails to deeply analyze the root cause of this phenomenon.\n2. A core assumption of this framework is that all models to be fused must operate on a shared vocabulary or state-action space. The authors acknowledge this, treating vocabulary mapping as a \"viable engineering problem.\" However, this assumption is a very strong limitation in practical applications, especially when fusing models from different sources.\n3. Most experiments in the paper use simple uniform weights. The authors attempted entropy-based dynamic weight adjustment but concluded the effect was \"not statistically significant.\" Why did entropy-based weighting fail?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)