LS-Merge: Merging Language Models in Latent Space
Authors: Bedionita Soro
Aoxuan Silvia Zhang
Bruno Andreis
Jaehyeong Jo
Song Chong
Sung Ju Hwang
Venue: ICLR 2026 Poster (2026)
Keywords: LS-Merge
LLM merging
latent space
weight space learning
Primary area: generative models
TL;DR: Merging Language Models in Latent Space

Abstract: Model merging in weight space is an efficient way to reuse pretrained models, but existing methods typically assume matching architectures or sizes, making heterogeneous merges brittle or infeasible. We address this limitation by encoding model weights into a smooth latent space, enabling cross-architecture operations, and performing the merge in the latent space before decoding back to weights. This approach faces two major challenges. First, LLMs contain billions of parameters, which makes latent encoding computationally demanding. Second, using high compression ratios often hinders the encoder’s ability to generalize to unseen weights. We tackle these issues with a transformer-based variational autoencoder (VAE) trained in a two-stage compression curriculum with structured layer-aware chunking: the model first learns a high-capacity latent representation and then distills to a compact code, improving both stability and out-of-distribution generalization. To align heterogeneous models, we introduce a dimensionality-matching projection that allows interpolation between models of different sizes. Empirically, latent-space interpolation is consistently more robust than direct weight-space averaging and yields stronger downstream performance when merging models of different sizes. Together, these components provide a scalable, architecture-agnostic recipe for model merging.

Published as a conference paper at ICLR 2026
LS- LERME: MERGING LANGUAGE MODELS IN LATENT SPACE
Bedionita Soro Aoxuan Silvia Zhang' Bruno Andreis
Jaehyeong Jo
Song Chong' Sung Ju Hwangl.21
KAIST *DeepAuto.ai UUniversity of Oxford 4Slater Labs
ABSTRACT
Model merging in weight space an efficient way to reuse pretrained models, but existing methods typically assume matching architectures or sizes, making heterogeneous merges brittle or infeasible. We address this limitation by encoding model weights into smooth latent space, enabling arosttccchee operations, latent space before decoding back to weights. This approach faces two major challenges. First, LLMs contain billions of parameters, which makes latent encoding computationally demanding. Second, using high com- pression ratios often hinders the encoder's 's ability to generalize to unseen weights. We tackle these issues with a transformer-based variational autoencoder (VAE) trained in two-stage compression curriculum with structured layer-aware chunk- ing: the model first learns high-capacity latent representation and then distills to compact code, improving both stability and out-of-distribution generalization. To align heterogeneous models, we introduce dimeeniionlity-matching projection that allows interpolation between models of different sizes. Empirically, latent- space interpolation is consistently more robust than direct weight-space averaging and yields stronger downstream performance when merging models of different sizes. Together, these components provide a scalable, architecture-agnosiic recipe for model merging.
and performing the merge
INTRODUCTION
Large Language Models (LLMs) are foundational modern artificial intelligence. However, their pretraining demands millions of GPU-hours, leading to a significant inefficiency if the acquired knowledge remains confined single model instance. To mitigate this cost and enhance utility, weight-space model merging has emerged as a promising approach. This technique combines parameters from multiple pretrained models integrate complementary capabilities and improve performance on diverse tasks, all without additional pretraining (Yang et al., 2024).
Existing merging methodologies range from straightforward linear interpolation (Wortsman et al., 2022) to sophisticated evolutionary search-basee fusion (Akiba et al., 2025), consistently demonstrat- ing practical benefits at scale. Despite these advancements, current techniques typically face two significant limitations: (i) Requirement for multiple source models: Most approaches necessitate least two distinct pretrained models for merging, which restricts their application when the goal is to enhance or adapt a single existing model. (ii) Architectural homogeneity: Merging methods frequently assume shape-compatible or architecturally homogeneous models, hindering their use with mismatched architectures (c.g., varying widths depths) (Yu et al., 2024a). These constraints significantly limit the broad applicability of merging across diverse LLM checkpoints as well as self-merging. Addressing these constraints is crucial for unlocking the full potential of pretrained LLM checkpoints and fostering a more flexible and efficient paradigm for model reuse.
"Equl contribution. 1 CCorreppnndecce to: www.via..aannekeaa
Bedionita Soro sorobedl ac.kr. kr. Aoxuan Silvia Zhang
Bruno Andreis bruno. andreisBeng ex ac uk. Jae- hyeong Jo 977 ist. aas.saa.ckk. kr.kr, Song Chong songchongāka at. Sung Ju Hwang @hwaang22kkii ac.

---

Published as a conference paper at ICLR 2026
Ei
2
D1
Model
Latent Space Aligument
D1
L
Linear Interpolation
E2
Heteragencom Model Merged Latent Vector
2/
D2
22
Merged Model
Model
a) Model Encoding
b) Latent Space Merging
<) LS-Merge Decoding
Figure I: LS-Merge: encode, align, and decode for LLM weight merging. (a) Pretrained weight tensors 2 and ze are chunked layer-wise and encoded by E1. E2 into latents 21, 2g: decoders D1, D2 reconstruct ,1, .2. (b) A latent-space alignment places and 22 in shared manifold where they are (lnner interpolation) yielding a merged latent za. supporting both homogeneous (same- architecture (same encoding model) and heterogeneous crrsss-achiteectree merges. (c) A target decoder (e.g., Dí) maps za back to weights Ta producing the merged model. Here, I; denotes the full weight set of model i, 2; its latent code, and tildes indicate reconstructions.
Recent advancements weight-space learning (Sch"urhole urrolt et al., 2024; Schürholt et al., 2022; Peebles et al., 2022; Knyazev et al., 2025) offer compelling opportunity overcome the aforementioned limitations in LLM merging. These approaches operate directly on pretrained model weights as the input data modality. By projecting parameters from diverse models into smooth, unified latent space of consistent dimensionality, we can inherently mitigate the challenge of architectural mismatch. Specifically, these parameter signals can be embedded into using powerful generative models such as Variational Autoencoders (VAEs) (Kingma & Welling, 2013), normalizing flows (Kobyzev et al., 2021), or diffusion models (Wang et al., 2025). While these innovative latent-space methods have demonstrated considerable success in vision and other domains, their application to the encoding and subsequent merging of pretrained LLM weights remains a largely unexplored frontier.
representations
We propose L.-Merge, novel framework that fundamentally shifts the merging process from the raw weight space to learned latent space. This paradigm enables both homogeneous and heterogencous merging by design: (i) Enabling single-model augmentation: A generative model can learn the latent manifold of single LLM parameters, facilitating "merging" operations within this latent space. This allows for the exploration of variations, specializations, or the enhancement of capabilities derived solely from the original model, obviating the need for an external second model. (ii) Facilitating heterogeneous integration: By projecting diverse LLM architectures into common, fixed-dimenssooal latent space, the stringent constraints of architectural homogeneity are significantly relaxed. Merging operations then occur within this unified latent representation, enabling the seamless integration of knowledge from models with differing widths, depths, or even fundamental architectural designs, our method operates solely on the tensor data within the latent space. In summary, LS L.-Mergee advances L.M merging through the following key contributions:
Weight statistics that matter for encoding. We show that LLM weights exhibit low variance with heavy tails, that could significantly affect the choice of encoding network.
Merging LLMs in latent space. We propose LS-Merge, a novel latent-space merging methodology that enables merging LLMs in their weights latent space.
Heterogeneous merging. We introduce a dimensionality-mann matehingg projection and OT-based latent alignment that enables interpolation between models of different depths widths.
2

---

Published as a conference paper at ICLR 2026
Consistent empirical gains. On Gemma and LLaMA pretrained models and LeRA-experts merging, our latent space merging method outperforms existing merging methods and remains robust under heterogeneity.
RELATED WORK
Model averaging The simplest merging approach directly averages weights. Model Soup (Wortsman et al., 2022) showed that averaging fine-tuned checkpoints from the same initialization improves generalization. Extensions include Uniform Soup (Wortsman et al., 2022) for LoRA weights and spherical linear interpolation (SLERP), which interpolates on the unit hypersphere but remains pairwise-only method that merges only two models at a time and requires identical architectures.
Interference-awall techniques Merging models from different tasks can cause misalignment. Yadav et al. (2023) uses trimming and sign aligmment, Yu et al. (2024a) sparsifies and rescales fine-tuned deltas, and Yu et al. (2024b) combine both. Greedy Soup (Wortsman al., 2022) and EvolMerge (Ak- iba et al., 2025) treat merging as an optimization problem, while Task Arithmetic (Ilharco et al., 2023) models task vectors parameter space. These methods require aligned architectures and incur computational cost.
Modular assembly Instead of merging weights, some approaches combine modules experts. Model Stocks (Jang et al., 2024) and LoraHub (Huang et al.. 2023) aligns LoRA adapters, Pack of LLMs (Mavromatis et al., 2024) learns expert routing. and cBTM (Gururangan et al., 2023) merges task-specific experts via unsupervised domain discovery. Modular systems are flexible but increase inference cost and do not unify knowledge into single parameter set.
All prior methods operate in weight or module space and assume architectural alignment. In contrast, we propose latent-space model merging, which generalizes beyond these constraints.
3 LS MERGE
We present LS-Merge, a framework for performing model merging in latent space. As illustrated in Figure encode pretrained LLM weights, operate on their latent representations (including alignment when heterogeneous), and decode back to parameters to obtain the merged model.
3.1 EXPLORING WEIGHT DYNAMICS LLMS
To motivate our design, we analyze the distributional properties of transformer submodules in Gemma- 3 (Team et al., 2025) and LLaMA-3. (Grattafiori et al., 2024). We compute the first four moments (mean, variance, skewness, and kurtosis) for self-attention layers (q-proj, k-proj, and o o-proj) and MLP layers (up_proj, down_proj, and gate_ gate_proj). Table reports the statics stands for per moment with full details in appendix Table 9. Weights exhibit near-zero means, low variances, and small positive skewness, but markedly high kurtosis earlier layers, especially in self-attention projections. High kurtosis, i.c., leptokurtic distributions, indicates sharp peaks with heavy tails: rare, maggitude parameters that are likely functionally important. This contradicts Gaussian assumptions used in prior work (Si al., 2025) and suggests that encoders must preserve tail events rather than over-regularize toward a narrow Gaussian. We observe this pattern consistently across sizes within both families.
Theoretical Compressibility of LLM Weights As the LLM weights distribution shows heavy tails and low intrinsic variance, we investigate whether simple VAE can compress LLM weights effectively. Let W E Raxm be a layer matrix and let w = VOC (W) E TR denote the row-wise concatenation (flattening) of W with D nm. Empirically (Fig. 2), the top where {A:} are
min(n, (in(n,m) m)
principal components capture nearly all variance, i.e., Σ A/ X; the eigenvalues of the empirical covariance of (equivalently, proportional to squared singular values of W). By Likkrt-Young thcorem (Eckart Young. 1936), the best rank-r approximation W, satisfies ||W W, E.sr X, and can be stored with O(r(n + m)) parameters (e.g.. U, x V, Rmxr). Thus, weights concentrate near www-imensional set. Assuming the collection of flattened weights W с R concentrates near a smooth d-dimensional manifold M with d & D, manifold embedding results (Bengio al., 2012; Lahiri et al., 2016) imply the existence of
3

---

Published as a conference paper at ICLR 2026
layers
laama.-2-3bbit
gemma---II-i
gemma-3--b-it
Iracar
var:
skew
mean
Yar.
var:
skew
kurt.
0.0017
0.0192
8.4032
0,0002
0.0031
0.0500
15,0505
0.0015
0.0501
15,2010
0.0013
0.0191
7.3438
0.0002
0.0030
0.0450
9.8347
0.0015
0.0267
7.3832
self_attn
0.0012
0.0178
6.2172
0.0001
0.0030
0.0389
9.0363
0.0014
0.0255
7.2277
0.0011
0.0104
5.4477
0.0001
0.0030
0.0288
8.6496
0.0014
0.0172
6.0731
avg (self_attn)
0.0005
-0.0002
1.4342
0.0012
-0.0009
0.0005
0.0009
2.6900
0:0005
0.0093
5.4740
0.0002
0.0010
0.0167
8.7665
0.0004
0.0266
6.3297
0.0005
0.0080
4.1465
0.0002
0.0010
0.0090
3.1514
0.0004
0.0155
5.9670
mip
0.0004
0.0075
2.6694
0.0002
0.0010
3.0577
0.0130
2.7454
0.00004
0.0074
2.4364
0,0002
0.0010
0.0071
3.0113
0.0004
0.0094
2.4371
avg (illp)
0.0003
0.0003
0.8435
0.0006
0
1.1739
0.0003
0.0006
1.0807
Table 1: Laser-wise distribution statistics of model parameters for three instruction-tunet LLMs: llama3-2-3b-it, gemma-3-1b-it, and gemma-3-4b-it. For each model and block type (sel (self_attn and mlp), we report the first four moments of the flattened weight tensors: mean, variance, skewness, and excess kurtosis. Per-row entries list representative layers in cach block; the avg aggregates across all layers for that block and model. Means are near zero and variances are small, while skewness and especially kurtosis (often 5 and up ~ 15) indicate pronounced heavy tails and asymmetry. These non-Gaussian, heavy-tailed statistics motivate encoders that preserve rare but nigh-magnitude outliers.
0.07
0.12
Exglained Vartance Ratio
Explained Varfance Ratio
0.030
0.06
Explained Varianna
0.10
0.025
0.05
0.08
0.04
0.06
0.03
0.015
0.04
0.02
0.010
0.02
0.005
1
3
1
6
10
3
5
8
2
4
6
8
9
10
Principal Componant
Principal Component
Princtal Component
(a) Llama3-2-3b-it layer k.proj
(b) Gemma-3-Ib-in layer 0 k_proj
(c) Gemma-3-4b-in layer k.proj
Figure 2: Plots the PCA explained-variance ratio for first projection weight matrices per model from three LLM checkpoints: Llama-3-2-3b-in Gemma-3-lb-it and Gemma-3-4b-it Additional results are reported in the appendix A. For each model, we show the four projection matrices of self-attention (q. k. v, The sharp drop after the leading principal components highlights low-rank structure that is consistent across architectures and model sizes.
PR R with k O )) < D that preserves pairwise distances on M within (1 + E). where V bounds manifold volume and its reach. This confirms that there exists projection map to a lower dimension which can be approximated by an encoder A VAE encoder fo R - R with decoder Jø R + R can approximate such compressive embeddings while modeling uncertainty, justifying our latent compression.
3.2 LLM WEIGHTS PREPROCESSING AND ENCODING
We first standardize pretrained LLM weights of varying shapes and store the normalized tensors offline. We then learn a compact latent representation with transformer- VAE, chosen for its strong generalization to unseen checkpoints and faster training than convolutional alternatives at comparable parameter count.
Preprocessing Sequence For each layer, flatten its weight tensor to R then zero- pad to Lp = c/ e and partition into n = Lp/c non-overlapping chunks (r of size c. A batch becomes X E Bxxxe Each chunk is embedded to Xemb E R2 Bxxxd and passed through a transformer encoder Es with optional token downsampling. The latent is
BxEa
(pooled over tokens)
Eo (Xemb))
BBaxx
(token-wise)
4

---

Published as a conference paper at ICLR 2026
The decoder D is trained to reconstruct chunked weights from the latent We optimize the 3- 3-VAE objective in eq.
== g w)) log po (o [ z)] + [KL(go(2] pp()))
(I)
with p(=) standard Gaussian and fixed B3
Stabilizing Training on Heavy-Tailed Weights Since LLM weights distribution has low variance and heavy tails (Section 3.1; Appendix 5), training VAES on these weights can lead to collapse the early training stage. We use (i) transformer blocks (Vaswani et al., 2017) 7) in Egg, D& for long-range coupling across chunks and (ii) two-stage curriculum: first train a deterministic autoencoder (KL off) convergence, then enable the KL term and fine-tune structure the latent space without sacrificing fidelity. The VAE performance is measured by the performance of the reconstructed weights when used to initialize the corresponding architectures.
3.3 LATENT SPACE ALIGNMENT AND MERGING
Self-Merging and Homogeneous Merging Self-merging encodes a single model and draws mul- tiple latent codes from its posterior (or the prior) to combine them, which is equivalent merging homogeneous models whose per-layer embeddings share the same dimensionality. To be specific, for two checkpoints with weights W., W, RRN, encode za F(Wa) 2) and 2 F.W)), and linearly interpolate them as 2 = (1 2) + A2b for À E [0, 1]. We obtain the weights by decoding the interpolated latent W (2x). Empirically, latent-
interpolation better preserves functional coherence than direct weight-space mixing, and common merge operators, for example, model soup or task arithmetic (Wortsman et al., 2022; Ilharco et al., 2023), transfer naturally by applying them to {Za, 26) before decoding.
Heterogeneous Mapping (depth/width mismatch) When two architectures match, layer by layer, in the ber of weight chunks, we employ a single VAE to embed all layers into common d-dimensional latent space. If the per-layer number and chunk counts differ, we instead deploy separate encoders for each architecture. Let the source have n, layers with size M, and the target n, lay- ers with size N. We embed per-layer latents fixed dimension d and rescale the source so that total capacity matches the target:
Source
Target
r 74 N
Merged
tisc, mapped) R
Z
E ga.xd
n2 M'
Figure 3: Visualization of embeddings of
This proportional aligmment across depth and width stan- dardizes each architecture to a common per-layer latent dimensionality, enabling interpolation and the use of stan- dard merge operators. However, when VAEs are trained separately per architecture (or under different training regimes), their latent distributions can differ, so equal shape does not guarantee comparability.
Gemma models and OT-aligned latents. The merged latent partially overlapped with the target latent.
Optimal Transport Alignment. While latent encoding standardizes per-layer dimensionality, it does not guarantee that two models' latent representations are geometrically compatible. As shown in Appendix C (Figure 9a), homogeneous models (c.g., checkpoints from the same pre- training run) exhibit overlapping latent support and often satisfy Linear Mode Connectiviry, making linear interpolation well-behaved. produce latent distributions Zxce Zg. that lie on disjoint manifolds with different covariance structures and density profiles (Figure. 9b). Interpolating between such misaligned latents could produce low performing weights that fall outside the target decoder's valid manifold and degrade on down stream tasks). To address this, we treat heterogeneous merging as manifold registration problem and align the source latent distribution to the target using Optimal Transport (OT) (Villani, 2009; Santambrogio, 2015). OT provides a principled framework for finding Zyg that minimizes geometric distortion while reshaping one distribution into the
contrast, heterogeneous models such as Gemma and Llama
map T Z
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Heterogeneous LLM Parameter Merging in Latent Space
Require: Source weights Waes target weights Wigt, encoder-decoder (E, D)
Ensure: Merged weights W merrged
Extract per-layer groups Lise: Lugii set N min(| Lgge and define pairs (1lee 1 y
2: for 1 to N do
3: Flatten & chunk ware Wigt encode (1) + -E(w)))
(1)
Aig < E(wlg))
4: Proportional mapping to fixed d: obtain 2uie Zud E Baaxd
5: OT alignment: Zue
OT-ALINN Lild 2.T
(solve equation 2)
6: Interpolate latents: - (1 1) Zu
Z(J)
+
7: Decode & assemble: w (5)
D (2 place into W mergod
merged
8: end for
9: return W Wmerzed
other. Formally, we solve the Monge problem under the 2- E-Wasserstein distance 2:
/ ||= ||=- ||=-T<>)]B (z) s.t. Ty Musce Hiiht
T = arg min T
(2)
where Hyee denotes the pushforward measure induced by T. This ensures that applying T to source latents produces samples distributed as the target. While the general Monge problem is computationally intensive, it can approximate each layer's latent distribution as a high-dimensional Gaussian defined by its empirical mean μs, and covariance Σ., Under this assumption, the optimal transport map admits a closed-form affine solution: Zare T" (Zarc)
Pt + A (Zare (a), where
A A= 1/2 1/// Σ, / 1/2 1/ This transformation aligns both the mean and covariance of the source latents to the target, effectively registering the two manifolds and removing the geometric mismatch. In practice, we use existing OT library from Flamary et al. (2021; 2024) in our work.
After alignment, the transported latents Zuc and the target latents Zu share a common support, and we perform interpolation in this aligned space: z = (1 A) Zngs + A 2 Because 2 now lies within the target decoder's valid density region (Agustsson al., 2019), the decoded weights remain stable and functional, enabling robust cross- aresstecchee merging even between disparate model families. The overall process for heterogeneous LLMS merging is summarized algorithm We extend latent-space linear merging from pairs to N models (e.g., LoRA experts) via convex
barycenter. For each expert W,, we sample latent codes (m)
(2 W.). We then merge experts by barycentric interpolation, Amerged = Ein1 A;zi, 2 Ee1 A; = 1, and decode the unified
weights W mergedd D( (mmerged)) Uniform and Greedy Soup correspond to different choices/updates of {Ai).
4 EXPERIMENTS
General Setup We evaluate space merging on Gemma-3-1B-it Gemma-3 -3-maa-3------ Llama-3 1.LB-instruuct Llama 1-aaa-2--- 2-, and benchmark latent-space expert fusion against weight-space merging using 10 LoRA experts on Gemma- 7B- -tt.
Datasets and Tasks We first use subset dataset used by Feng al. (2024b), such as language under- standing (MMLU (Hendrycks et al., 2021), MMLU-pro (Wang et al., 2024b)), commonsense reason- ing (HellaSwag (Zellers et al., 2019)), math (GSM8k (Cobbe et al., 2021)), and knowledge-intensin tasks (Knowledge Crosswords (Ding et al., 2024), NLGraph (Wang et al., 2024a), TruthfulQA (Lin al., 2022), AbstainQA (Feng al., 2024a)).
Training data consist of pretrained weight snapshots for C -3-1B-it and Cemma-3-4B- plus LoRA experts from Feng et al. (2024b).
Baselines We compare against reference-free weight space methods: spherical linear interpolation (SLERP), uniform soup (Wortsman al., 2022), greedy soup (Wortsman et al., 2022), data-level

---

Published as a conference paper at ICLR 2026
MMLU
MMLU-pro
HellaSwag
Gsm8k
Gemma-3-4b-it
53.10
20.90
47.40
29.90
VAE
54,10 + 0.36
20.80 0.20
49.03 0.70
31.27 + 0.55
LS- LS-Merge
54.20 + 0.00
21.02 0.03
50.10 0.00
32.20 + 0.05
Gemma-3 Ib-it
32.20
7.10
28.70
16.90
VAE
32.60 + 0.26
7.60 + 0.56
28.57 0.12
16.77 + 0.12
LS-Merge
35.13 + 0.02
10.30 0.20
31.16 0.14
17.50 + 0.01
Table 2: Benchmark scores for pretrained model, VAE, and LS-Merge.
merging, the raw pretrained checkpoints, and the best single expert per task. This excludes approaches that require access an unmodified base reference model.
Evaluation Protocols We evaluate compression and merging in four scenarios:
Self-Merging: sample multiple codes from one model's latent distribution and merge them.
2. Expert Merging: merge LoRA experts in latent space vs. weight space.
3. Cross-Architecture Merging: align and merge models with various depths and widths.
4. Ablation study: reconstruction or compression behavior and generalization to unseen checkpoints.
Weight-Encoding Models We evaluated Transformer- VAE and ConvNet- VAE (Soro et al., 2025). We used AdamW (Ir -Ir=18 le-4, weight decay = le-5) with a cosine-to-zero schedule. The checkpoints are from Hugging Face, and further details are given the supplement. The learning rate was chosen on the basis of hyperparameter tuning.
4.1 SELF-MERGIN FOR ENHANCED PERFORMANCE
We investigated a self-merging technique designed to enhance single model's performance by exploring its learned latent distribution. The process involves encoding a model, sampling multiple latent codes from its posterior distribution, merging these codes into a single representation, and decoding it back into the parameter space. For this experiment, we used a single Transformer VAE with six encoder and decoder blocks, trained jointly weights from both Gemma-3-1B- 3-1t-it and Gemma-3- 41--it--- it with the compression ratio held constant at 2. As shown in Table 2, this approach yields average performance improvement of 4% over two key baselines: the original base model and standard VAE reconstruction from a single latent sample. Notably, the gains are more pronounced on the smaller model, consistent with it having tighter capacity constraints.
4.2 MERGING LLM EXPERTS LATENT SPACE
Next, we evaluated the primary application of our work: combining specialized LoRA experts. We compared our latent space approach against traditional weight-space interpolation methods using the experts from Feng et al. (2024b). In our method, each expert is encoded, their latent representations are merged, and then the resulting latent vector will be decoded into single fused model. As shown in Table our latent-space fusion consistently outperforms all weight-space baselines, including both linear and SLERP interpolation across uniform and greedy soup. We found this advantage stems from increased robustness. For example, greedy soup is highly sensitive to initialization, the checkpoint with the best validation accuracy often fails to yield the best test performance. By sampling multiple latent codes for each expert before merging, our method explores the learned parameter distribution instead of relying on single point estimate, creating more robust and effective combinations.
4.3 COMPARISON TO REPRESENTATION-
METHODS
To assess the robustness of our latent space approach, we benchmark against leading representation- merging methods; Task Arithmetic (Ilharco al., 2023) and Activation-Informeed Merging (AIM) (No- bari et al., 2025), which operate on model activations rather than parameters. contrairily to the previous experiments, in this setting the baselines. For this comparison, we merged L alama- -aaa--211 2-12-11 13B models tine-tuned on distinct do- mains, utilizing single VAE trained on the combined weights of all constituent models. The results
Im-eval tool (Gao et al., 2024) for fair comparison with
7

---

Published as a conference paper at ICLR 2026
MML.U
MMI.Uppro
HellaSwag
Gsmök
TruthfulQA
NI.Graph
K-Crossword
AbstainQA
Best expert
45.7
14.3
46.6
26.1
32.4
51.7
32.7
-10.8
model
48.8
18.1
47.5
28.0
Data Merge"
44.5
52.7
14.3
10.7
42.3
Uniform Soup
49.7
19.4
54.0
7,9
31.2
47.5
29.6
-0.1
SLERP(t=0.45 45)
52.5
18.8
50.4
25.5
28.7
49.8
30.0
-0.2
Greedy Soup
50.8
22.1
54.6
23.9
31.9
52.9
28.0
3,3
Dare-Ties
49.1
18.8
53.7
7.3
28.2
52.8
29.0
LS (S-Merge(lerp)
54.7
21.6
58.1
28.1
33.0
53.1
35.6
2.0
LS- (srMeggosssop)
56.0
22.2
24.2
32.0
56.1
35.2
Table 3: Results on merging expert LoRA weights in the raw weight space and the latent space.
Method
MMI.U
IFEval
MBPP
HumanFyal
GSM8K
base
52.18
25.10
27.80
17.07
4.20
code instruct
52.91
26.25
31.60
17.07
24.10
53.41
35.67
34.80
26.83
43.40
code + instruct (Task Arithmetic)
52.18
25.10
34.40
26.83
4.20
code + instruct (AIM)
54.18
32.00
36.00
29.27
46.20
code + instruct (LS-merge)
55.07
36.41
36.02
28.14
44.12
Table 4: Comparison of our latent space merging (LS-mergee with Task Arithmetic and AIM
Llama-2- -2mm- fine-tuned models.
presented in Table 4 demonstrate that our method is highly competitive. It achieves performance comparable to the state-of-the-ard AIM and substantially outperforms Task Arithmetic. This finding is significant, it shows that latent weight-space approach match the performance of prominent methods that require access to model activations.
4.4
MERGING
Our latent-space approach supports merging across substantial architectural gaps. We study two settings: (i) intra-family (different sizes within Gemma) and (ii) cross-family (Gemma LLaMA).
Intra-family (Gemma-3-4B-it - Gemma-3-1B-it), Direct latent interpolation between indepen- dently trained models unstable (Fig. 4b). Aligning the larger model's latents the smaller model's distribution before interpolation yields consistent gains across the mixing range. As shown in Fig. 4a, small injections from the source (A E [0.05, 0.20)) deliver the best improvements over the Gemma-3- 3-n---3-BBii- baseline.
Cross-family ----M----2-----insttuue + Gemma-3-1B-in Family-level transfer is more challeng- ing: baseline parameter/latent mixing without alignment degrades performance. Table 5 shows that distributionally aligned latent merging recovers and surpasses these baselines, modest interpolation weight (A = 0.1) achieves the best overall scores. The evaluation for cross family evaluation is performed using im-eval for simplicity and also due to some issues with llama model when using the previous evaluation code.
Takeaway. Matching latent dimensionality insufficient for heterogeneous merges, aligning latent distributions is crucial. Once aligned, single knob À reliably controls how much capacity is injected from the source into the target.
5 ABLATION STUDIES
To better understand the properties of our method, we conduct two targeted ablation studies. In this section we use Im-eval for evaluation unless stated.
5.1 COMPONENT CONTRIBUTIONS IN LATENT MERGING
We first analyze how different submodules contribute to the merged model's performance by merging MLP layers only, self-atter self-attention layers only, both jointly. The results in Table 6 show that merging MLP layers alone provides modest gains, while merging attention layers alone degrades performance. Optimal results are achieved by merging both, indicating that MIP and self-attention parameters
8

---

Published as a conference paper at ICLR 2026
Base-medel Mixod-latent
Ne-OPT With-OFT
225
20
test valigro)
test(pro)
Vallig
TestAvg
Vallo.5
Valo.ai
Test
Heterogeneous latent space merging.
(b) Varying coefficient MMLU.
Figure 4: (a) Accuracy gains obtained by latent space merging of Gemma-3-44 3-4B--4------ 4B-it with Gemma-3-1B-1 Bars show performance the validation (val) and test splits of the MMLU and
MMLU-PRO benchmarks (b) Impact of the merging coefficient t on validation and test accuracy in latent-space merging.
Strategy WinoGrande
ARC-C
HellaSwag
Strategy
WinoGrande
ARC-C
MMLU
Base 56.83
42.78
49.07
Basc
56.83
42.78
40.76
ML.
56.84
43.89
41.02
OT only 51.13
34.25
48.50
Attention
56.67
40.23
39.80
OT interp. 57.75
43.34
50.10
Attention + MLP
57.75
43.34
42.10
Table 5: Downstream accuracy WinoGrande, Table 6: Ablation:
Merging subsets of layers. ARC-Challenge, and HellaSwag for different align- MLP-only outperforms on ARC-C, combining ment strategies with À 0.1.
MLP and Attention achieves the best results.
encode complementary functional knowledge, and that altering one without the other can disrupt learned ro-adaptations.
5.2 VAE GENERALIZATION AND THE COMPRESSION TRADE-OFF
Next, we assess the VAE's zero-shot generalization by training it on Cemma-3-4B- and evaluating it LLaMA-3. 1.0MM--322-B---- Table reveals a clear trade-off between compression and generalization. At low compression ratio (r 1.6), the VAE maintains strong performance both unseen models. However, performance degrades substantially at higher ratios (r = 2, 4). This suggests that while the VAE learns transferable representations of weight structures, higher compression ratios lead to posterior collapse due to the fact most of the data sample are cluster aroung zero.
unseen models: an in-family Gemma-3-1B-it and cit-of-faamlly
5.3 LINEAR SUBSPACE VS. NON-LINEAR MANIFOLD LEARNING
In this section investigate the use of incremental PCA for weights encoding gemma-3-1 b-it compare to VAE based en coding. Although Section 3.1 showed that LLM weight matrices exhibit low-rank structure, this does not imply that the space of functional parameters forms linear subspace. To assess whether linear methods are sufficient, we compare our non-linear Transformer- VAE against PCA across compression ratios E (1.6, 2.0, 4.0) (Table 8).
PCA collapses functional performance. Across all ratios, PCA-reconstructed models regress to near-random accuracy on MMLU (=25..5% at 1.6) and exhibit global drop across benchmarks (e.g., ARC-C: 42.41% + 27.65%). Notably, performance is equally poor at r = 1.6 and = 4.0, indicating that the failure is not due insufficient latent capacity but a structural mismatch: the of valid pretrained weights does not reside in a linear subspace.
VAE preserves the functional manifold. In contrast, the LS-Merge VAE retains near-original accuracy at all compression levels. At r 1.6. it recovers 96% of the base model's MMLU performance (39.89 99%%%% vs. 41.44%) and even improves Winogrande (56.64% vs. 55.41%). Remarkably, VAE reconstructions remain stable at r = 4.0, whereas PCA has already collapsed at 1.6. This indicates that pretrained weights lie on a non-linear manifold that requires expressive encoders and decoders to model its curvature. Linear projections such PCA cannot preserve the structure of the pretrained weight manifold and produce functionally invalid models even under mild compression. The VAE's non-linear latent geometry is therefore stylistic preference but geometric necessity for faithful reconstruction, compression, and downstream operations such as interpolation and merging.

---

Published as a conference paper at ICLR 2026
Model
Winogrande (5-shot)
ARC-Challenge (25-shot)
HellaSwag (10-shot)
MMLU (5-shot)
Gem-a-3-IB-it (base)
56.83
42.78
49.07
40.76
(LMMA-.2-IB-ii(buusu)
61.56
41.11
61.62
46.55
Gemma-3-IB-it (VAE, 1.6)
$6.67
42.83
47.31
39.98
3. (VAF, 1.6)
61.25
41.47
60.80
46.06
Gemma-3-1B-itt (VAE,
56.67
38.23
38.88
32.22
LLaMA-3.2-IB-44 (VAE.
59.43
37.12
55.99
39.73
Gemma-3-1B-it (VAE,
46.49
28.24
25.66
LLaMA-3.--BB-iVAAE...
49.20
26.28
25.70
26.76
Table 7: Accuracy of VAE models (trained
Genma-3-4B-ir when evaluated
Gemma-3-1m 3-mnn- and 1..aMM-..211B-- Performance remains stable for = 1.6, but de- teriorates the compression factor increases.
Table 8: Functional Reconstruction Fidelity vs. Compression Ratio. Zero-shot accuracy comparison standard benchmarks (MMLU, HellaSwag, Winogrande, ARC-C) for Gemma-3-18 ---mma---1B-i.. We compare our non-linear Transformer- VAB (LS--Mrree) against linear PCA baseline at compression ratios (1.6, 2, 4). While the VAE maintains strong performance linear compression suffers significantly as the bottleneck tightens, validating the need for non-linear manifold learning.
1.6 and
Method
Ratio (r)
MMLU
HellaSwag
Winogrande
ARC-C
Gemma-3- (emma-3-Ib-it)
1.0x
41.44 + 0.00
49.05 0.01
55.41 + 0.03
42.41 0.02
PCA (Linear)
1.6x
25.50 + 0.37
25.56 + 0.04
50.12 + 0.01
27.65 + 0.01
LS-Merge VAE
1.6x
39.89 = 0.01
48.57 0.25
56.64 + 0.15
41.64 - 0.01
PCA (Linear)
2.0x
24.12 + 0.00
25.27 + 0.12
46.27 = 0.01
26.24 0.32
LS-Merge VAE
2.0x
39.80 = 0.00
49.29 = 0.10
54.14 = 1.02
42.32 0.21
PCA (Linear)
4.0x
24.13 + 0.15
24.79 + 0.23
49.57 + 0.01
25,89 + 0.23
LS-Merge VAE
4.0x
39.83 + 0.00
49.30 0.21
56.06 + 0.15
42.75 0.20
DISCUSSION
Limitations Despite its strong performance, training the weight-encoding VAE at high compression ratios can be challenging(mode collapse). largely due to the heavy-tailed nature of LLM weight distributions. Nonetheless. remains highly effective when utilizing an overcomplete latent space, which eases optimization while successfully unfolding the weight manifold.
merging paradigm does not strietly require tight bottleneck; it
This work opens several avenues for future research. Promising directions include developing more efficient, property-aware weight encoders (Navon et al., 2023; Zhou et al., 2023) improve scalability. Additionally, extending our framework beyond simple merging via iterative latent barycentric interpolation (Akash al., 2022: Pennec, 2017) could unlock complex uni-model compositions, paving the way for advanced multimodal generation.
CONCLUSION
In this work, we introduced LS-Merge, a novel framework that reimagines model merging by operating in learned latent space of model parameters. By mapping weights to a continuous manifold and critically employing OT for principled alignment, our method successfully merges models with heterogeneous architectures, overcoming fundamental limitation of prior weight-space techniques. Our comprehensive experiments demonstrate that this approach not only excels at standard expert fusion but also enables robust cross-scale and gross-family model merging for the first time. LS- L--Mrrge establishes a scalable and architecture-agnvee paradigm for model composition, paving the way for more flexible and efficient reuse of pre-trained models.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reliable and reproducible results, we have provided detailed experiment settings in the Appendix. We plan to open-source our implementation here: attps://githun com/ scrobedio/ls Lemeed/////-mere/////
ACKNOWLEDGMENTS
This work was supported by the Institute for Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (RS-2019-1110077, Artificial Intelligence Graduate School Program (KAIST)): and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RSS2023-00256255) This research was also supported by a grant from the Korea Machine Learning Ledger Orchestration for Drug Discovery Project (K-MELLODDY), funded by the Ministry of Health & Welfare and the Ministry of Science and ICT, Republic of Korea (Grant No. RS-2022 2--22220006087000 70): and by the Institute of Information & Communications Technology Planning & Evaluation (ITTP) under the Open RAN Education and Training Program (IITP-2024-B) 1---0024---22----000000 RSS2002-000429888) funded by the Korea government (MSIT).

---

Published as a conference paper at ICLR 2026
REFERENCES
Eirikur Agustsson, Alexander Sage, Radu Timofte, and Luc Van Gool. Optimal transport maps for distribution preserving operations on latent spaces of generative models. In International Conference on Learning Representations, 2019.
Aditya Kumar Akash, Sixu Li, and Nicolás García Trillos. Wasserstein barycenter-based model fusion and linear mode connectivity of neural networks, 2022. URL URLhttps: http:p::///axxv..oo/// abs/ fos/22100 06671.
Takuya Akiba, Makoto Shing. Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195-204 January 2025. ISSN 2522-5839. doi: 10. 10.10388/4226--44-000777
URL http: /ttp://dxxxx /tx./xxdooo// doi. wwwpddoo.og//......oooooo/ 10338/
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15:1798-1828. 2012.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint wrXiv:2110.14165 2021.
Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric knowledge reasoning with large language models, 2024. URL https:/ / arxiv. org/ abs 0. 01 01290.
Carl Eckart and G. Marion Young. The approximation of one matrix by another of lower rank. Psychometrika, 1:211-218, 1936.
Shangbin Feng, Weijia Shi, Yike Wang. Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't 't hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume
1: Long Papers), August 2024a.
Shangbin Feng, Zifeng Wang, Yike Wang. Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, Chen- Yu Lee, and Tomas Pfister. Model swarms: Collaborative search adapt Ilm experts via swarm intelligence, 2024b. URL
https: / aarivv.iivv org/ org/abs/////ooooooooo/ooo////oer 2210 11163.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78))1-8, 2021. URL http:/ jmlr. org/ papers/ 20122/20-455 451 html.
Rémi Flamary, Cédric Vincent-Cuaz, Nicolas Courty, Alexandre Gramfort, Oleksii Kachaiev, Huy Quang Tran, Laurene David, Clément Bonet, Nathan Cassereau, Théo Gnassounou, Eloi Tanguy, Julie Delon, Antoine Collas, Sonia Mazelet, Laetitia Chapel, Tanguy Kerdoncuff, Xizheng Yu, Matthew Feickert, Paul Krzakala, Tianlin Liu, and Eduardo Fernandes Montesuma. Pot python optimal transport (version 0.9.5), 2024. URL https: / //g/gttth//e githhu..... hwwp//.w.ooooooooooooooo......oe
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: /et::///eendde /wzzzzzoooooooooooooooooooooooo. www./dooooo//oooooooooooooooooro records dss 12608602.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem
12

---

Published as a conference paper at ICLR 2026
Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika lyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bushlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whimey Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allic Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic. Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton. Catalina Mejia. Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty. Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna lakshninarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan
13

---

Published as a conference paper at ICLR 2026
Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo. Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez. Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala. Stephanie Max, Stephen Chen. Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad. Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Veto,o 7ef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llaaa 3 herd of models, 2024.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. arXiv preprint arXiv:2303.14177, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICIR), 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: /oppnreeeiiieeeeeeeee ew. net. forum? iid= 6t.Kkff-
Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just a few fine-tuned models. In Proceedings of the European Conference Computer Vision, 2024.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL https: / /app..pi/.// semant icscholar. org/ Co:pcocppiiiii111a00
Boris Knyazev, Abhinav Moudgil, Guillaume Lajoie, Eugene Belilovsky, and Simon Luciste-Julien. Accelerating training with neuron interaction and nowcasting networks. In The Thirleenth Inter- national Conference on Learning Representations, 2025. URL https: /oppoppeeevvveeeeeeeeeeeeeeeeeee ewe.ew..ne n forum? dd=CUFII IILIIIIII 6hEG.
Ivan Kobyzev, Simon J.D. Prince, and Mareus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 (11:3644-399) 3979, 2021. doi: 10. 1..110//TPMM..200.29999999
14

---

Published as a conference paper at ICLR 2026
Subhaneil Lahiri, Peiran Gao, and Surya Ganguli. Random projections of random manifolds, 2016. URL
org/ abs/160 04331.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL httpps://axxi.. http://awwvvv....aaaaaaaa/ wwt./o//////./o.oooo.o...o....st abs /ww.. 07958.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ilms: Model fusion at test-time via perplexity optimization. ArXiv, abs/2404.11531, 2024. URL https://apil
269188153.
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equiv- ariant architectures for learning in deep weight spaces. arXiv preprint MrXiv:2301.127800 2023.
Amin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, and Navid Azizan. iitivaationinformeer merging of large language models, 2025. URL https: //arxiv. wwg.
02421.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A. Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints, 2022. URL wwt.h:tpp///aaet org/abs/// 2209.. 12892.
Xavier Pennec. Barycentric subspace analysis on manifolds, 2017. URL htthtsp::/aaxxioooo abs/1607 02833.
Filippo Santambrogio. Optimal Transport for Applied Mathematicians: Calculus of Variations, PDEs, and Modeling, volume 87 of Progress in Nonlinear Differential Equations and Their Applications. Birkhääser, 2015.
Konstantin Schürholt, Boris Knyazev, Xavier Giró Nieto, and Damian Borth. Hepeer-epssssnntations as generative models: Sampling unseen neural network weights. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: htt///ppeevveieeeeeee net forumfirumimo wow.:roinddeeE////eHHH.
Konstantin Sch" urholt Michael W. Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Proceedings of the 41st International Conference Machine Learning (ICML), 2024.
Chongjie Si, Jingjing Jiang, and Wei Shen. Unveiling the mystery of weight in large foundation models: Gaussian distribution never fades, 2025.
Bedionita Soro, Bruno Andreis, Hayeon Lee, Wonyong Jeong, Song Chong, Frank Hutter, and Sung Ju Hwang. Diffusion-hased neural network weights generation. In The Thirteenth International Conference on Learning Representations, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alcxandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng. Alexander Kolesnikov, Alexci Bendebury, Alvin Abdagic. Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimezak- Pluciíska. Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan
15

---

Published as a conference paper at ICLR 2026
Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabíla Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq lqbal, Shashir Reddy, Shruti Sheth, Siim Pöder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra. Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Cédric Villani. Optimal Transport: Old and New, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer, 2009.
Heng Wang, Shangbin Feng. Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?, 2024a. URL https: //rxiv.. axg/abs/2305. 10037.
Kai Wang, Dongwen Tang, Wangbo Zhao, Konstantin Schürholt, Zhangyang Wang, and Yang You. Recurrent diffusion for large-scale parameter generation, 2025.
Yubo Wang, Xueguang Ma, Ge Zhang. Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 24Xiv:2406.01544 2024b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lpess, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith. and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, pp. 23965-23998, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES- Resolving interference when merging models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL. https: http::///eeeeeeeeeeeeeeeeeeeeeep net forum? d=
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: ab- sorbing abilities from homologous models as a free lunch. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 '24 JMLR.org. 2024a.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In International Conference on Machine Learning, 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https:::: /ttpppp:////i....////// org/ /1 / 905 07830.
16

---

Published as a conference paper at ICLR 2026
Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL 1d= EmYmmNPPhP
net ntt.:oorutttttt
A
APPENDIX
IMPACT STATEMENT
We introduce VAE-based latent-space merging technique for pretrained LLMs that encodes weights into compact codes, blends them, and decoding new higgh-erfformncee parameters. This enables fast, compute-efficiet model customization and smooth interpolation across architectures. However, blending latent codes can unintentionally merge biases or toxic behaviors, obscure the origin of capabilities, and be misused to graft malicious functionality. We advocate for rigorous bias / toxicity audits, transparent provenance tracking, and clear reporting guidelines to ensure responsible deployment.
A EXTENDED DISTRIBUTION ANALYSIS
Section 3.1 characterized layerwise weight statistics to inform encoding and merging. Here we extend that analysis with (i) complete per-layer distribution plots, including all MLP layers for GEMMA-3- IB-IT, GEMMA-3-4B-I and LLAMA-3.2-11 1-33--NSTRUCT. and (ii) cumulative variance-explan curves from PCA on those MLP layers.
A.1 DATA AND PROCEDURE
Models. GEMMA-3-IB- IB-IT, GEMMA-3-4B-IT, LLAMA-3.2-3B-INSS -NNTRUCTT, plus the main-paper models.
Layer selection. For each transformer block, we analyze self-attention (Q/K/V/O projections) and MLP (up/gate/down) weight matrices independently.
Preprocessing. Unless noted, statistics are computed on raw matrix entries. For PCA, matrices are mean-centered and the spectrum is computed over flattened rows; report cumulative variance explained. We use excess kurtosis (Fisher convention) so a Gaussian has Kox=0:
[W - -4) 3.
Kix
σ²
Means and variances are standard sample estimates; skewness uses the unbiased estimator.
Aggregation. We report per-layer curves and layer-type aggregates (self-attenioon MLP). Family summarics first average across layers within a model, then across models the same family.
A.2 COMPLETE PLOTS
Figure 5 shows layerwise distribution plots (self (self-attention and MLP). Figure 6 reports cumulative PCA variance for all MLP layers (attention spectra are in the main text).
A.3 FINDINGS AND IMPLICATIONS
Self-attention layers exhibit heavier tails in Gemma. In self-attention projections (Fig. 7a, Table 9), GEMMA-3-1B. I-T and GEMMA-3-4B-IT show pronounced positive excess kurtosis with depth- localized peaks, indicating heavy tails and more extreme outliers. LLAMA models track closely across scales, suggesting family-level statistical stability.
MLP layers more stable and closer to Gaussian. Across models (Fig. 7b, Table 9), MLP projections are near- near-Gaussian with excess kurtosis typically in [0, 2] and fewer dependett spikes. Their PCA spectra (Fig. 6) decay faster than attention, indicating lower intrinsic dimensionality.
17

---

Published as a conference paper at ICLR 2026
(a)
(b) gemma- 3- b--it-self_attnllayaee
(c) gemma- gemma-3-4b-iteeeeeeeee 3--b-it-sooffaatm-laaaea
(d) Hlama-2-3b-itmmpplaaeett
(c) gemma-3 1arimna---l--ae 1b--ittmpp-lyyerr---
gemma-3 Hormma---4b-it-ee 3-b---t-mmmmpplayerrd
Distribution plot of the self-attention and the mlp modules in llama3- 1-8------b-aa 1-8b-instruct, gemma- 3-1b-it, and gemma-3-4b-il for attention layer
Figure 5:
llama3-2-3b-it
gemma-3- Ib-it
gemma-3-
3--mma---44-it
layers
skew
kurt.
skew
kurt.
skew
kurt.
0.0017
0.0192
8.4032
0.0031
0.0500
15.0505
0.0015
0.0501
15.2010
0.0013
0.0191
7.3438
0.0030
0.0450
9.8347
0.0015
0.0267
7.3832
selfattn
0.0012
0.0178
6,2172
0.0030
0.0389
9.0363
0.0014
0.0255
7.2277
0,0011
0.0104
5.4477
0.0030
0.0288
8.6496
0.0014
0,0172
6.0731
avg (self attn)
0.0005
-0.0002
1.4342
0.0012
+0.0009
3.2858
0,0005
0.0009
2.6900
min (self_attm)
0.0001
-0.0131
0.3027
0.0002
-0.0418
0.1589
0.0001
+0.0305
0.2412
0.0005
0.0093
5.4740
0.0010
0.0167
8.7665
0.0004
0.0266
6.3297
0.0005
0.0080
4,1465
0.0010
0.0090
3.1514
0.0004
0.0155
5,9670
mlp
0.0004
0.0075
2.6694
0.0010
0.0077
3.0577
0.0004
0.0130
2.7454
0.0004
0.0074
2.4364
0.0010
0.0071
3,0113
0.0004
0.0094
2.4371
avg (mlp)
0.0003
0.0003
0.8435
0.0006
1.1739
0.0003
0.0006
1.0807
min (mlp)
0.0003
-0.0139
0.0892
-0.0001
-0.0184
0.1552
0
-0.0076
0.1559
Table 9: Statistical moments of the self-attention and MLP layers across three models.
Design implications for encoding and merging.
Allocate capacity to attention. Heavy tails in attention (notably in Gemma) warrant encoders with higher capacity or robust priors; Gaussian assumptions under-represent extremes.
MLP is the easy regime. More Gaussian, stable MLP statistics admit accurate compression with standard VAE settings and fewer latent pathologies.
Family consistency aids calibration. The alignment LLAMA statistics across scales simplifies cross-scale latent calibration and reduces merging friction.
Summary. Attention weights (especially in Gemma) are the dominant source of heavy-tail behavior; MLP weights are comparatively benign. Pail-aware encoding and depthwise calibration are most critical for attention, while default settings suffice for MI.P
18

---

Published as a conference paper at ICLR 2026
(a) Llama3-2-2 3-3b-it-eeff.att--- layer0
(b) Gemma-3 3-iina----b-it-eeeee bo-iit-ellatttnllyeee
(c) Gemma-3-44 1-amma---4b-ie ---4-it-sel..attnllaae
(d) Lamma3-----ii--mlppayttttt
(e) Gemma-3-mm ---------------m---aarr----
(I) Gemma-3-44-ite 3-4b-it-mlp-layerO.
Figure 6: Plots of the PCA explained-variance ratio for individual weight matrices in the first self-attention block (top row of each subpanel set) of three LLM Gemma-3-Ib-it and Gemma-3-4b-it. For each model, show the four projection matrices of self- attention (q. k V. o). The sharp drop after the leading principal components highlights a pronounced low-rank structure that consistent across architectures and model sizes.
(a) Mean kurtosis per layer (Self-attention layers).
(b) Mean kurtosis per layer (MLP layers).
Figure Comparison of mean kurtosis across layers self-attention and MLP components for Gemma and LLaMA models.
B VARIATIONAL ENCODER ARCHITECTURE AND TRAINING DETAILS
Table 10 lists the configuration of our transformer VAE. The encoder applies twofold down- projection of the token (compression ratio r=2), which the smallest bottleneck we found that preserves reconstruction while generalizing to unseen checkpoints. The decoder mirrors the encoder to restore the original dimension. For heterogeneous merges, we set the latent size dy by the source to target layer mapping ratio and keep all other hyperparameters fixed, ensuring comparable latent scales across architectures.
Objective. We optimize the ELBO (reconstruction KL to (N( I)) with constant learning rate 1x10-6 for 10,000 epochs.
Training setup. Single NVIDIA A6000, bíloat16 for forward and backward, the remaining settings (optimizer, batch size, weight decay, clip-norm) are Table 10. Seeds are fixed for data order and initialization.
19

---

Published as a conference paper at ICLR 2026
Field
Value
Notes
length
10,240
Sequence length processed
n_layers
6
Transformer depth
chunk_size
640
Per-token chunk width
embed.dim
768
Matches global embod_dd dim
latent_dim
640
Size of latent vector per chunk
n_heads
4
Attention heads
rope rope.base base
20,000
RoPE frequency
conv
false
No convolutional patching
llatten
true
Flatten to original shape.
Table 10: Transformer block settings shared by the encoder and decoder.
compression)
Run 12x compression)
compression)
PVAE (2x compression)
PVAE compression)
500
Train Loss
400
300
100
25
50
75
100
125
15.0
175
Training Step
Figure 8: Mean reconstruction error curves (smoothed with a 5-step moving average) for pVAE (VAE initialized from existing pretrained VAE with reduction factor 1) and VAE (random initialized VAB). r denotes the compression ratio.
Ablation summary. Bottlenecks weaker than r=2 overfit and degrade duitt--disttibution reconstruc- tion, stronger bottlenecks raise reconstruction еттоr and harm downstream merging, Choosing d, via the mapping ratio stabilizes cross-arch alignment without retuning.
B. ANALYSIS WEIGHTS ENCODING EVOLUTION
Pretrained weight matrices exhibit substantial variation tail behavior (high near-zero excess kurtosis; ef. Section 3. We study how this variability interacts with the compression ratio in the VAE and how it impacts generalization to unseen checkpoints.
Convergence vs. compression. Figure 8 shows a clear monotone effect: optimization slows as the bottleneck tightens. With r=2 (twofold down-projection), storage is halved and the reconstruction loss increases by only x 6%; at r-4, the penalty rises to RS 10%. These curves set practical operating point: r=2 balances footprint and fidelity with minimal training overhead.
Sensitivity to distribution shape. Layers with heavier tails (high excess kurtosis) to increasing r. exhibiting larger loss gaps at fixed training budget, near-Gaue near-Gaussian layers degrade more gently. In models, attention projections in GEMMA fall into the former regime, whereas MLP projections across families align with the latter (App. A).
sensitive
Training schedule. Two-Stage Training. To improve stability under tight bottlenecks, employ two-stage curriculum: (1) train a deterministic autoencoder to convergence with the KL term disabled, then (2) enable KI. regularization and fine-tune as VAE. This approach prevents early posterior
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 Details of Heterogeneous LLM Parameter Merging in Latent Space
Require: Source weights W srs target weights Wigt, VAE configs Cere: Cigt
Ensure: Merged weights W Wmerged
1: Initialize VAEs: (V Vigi) < innil(Csee Cigt)
2: Load pretrained parameters into Vire. Vigt
3: Split weights into per-layer groups:
Lasc f- (all layers from W
Ligi f- (all layers from )
4: Determine the number of pairs to merge: N f- min( Larcl Lggs
5: Define aligned layer pairs: (149) ( for ,,, N
lgi
6: for each aligned pair (L ( do
7: Flatten & chunk weights into {ussc (2)
Wigt
8: Encode:
Vaco (urce
(usre) ), Argt + (uerr)
9: Align latents via OT: Pallinn + OTTT2 Figt
(1)
10: Merge latents: "merged Figt +-B. (=
(1)
(8)
(1)
Pngt
11: Decode: w merrged Vigi
( (#merged)
wy 12: Store "mrrgod in W merged
end for
14: Initialize and evaluate final network with complete W Weerged
return W merged
collapse and yields stable convergence even at high compression ratios, without increasing total training time.
Computational Cost All experiments were conducted on a single NVIDIA RTX 6000 Ada GPU. Training 200M parameter VAE at compression ratio 1.6 requires approximately 1-2 hours for IB-scale models and 3-4 hours for 7B-scale models (~5000 chunks). Higher compression ratios increase training time proportionally. LoRA weights converge notably faster than full model weights due to their lower-rank structure. At inference, the full encode-decode pipeline completes in 5-10 seconds, enabling rapid iteration during merging experiments.
Reducing the Training Time: Interestingly, in the case of heterogeneous models expansion < 1 can offer an alternative trade-off. Lifting weights into overcomplete latent space can unfold curved manifolds, making linear interpolation better approximate geodesics on the original weight manifold. expansion forces the encoder to learn non-trivial transformation, avoiding identity collapse. Practically, expansion significantly accelerates training the relaxed bottleneck cases optimization and improves gradient flow the cost of increased memory.
C HETEROGENEOUS MODEL MERGING IN LATENT SPACE
Merging models with matched layer shapes is straightforward in weight space. Heterogeneous pairs are harder: to avoid truncating the higher-capacity model, we train sepurate VAEs per architecture and merge in shared, aligned latent space. Concretely, first calibrate latents per layer, fit an alignment map between the source and target latent spaces, and then interpolate on the target side before decoding. This preserves each model's capacity while enabling stable cross-architectune merges without ad hoc dimensionality cuts. The full procedure is given in Algorithm
C. INTUITION BEHIND LATENT SPACE ALIGNMENT
The rationale for restricting explicit latent alignment specifically to heterogeneous merging scenarios relies on the geometric properties of the neural loss landscape and the topological structure of the parameter manifolds.
21

---

Published as a conference paper at ICLR 2026
Homogeneous Merging and Linear Mode Connectivity. In the case of homogeneous mod- els specifically those finetuned from a shared pretrained initialization (c.g., distinct mee-tunes of LLama-2-13B) the parameters remain within shared basin of attraction. Consequently, their weight vectors 0, 0₂ reside on connected region the high-dimensional parameter manifold M. When encoded into the latent space 2, the resulting distributions p(=(01) and p(z(02) naturally share a common support and overlapping density. Therefore, interpolation in Z corresponds to traversing flat, low-loss path the underlying manifold, rendering explicit distributional alignment redundant.
The Heterogeneous Disconnect. Conversely, heterogeneous models (e.g.. Gemma-3-IB vs. LLaMA- 3.2-IB) possess fundamentally distinct architectures. Even when projected into latent space of identical dimensionality d, their representations occupy disjoint manifolds Mare and Migs with divergent geometric structures and density profiles show Figure
This disjoint nature necessitates non-linear mapping to bridge Mare and Megt- We employ Optimal Transport (specifically the Monge map) to push the source distribution Mare onto the target distribution Megt by minimizing the Wasserstein-2 distance. This process essentially "registers" the two manifolds, aligning their statistical moments and geometric structure. By enforcing this alignment, we ensure that the interpolated latent codes za remain within the valid density region of the target decoder, thereby preserving functional competence across architectural boundaries.
WeorEM SIEVEE
HaW 23
Dimsekire
(a) Architectee Fine-tuning Variants: LLaMA- (b) Cross- Comparison: Gemma-3-1B L.AMAA3.2-1BB
2-13B Code-Alpaca MizandIMM11B--1..
Figure 9: VAE latent space reveals architectural signatures in weight distributions. t-SNE projections of encoded k k_proj weights show (a) fine-tuned variants of the same base model (LLLaMA- 2-13B) remain indistinguishable while (b) distinct clusters for different architectures (Gemma vs LLaMA), suggesting the learned latent space captures intrinsic architectural properties.
22


Reviews:
[{"review_id": "R6GCDlznrs", "reviewer": "Reviewer_XrKv", "summary": "This paper presents a novel method for model merging where the size of the models do not need to match. In order to do so, the authors use variational auto-encoders, optimal transport, linear interpolation,  and projections down to a merged weight space. The results beat strong baselines (including very recently published methods) on a wide variety of tasks. I’ll keep this reviews short because, overall, I would be very happy to see this published in the conference.", "questions": "The method seems like it could be a bit computationally expensive - and this is mentioned in the limitations. How expensive is it exactly? I don't have a particular way I'd like to see this question answered, but whatever makes sense. For instance, maybe time on a GPU? What sort of GPU (which is definitely dependent on the models)? Maybe percent of compute needed compared to doing something from scratch? Or comparison to another method (i.e. AIM)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Though the paper is dense and covers a lot of complex topics, it is well-written and easy to follow. In addition, things are presented clearly, such as the caption of Figure 1.\n\nThere are a lot of experimental results with very strong baselines that they beat.\n\nIt is an interesting idea intellectually and I would have liked to have seen the paper published even if the results had not beaten the baselines – but they did.", "weaknesses": "Perhaps I missed it, but I think a bit more background on OT could be useful for the presentation to the reader. The paragraph at the end of section 3 could be expanded a bit more. I ended up needing to look at one of the cited papers. However, most of the other parts of the paper explained complex topics very well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ITBKguzztn", "reviewer": "Reviewer_593M", "summary": "LS-Merge is a method that uses a Transformer-based VAE to compress chunks of weight matrices from various LLMs, enabling heterogenous merges where models have different architectures. THhe VAE encodes weight chunks into lower dimensional representations, aligns them in latent space, interpolates them, and then decodes them back into model space. This method can improve performance of single models, improve merges of multiple LoRA adapters, and perform competitively with direct weight space merging methods.", "questions": "1. In section 3.1, is the v_proj included in this analysis? It is missing from the description of the moment analysis of the key LLM weights. \n2. What exactly is a layer matrix in section 3.1 line 150 and section 3.2 line 204? Is it a single weight like q_proj or up_proj or is it the entire Transformer layer?\n3. Is the embedding from line 206 part of the transformer encoder? Or is it separate?\n4. What is the exact operation for the rescaling procedure described for heterogenous mapping? It is not clear how the value r is used in this mapping. \n5. What is the model used as base in Table 5? And which model is weighted 0.1 in the mixture?\n\nTypos\n1. Line 192, porjection\n2. Line 139 up_porj, down_porj\n3. Line 233 artihmetic \n4. What is the \"fixed of 2\" mean on line 321?\n5. Line 796 is missing a reference to a figure in Latex. \n6. Line 403 familly\n7. Line 404 \"is perform\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. By applying the LS-Merge VAE on pre-trained Gemma models, the output model can improve on benchmarks like GSM8K and MMLU without additional gradient steps on the model weights, just on encoding and decoding the weights with the pretrained VAE. This represents a unique and nice result of improving singular model.  \n2. The proposed method of using weights as data to learn via a VAE is novel, interesting, and a potential direction to expand upon in future work, especially given interesting results like point 1. Also, it appears that only a few models worth (2 Gemma models) is sufficient to train on to achieve useful results. \n3. This method can achieve heterogenous merging, mostly on intra-family based merges.", "weaknesses": "1. While the results are impressive, the description of the method is not entirely clear at times and detracts from the contribution of the paper. For example, it is not clear how heterogenous rescaling occurs according to the description, and it is not when the OT based alignment occurs in the latent space training. The contribution of the paper seems solid, but its presentation seems rushed and the paper does not seem reproducible or easily understandable in its current state. Another example is the statement that the evaluation of cross family evaluation is performed using lm-eval due to issues with the llama model when using previous evaluation code. What does this mean here? \n2. Despite good results, this work lacks some analysis of what is learned by the VAE model, as well as some ablations of key choices. It does not seem clear why this method works, or what about it makes it work. What is the latent size used in this work? And what is the chunked size? And how were these values set? \n\nThis paper is quite interesting but unfortunately its presentation and polish is very lacking, which brings into question the correct execution of this work. I think this paper could be impactful and a nice contribution, but in its current form I cannot recommend accepting it as it seems only partially finished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjEA6XtKad", "reviewer": "Reviewer_mkDJ", "summary": "This paper propose a novel method to replace fragile weight-space merging with LSMerge: a latent space merging technique. It is an encode–align–decode pipeline that operates on a latent space of model weights learned by a transformer VAE. Specifically: the method \n(1) encodes parameters into latents via layer-aware chunking, \n(2) aligns heterogeneous models’ latent distributions (depth/width mismatches) using Optimal Transport (OT), and \n(3) decodes an interpolated latent back to weights. \nEvaluations demonstrates performance gain in (i) self-merging of a single model’s latents, (ii) LoRA expert fusion vs. model-soup/SLERP baselines, and (iii) heterogeneous merges (within-family size changes and cross-family).", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Principled formulation: The core idea of encoding model weight into a shared latent manifold and aligning distributions via OT is right on target to resolve the pain point of current merging methods in shape matching requirements. This paper gives a more grounded understanding of how to merge models by aligning the model's representation geometry.\n\n2. Comprehensive experiment setting: The same recipe can be used for many use cases, including self-merging, LoRA-expert fusion, and cross-family merging. This shows that this method is a more fundamental framework rather than a \"trick\".", "weaknesses": "1. Lack of analysis on reconstruction error: The main component of the merging system is a VAE which introduces lossy compression during reconstruction, yet there isn't an analysis of how that bottleneck correlates with downstream task accuracy. This matters as it could confound where the gains are from. I'm curious if the author can ablate on compression level vs. KL-weights. \n\n2. Lack of understanding of successful merging condition: The experiments are diverse in terms of merging various types of models (self, LoRA etc.) but it is unclear what is the boundary condition of successful latent space merging? I'm curious to see if the authors have found any failure case and if so, have done any analysis of what's the key difference between merges that fails/succeeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)