Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
Authors: Ryo Bertolissi
Jonas Hübotter
Ido Hakimi
Andreas Krause
Venue: COLM 2025 (2025)
Keywords: test-time training
model merging
mixture of experts
language modeling
local learning
transductive learning
TL;DR: We propose Test-Time Model Merging (TTMM) which approaches the performance of Test-Time Training (TTT) without almost any test-time overhead.

Abstract: Mixture of expert (MoE) models are a promising approach to increasing model capacity without increasing inference cost, and are core components of many state-of-the-art language models.
However, current MoE models typically use only few experts due to prohibitive training and inference cost.
We propose _**T**est-**T**ime **M**odel **M**erging_ (TTMM) which scales the MoE paradigm to orders of magnitude more experts and uses model merging to avoid almost any test-time overhead.
We show that TTMM is an approximation of test-time training (TTT), which fine-tunes an expert model for each prediction task, i.e., prompt.
TTT has recently been shown to significantly improve language models, but is computationally expensive.
We find that performance of TTMM improves with more experts and approaches the performance of TTT.
Moreover, we find that with a 1B parameter base model, _TTMM is more than $100\times$ faster than TTT_ at test-time by amortizing the cost of TTT at train-time.
Thus, TTMM offers a promising cost-effective approach to scale test-time training.

Published as a conference paper at COLM 2025
Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
Ryo Bertolissi; Jonas Hübotter, Ido Hakimi, Andreas Krause ETH Zürich, Switzerland
Abstract
Mixture of expert (MoE) models are a promising approach to increasing model capacity without increasing inference cost, and are core components of many state-of-the-art language models. However, current MoE models typically use only few experts due to prohibitive training and inference cost. We propose Test-Time Model Merging (TTMM) which scales the MoE paradigm to an order of magnitude more experts and uses model merging to avoid almost any test-time overhead. We show that TTMM is an approximation of test-time training (TTT), which fine-tunes an expert model for each prediction task, i.e., prompt. TTT has recently been shown to significantly improve language models, but is computationally expensive. We find that performance of TTMM improves with more experts and approaches the performance of TTT. Moreover, we find that with a 1B parameter base model, TTMM is more than 100x faster than TTT at best-time by amortizing the cost of TTT at train-time. Thus, TTMM offers a promising cost-effective approach to scale test-time training.
Test-time scaling
Train-time scaling
1.00
basc model
base model tine-tuning
Number of active expers
Normalized perplexity
fine-tuning
(lower is better)
0.75
TTMM
ensembling
TTT
0.50
0.25
commony Molic
TTMM
10
0.00
10a
10a
10a
10
10a
10a
Test-time compute (ms)
Total number of experts
Figure 1: Accuracy gains of TTMM when scaling test-time compute (left) and pre-training
compute (right). The normalized perplexity between base model and TTT model is averaged across evaluation datasets and models. Left: We measure language modeling ability against the time to generate 100 tokens. TTMM approaches the performance of TTT without almost any test-time overhead when increasing the number of active experts. Ensembling rather than merging experts improves performance, but at much higher test-time cost. Standard fine-tuning on the training data improves performance without any test-time overhead, but does not yield close to the performance of TTT. Right: We measure the performance of TTMM against the total number of experts. Whereas common MoE models use few experts, TTMM scales to an order of magnitude more experts, improving performance with more experts. Scaling the number of experts in TTMM does not affect training FLOPs or required GPU memory.' Training the individual experts is embarrassingly parallelizable
"Equal contribution. Correspondence to j terbift .ethz.c. SSallnng the number of experts increases CPU host memory, which is cheaper than GPU memory.

---

Published as a conference paper at COLM 2025
train local data
merge local models
prompt
a) pre-training
b) test-time training
c) test-time model-merging
inductive
transductive
Figure 2: Standard inductive language modeling pre-trains a single model on the dataset (shown in gray) which is then used for prediction. Test-Time Training (TTT) improves performance by fine-tuning the model at test-time on data related to the prompt, but is computationally expensive. Test-Time Model Merging (TTMM) fine-tunes the pre-trained model to many local neighborhoods at train-time, and merges the local models related to the prompt at test-time. Both TTT and TTMM target models specifically to the prompt (i.e., are transductive), yet, TTMM does not require expensive fine-tuning at test-time.
1 Introduction
The standard paradigm of machine learning separates training and testing. In training, we learn model by inductively extracting general rules from data, while testing, we apply this model to new unseen data. We study an alternative transductive paradigm called fest-time training (TTT, Sun al., 2020) where we use a specific expert model for each prediction task. Variations of this paradigm have been studied since the inception of the machine learning field (Cleveland, 1979; Cleveland & Devlin, 1988; Atkeson et al., 1997; Bottou & Vapnik, 1992). More recently, fine-tuning large pre-trained neural networks at test-time has regained interest in computer vision (e.g., Jain & Learned-Milles 2011; Shocher et al., 2018; Sun et al., 2020; Gandelsman
al., 2021; Ruiz et al., 2023) and language modeling (Krause et al., 2018; 2019; Hardt & Sun, 2024; Sun et al., 2024; Hübotter et al., 2025). Hardt & Sun (2024) and Hübotter al. (2025) fine-tune a pre-trained language model on data related to the prompt, and show that this can significantly improve language modeling performance, Beyond pure language modeling, TTT has been instrumental in sate-of-thheart approaches to abstract reasoning (Akyürek et al., 2025), video generation (Dalal et al., 2025), and also been shown to improve reasoning language models (Zuo et al., 2025; Simonds Yoshiyama, 2025).
While TTT can significantly improve model performance, it comes with a high computational cost at test-time: TTT requires fine-tuning the model for every task, i.e., each prompt. We propose Time Model Merging (TTMM) which amortizes the cost of TTT at train time:
At train-time, TTMM clusters the training data into many local neighborhoods and trains separate small expert LoRA adapters (Hu et al., 2022) for each cluster.
At test-time, TTMM dynamically selects subset of LoRA adapters and merges their parameters to form single task-specific model.
In our experiments, TTMM approaches the language modeling performance of TTT without incurring almost any sigmificant compute or memory cost (cf. Figure 1). Figure 2 illustrates the difference between TTT and TTMM.
Another extensive body of work studies merging multiple models combine distinct capabilities and knowledge (Yang et al., 2024b). We discuss the connection of TTMM to this literature on model merging and mixture of experts more extensively in Section 2. Model merging is typically used in a multitask setting with few "meta-tasks" (e.g., coding, math, law, etc.) and corresponding expert models. This multitask setting differs from the TTT setting where each individual prompt is considered its models, even within a single meta-task (say for all kinds of different coding tasks). Then, for each prompt at test-time, TTMM merges the most related local models. As such, TTMM can
task. Hence, TTMM trains many local
2

---

Published as a conference paper at COLM 2025
be seen as bridging multitask model merging and TTT by dynamically constructing an expert model at test-time from an order of magnitude more meta tata- than in multitask learning.
We contribute the following key findings, summarized in Figure 1:
1. TTMM outperforms multitask model merging: We find that training and merging many expert models, even within a single domain such "coding in Python", outperforms training and merging few models specialized on broader tasks.
2. TTMM approaches performance of TTT without almost any test-time overhead: We discuss how to merge the parameters of the expert models at almost no additional compute or memory cost. We then evaluate TTMM on the Wikipedia and GitHub Python corpora and find that it achieves performance close to TTT, while matching the performance of ensembling without incurring its test-time overhead.
Related Work
We propose TTMM as an efficient amortization TTT, which realizes comparable accuracy. We next discuss how TTMM can be seen as an extension to the literature on model selection and model merging. In Appendix A, we discuss the link to retrieval-augmented generation.
Multitask Model Merging. Combining the predictions of multiple models has been a long-standing practice in machine learning, e.g,, in the form of majority voting (Wang et al., 2023) or by ensembling of predictions (Dietterich, 2000; Wortsman et al., 2022), which has been used to avoid catastrophic forgetting (e.g,, Bagatella et al., 2025). However, merging predictions of many neural networks is computationally inefficient since it requires separate forward pass for each model. Instead, TTMM merges in parameter-space, and only requires a single forward pass of the merged model, resulting in negligible increase of compute and memory cost compared to inferencing single model. The idea of merging multiple models by averaging their parameters (not their predictions!) is grounded in the surprising phenomenon of mode-comeectizity (Garipov et al., 2018; Wilson, 2025), which observes that modes in ensembles of models are connected by paths of small loss. This has motivated a variety of model merging techniques, including stochastic weight averaging (Izmailov et al., 2018) and model soups (Wortsman et al., 2022). Recently, model few merging has been studied predominantly in the context of merging, few expert models with the aim of retaining performance on their respective meta-tasks (Ilharco et al., 2023; Matena & Raffel, 2022; Jin et al., 2023; Ainsworth et al., 2023). Merging model parameters may lead to interference, which has been a key focus of recent work (Yadav et al., 2023; Yang et al., 2024c; Tang et al., 2024b; Daheim et al., 2024; Yu et al., 2024; Jung et al., 2024).
Dynamic Merging of Multitask Models. Recent work in multitask model merging studies selecting merging coefficients dynamically depending on the prompt. Oh et al. (2025) select coefficients based on the entropy of the individual model's predictions, which is computationally expensive since it requires forward passes of all networks. Instead, TTMM computes merging coefficients based on a measure of similarity between prompt and experts which is tuned on holdout data at train-time (Lu et al., 2024; Tang et al., 2024a; Cheng et al., 2024; Lai et al., 2025), and which incurs only negligible test-time overhead. In the tangentially related literature on black-box model fusion, dynamic model selection has been widely studied (e.g., Liu et al., 2021; Jiang et al., 2023; Mavromatis et al., 2024). However, in the setting of TTMM, we control data and architecture used for training the local models, which enables us to compute merging coefficients more efficiently than in black-box model fusion.
Clustering and Dynamic Model Selection. While the above literature focuses on multitask learning with typically only a handful of expert models, TTMM clusters the training data into many local neighborhoods and trains separate local model for each cluster. Like TTMM, Branch-Tran Frranh-Trrin--erre Train-Mergee (BTM, Li et al., 2022; Gururangan et al., 2023) clusters the training data into local neighborhoods, trains a separate local model for each cluster, and dynamically selects a subset of these models at test-time. However, unlike BTM, TTMM uses order of magnitude more local models and merges their parameters instead of their predictions, leading to substantially faster inference. In concurrent work, Qiu et al. (2025) study TTMM in continual learning setting.
3

---

Published as a conference paper at COLM 2025
train-time
test-time
prompt
2. merge
cluster experts
select experts
3. inference on merged model
At train-time (Algorithm 1), TTMM clusters the training data (shown in gray) into many local neighborhoods and trains a separate expert model for each cluster. At test-time (Algorithm 2), TTMM dynamically selects subset of expert models related to the prompt and merges their parameters to form a single task-specific model. The test-time stage is compute and memory efficient, using the previously trained experts.
Figure 3: Illustration of TTMM:
Mixture of Experts. The mixture of experts (MoE, Shazeer et al., 2017) paradigm increases model capacity without increasing inference cost by introducing conditional routing of inputs to subsets of learnable parameters (i.e., "experts"). Training MoE models is challenging due to the need for joint training of the experts and the router on substantial multitask data. Moreover, MoE models evaluate all active experts individually, which can still require substantial memory. This in turn implies that the number of experts is typically limited to small number. Recent studies have focused on challenges such as reducing synchronization (Sukhbaatar et al., 2024; Zhang et al., 2024b; 2025), load balancing of experts (Clark et al., 2022; Zhou et al., 2022), and improving expert specialization (Dai et al., 2024). In contrast, TTMM approximates the MoE paradigm by embarrassingly parallelizing the training of the local models and merging model parameters at test-time, which permits scaling to many orders of magnitude more experts and which enables approximation of the "maximally local" TTT.
3 TTMM: Test-Time Model Merging
Our proposed method, TTMM, has two main stages: train-time and test-time. At train-time. TTMM clusters the training data into many local neighborhoods and trains separate expert model for each cluster. Then at test-time, TTMM dynamically composes these expert models to form single task-specific model. We discuss both stages in more detail below.
We consider a domain X' of token sequences and a training set D X. We further assume that have access to a pre-trained autoregressive language model f(x; 0) that maps token sequences x E X' to probability distributions over the next token. Finally, we assume access to a sequence embedding model (xx) returning normalized embeddings. In our experiments, we use a mean-pooled sequence embedding model, but one could also use the last-layer token embeddings from the pre-trained language model (see Lu et al., 2024).
Train-time: Clustering. To cluster the training dataset, we use bisect- ing k-means, hierarchical clustering algorithm which recursively applies k-means with k 2 the cluster with the largest diameter (Jain, 2010). We use bisecting k-means since is more efficient than k-means and tends to lead to more balanced clusters, however, any be used. We then train a small LoRA adapter on each cluster for one epoch. Finally, average the embeddings of the sequences in each cluster
Algorithm 1 TTMM at train-time
Require: Language model f(x; 0) with pre-trained weights 0; sequence embedding model (xx); training set D; number of clusters K.
1: Cluster training set into clusters D Dx.
Train a LoR adapter 0 on each cluster Dj 3: Compute cluster-specific embeddings:
Φ + normalize ExED, 4(x)
clustering algorithm
4

---

Published as a conference paper at COLM 2025
to obtain sluster-specific embeddings (i.e,, "centroids") that represent the capabilities of the corresponding expert model. We renormalize the centroids to avoid biasing the embedding norm of larger clusters toward zero. Without the additional normalization, centroids of larger clusters have smaller norm despite (xx being normalized.
Test-time: Model Merging. At test- time, given a promptx* É X and its em- bedding x- = ((xx), we compute merg- ing coefficients w for each expert model 0g using a sparse cross-attention mecha- nism with sparsity parameter E [0, z):
Algorithm 2 TTMM at test-time
Require: Prompt x* with embedding s". Language model f(x; 8) with pre-trained weights @; expert models {02z &x )))= temperature & and spar- sity (tuned on holdout data).
sparse-softmax, (z)
1: Compute cluster-specific merging coefficients: wx + sparse-softmax, ( (1 4*)
ReLU(ssftmax(z)) 7)
Σe Res.Usssftaax -T)
2: Merge into single expert model:
max f z]. This sparse softmax returns sparse distribu- tion over the experts, where experts with weight below are pruned, and the re- maining weights are renormalized. Pruning experts with low weights improves test-time efficiency since less expert parameters need to be merged (see Figure 4 (right)), with minimal degradation accuracy. The sparsity being smaller than 1/K ensures that always at least one expert is selected. The model selection can easily be parallelized for all clusters and multiple queries by writing the computation as a cross-attention matrix product:
where ReLU(z)
0 - Σx= wy00
Use f(xx;0") to generate the next token.
sparse-softmax, (1
θ
with keys @ = - E kkdd queries @ - i
..... D E Ruxd and values θ = 0 0x] E RRkx computing the collection of merged expert models O Rx*P,
3.1 Latency
In contrast to MoE models, the number of experts in TTMM is typically too large for all experts to fit in GPU memory at once. For example, with Llama-3.2-1B as base model, each LoRA adapter has size approximately 172 MB, and with 1000 clusters, the expert models would require prohibitive 172 GB of GPU memory. Instead, we dynamically load active experts into GPU memory and merge their parameters before inference.
The test-time overhead of TIMM consists therefore of the time required to compute the merg- ing coefficients (select), the time to load active experts from CPU to GPU memory (load), and the time to merge the expert parameters (merge). In Figure 4 (middle), we benchmark this overhead across varying numbers of active experts. On our infrastructure and with 10 active experts, selection takes roughly 5 milliseconds, loading takes around 30 milliseconds, and merging takes approximately 80 milliseconds. We use PyTorch's einsum (Paszke, 2019) for computing the merged LoRA adapter AW- Σx w; B,A, (cf. Figure 7 in the appendix), with A₁ and B; the low-rank factors the LoRA adapter 0g. This is around 20% faster than the naive implementation with for-loop over the experts, without any additional memory over- head. We anticipate that test-time overhead may be reduced further by interleaving loading and merging by overlapping computation and communication (see, e.g., Tang et al., 2025).
TTMM costs 220 tokens. In Figure 4 (middle), we evaluate the practical test-time overhead of TTMM for language generation. TTMM incurs only a small fixed overhead which, when merging 10 experts, equal to generating around 20 tokens. When generating more than 100 tokens, the overhead of TTMM compared to inference with the base model is negligible. In contrast, ensembling has multiplicative overhead that grows with the number of active experts. Comparing to TTT, a single test-time gradient step with Llama-3.2-1B takes approximately 150 milliseconds (Hübotter et al., 2025). With 100 test-time gradient steps, the total test-time overhead (without any data selection and loading) is already around 15 seconds. In contrast, TTMM with 10 active experts has a constant overhead of around 115 milliseconds, which is more than 125x speedup compared to TTT.
5

---

Published as a conference paper at COLM 2025
Inference cost
Test-time overhead
of active experts
10.
base model
10
seloct
# of active experts
-
0.1
Time (ms)
104
ensemblinng TTMM
10a
load merge
10
0.01
0.001
Time
10.
10
10a
10a
100
100
10a
102
10a
10
10
102
0.1
0.2
Number generated tokens
Number active experts
Temperature 3
Figure 4: TTMM has negligible compute overhead at test-time. Left: Inference cost of ensembling and TTMM with 10 (dark) and 3 (light) active experts, respectively. TTMM with 10 experts costs around 20 generated tokens, while with experts, TTMM costs only about 10 generated tokens. In contrast, ensembling has multiplicative overhead that grows with the number of active experts. Middle: Test-time overhead of TTMM compared to inference with the base model. When using up to 10 experts, the test-time overhead is small. Right: Number of active experts with a given temperature B and sparsity T on the Wikipedia dataset with base model and consider LoRA adapters with rank 64. Each LoRA adapter has size = 172 MB. For loading, we assume a CPU to GPU bandwidth of 50 GB/s. s. For computing embeddings, we use the all pall-mpeet-bseetee model. We measure selection and merging on an RTX 4090 GPU with 10 seeds.
-= 1000 experts. Setting: We use Llama-3.2-11
Balancing Latency and Performance. The exact Pareto frontier of latency and performance varies depending on generation length. Figure 1 plots the Pareto frontier for 100 generated tokens and Figure 4 (left) shows how the latency changes for other generation lengths. We do not show TTT in Figure& 4 (left) since the constant overhead of TTT before generating any token larger than the time generate 1k tokens with TTMM (and 10 active experts).
3.2 Test-Time Model Merging approximates Test-Time Training
TTMM can be seen as an approximation of TTT that amortizes TTT's compute cost at train-time. We formalize this connection by showing that in the limit of exponentially many experts, TTMM closely approximates TTT (Hardt & Sun, 2024), which fine-tunes the pre-trained model on the N nearest neighbors to the prompt in the training data. To see the intuition behind this approximation, first consider the set of all subsets of D of size N. Suppose we pre-train an expert model on each of these subsets. Then, at test-time, we select the expert model whose training data embeddings have the shortest distance to the prompt embedding. Note that this is equivalent to TTT, without the need for fine-tuning at test-time. We have simply pre-computed all possible expert models at train-time, and at test-time fetch the right one. TTMM approximates this interpretation of TTT in three steps.
Approximation 1: Fewer amortized experts. The number of pre-trained expert models in the above scenario not practically feasible, it grows exponentially with the training data. Thus, TTMM considers a more practical scenario with a smaller number of clusters where we do not pre-compute all possible TTT models. Our following proposition indicates that even an expert model different to the TTT model, yet trained on a cluster D' sufficiently close to the prompt embedding, may still be good approximation of TTT:
Informal Proposition 3.1 (Formalized in Proposition C.1). Suppose the neural network f (x; 0) is L-Lipschitz in 9 and that the loss is G-smooth in x, with any L,, G > O. Further, suppose we adapt models via T > steps of gradient descent with step size 17 from shared initialization O. For any prompt x* E X and N > 1, let 02* be the TTT model trained on the set Dz+ of the N nearest neighbors to (xx*) in D. Let 6' be the expert model trained on any D' C D. Then, if D' contains al least one nearest neighbor from Dx²,
[x( f(x*; 8')l| (/TLG(dia diam
(1)
To understand the intuition behind this proposition, consider an expert model trained on cluster D' that contains parts of the local neighborhood of the prompt embedding (for this,

---

Published as a conference paper at COLM 2025
the entire clustering must cover the dataset). Then, the above proposition shows that this expert model is a good approximation of the TTT model if the diameter of D' is also small. Consequently, this motivates the approximation of TTT by TTMM with many expert models covering the entire dataset, each trained on a small cluster. Training expert models on this partitioning of the dataset does not cost more FLOPs or GPU memory than training single model on the entire dataset. This contrasts with TTT, which considers a separate expert model for each possible recombination of the training data into subsets of size N.
Approximation 2: Summarizing experts with their centroids. Instead of selecting the expert model by minimizing the sum of distances of individual data points to the prompt embedding (i.e., ExED, (xx(xx) (xx) [), TTMM summarizes each expert model by its
centroid 1/DD, DxED, (x). Selecting the expert with the closest centroid to the prompt embedding further speeds up inference, since we only need to compute K distances, rather than the distances to all training data embeddings. This simplification does not come for free: Consider the one-dimensional set of embeddings 1, and a prompt embedding of 0 with N = 2. Then, the optimal cluster is either {-1,0} or [0,1}, yet TTMM would pick {-1,1} since its centroid averages out differences in the embeddings. Since TTMM first clusters the data, there is reason to expect that each cluster has small diameter, in which case the centroid is a good approximation of the individual data points.
Approximation Merging multiple experts. As final step, to compensate for the ap- proximation error of the chosen expert model induced by steps (1) and (2), TTMM merges multiple expert models. Intuitively, and as visualized in Figure 3, this is motivated by multi- ple experts being able to cover distinct features of the prompt that may not all be captured by a single expert model. We find empirically that merging a small number of experts can improve model performance without incurring significant test-time overhead (cf. Figure 1). TTMM computes merging coefficients via pross-attention, which is equivalent to weighting the active experts with an RBF kernel.2
softmax (5 Φ-) = normalize exp
px1l3
10p
2B
Weighting data (not models!) by an RBF kernel has long been common approach in machine learning and used, e.g., in kernel density estimation (Rosenblatt, 1956; Parzen, 1962), kernel regression (Nadaraya, 1964, Watson, 1964), RBF networks (Lowe & Broomhead, 1988; Moody & Darken, 1989), and local learning (e.g., Bottou & Vapnik, 1992; Atkeson et al., 1997). TTMM additionally sparsifies the selected models to minimize the test-time compute overhead due to loading and merging.
Results
We evaluate language modeling with causal language models. We report the perplexity of the models on holdout data from the Wikipedia (English, Wikimedia Foundation, 2025) and GitHub (Python code) corpora, comprising 6.4M and 7.2M documents, respectively. As base models, we use Llama-3.2-1B (Grattafiori et al., 2024) and Qwen2.5-1.5B (Yang et al., 2024a). For clustering and model selection, model (Reimers & Gurevych, 2019; Song et al., 2020). Following Hardt & Sun (2024) and Hübotter et al. (2025), we evaluate TTT by training a LoRA adapter for a single gradient step each on the 100 nearest neighbors to the prompt embedding, most to least similar. We also compare against a single model, which fine-tuned for a single epoch on the respective corpus. This fine-tuned model can be considered an upper bound on the performance of approaches to multitask learning, since multitask learning typically trains one model per meta-task (i.e., per corpus) and merging models from multiple meta-tasks typically reduces performance on any individual meta-task.
embeddings from the all-mpnet-base-v
Our code is available at https: //gi thub. com// We share fine-tuned baselines and TTMM-MoES on Huggingface.
25ee Appendix B for a proof.
7

---

Published as a conference paper at COLM 2025
Wikipedia
GitHub (Python)
Model
Merging
Ensembling
Merging
Ensembling
Llama-3.2-1B
8.674
2.611
fine-tuning
7.849
2.581
+ TTMM (1 active expert)
7.669
2.552
+
TTMM (3 active experts)
7.571
7.579
2.519
2.512
TTMM (10 active experts)
7.510
7.509
2.492
2.464
+ TTT
7.559
2.441
Quven2.5-1.58
8.570
2.335
+ fine-tuning
7.702
2.317
+ TTMM (1 active expert)
7.166
2.330
+ TTMM (3 active experts)
7.101
7.096
2.304
2.294
+ TTMM (10 active experts)
7.080
7.061
2.287
2.257
7.231
2.177
Table 1: Perplexity (lower is better) across evaluation datasets and base models with 100 perts. Bold numbers denote the best test-timg fest-time efficient method, and underlined numbers de- note the best overall. TTMM outperforms standard multitask fine-tuning of a single model.
Wikipedia
GitHub (Python)
Normalized perplexity
Cross-attention
Uniform weighting
Cross-attention
Uniform weighting
0.6
0.6
0.4
0.04
0.10
0.1
0.2
0.01
0.03
10
10
0.01
0.03
0.05
10"
10'
Temperature /
Number of active experts
Temperature /
Number of active experts
active expert
0.001
0.01
Figure 5: On each dataset, evaluate the performance improvement of TTMM's cross-attention mechanism compared to uniform weighting of active experts. Uniform weighting (dotted) selects the experts closest to the prompt embedding, and weights them equally. We see across merging (red) and ensembling (light blue) that the cross-attention mechanism outperforms uniform weighting significantly. Notably, on Wikipedia, combining multiple experts helps only when using cross-attens noss-attention. The number of experts is K = 1000.
Insight 1: TTMM approaches the accuracy of TTT at almost no test-time overhead. We show in Figure and Table that TTMM with 10 active experts achieves close to the accuracy TTT, and much higher accuracy than single model that was fine-tuned on the respective corpus. Instead of merging experts in parameter space, we also evaluate merging experts in prediction space by ensembling the experts. We find that this yields marginally higher accuracy, yet has significantly larger test-time overhead. Overall, merging experts in parameter space achieves the best trade-off between accuracy and test-time efficiency. Interestingly, TTMM outperforms TTT on the Wikipedia dataset. This may be because few
TTMM can incorporate knowledge from larger set of experts, each trained thousands of data points. In contrast, TTT is only training on the 100 most related neighbors, which may miss some information if there more than 100 related data points. In our view, the empirical difference between TTMM and TTT is an interesting direction for further study,
Insight 2:
Cross-attention more effective than other merging coefficients. In Figure 5, evaluate perplexity with merging coefficients computed via cross-attention to uniform

---

Published as a conference paper at COLM 2025
Wikipedia
GitHub (Pythea)
x10
Determining K
Determining
1.0
pass @ N accuracy
k-means loss
Clister
4
0.8
100 clusters
0.6
250
500
1000
10a
10
Number of clusters (K)
Number of active experts (N)
Expert model
y dataset. The lowest perplexity is achieved on the diagonal, ie,, when the expert model is evaluated on holdout data from the cluster it was trained on. The block-diagonal structure is an artifact of bisecting k-means, which leads to similar neighboring clusters. Middle: In our evaluation datasets, the rate of reduction in k-means loss largest until around 100 clusters. Right: For 100 random samples from each cluster's 's holdout set, we measure whether the correct cluster is among the top-N closest clusters and report the average pass@N accuracy.
Figure 6: Left: Perplexity (brighter is better) of 1000 experts clusters of the Wikipedia
weighting of active experts. We find that cross-attention significantly outperforms uniform weighting. In particular, on the Wikipedia dataset, merging multiple experts does not help at all when weighting them uniformly. We also compare against merging coefficients, such as leveraging the logit-entropy of the experts (cf. Appendix D.3), but we do not find them to out- perform 1TMM's cross-attention while being significantly more computationally expensive. We find that sparsity T = 0.01 provides the best trade-off between accuracy and test-time efficiency, and we fix it in all other experiments. Furthermore, on our evaluation corpora, we need to select roughly 10 active experts to cover the local neighborhood of the prompt (cf. Fig- ure 6 (right)), which is also when the performance of TTMM begins to saturate.
Insight 3: Local experts outperform the base model in their local neighborhood. We find in Figure 6 (left) that expert models perform significantly better on holdout data in their cluster than the base model or other expert models. This reaffirms the earlier findings of Hardt & Sun (2024) that TTT on local neighborhoods improves accuracy. We also find a large qualitative difference between the text generated by different experts and the base model, and include examples in Appendix F.
Choosing the number of (active) experts. We find, summarized in Figure 1, that TTMM with 100 experts significantly outperforms TTMM with 10 experts or the multitask fine- tuned model (i.e., single expert). This indicates that scaling the number of experts beyond the number of meta-tasks can increase accuracy substantially. Nevertheless, we also find that scaling the number of experts to 1000 does not further improve accuracy on our evaluation corpora. In the following, we discuss considerations for the number of experts in TTMM:
Number of experts (K): The total number of experts has almost effect on latency. However, more experts require a large amount of static CPU memory. Further, increasing the number of experts also increases the separation of knowledge/ /kkils between experts. This then requires more active experts to retain the same performance, which also increases latency and may lead to model interference due to merging. To prevent knowledge fragmentation, K should be determined depending on the training set. In our experiments, choosing 100 experts is indicated by the "elbow method", common in clustering (cf. Figure 6 (middle)).
direct
Number of active experts (N): The number of active experts has the strongest effect on latency, since all active experts have to be loaded from CPU into GPU memory before inference. Our ablation in Figure 6 (right) indicates that with 10 active experts, the "correct" expert is likely to be active, while latency remains minimal.
Evaluation MMLU. We additionall evaluate TTMM on MMLU (Hendrycks et al., 2021) as a non-perplexity task. We use (non-instruct) Llama-3.2-1B as base model, which we then fine-tune for one epoch on the MMLU training set. This fine-tuned model achieves approx-

---

Published as a conference paper at COLM 2025
Model
Humanities
Social Sciences
STEM
Other
Overall
Fine-tuned
44.8
54.24
40.79
54.43
48.10
TTMM (1 expert)
45.25
54.92
40.98
54.30
48.41
TTMM (3 experts)
45.33
55.02
40.98
54.39
48.48
TTMM (10 experts)
45.63
55.15
41.39
54.55
48.74
TTMM (15 experts)
45.46
55.31
41.77
55.26
48.96
Table 2: Accuracy (in %) of TTMM on MMLU. Bold numbers denote the highest accuracy.
imately the same performance as instruction-tuned Llama-3.2-1B. We use the non-instruct model to avoid any confounding effect of RL-training or instruction-tuning on privileged data from the instruct model. As embedding model, we use Stella obtain similar results with MPNet (ef. Appendix D.2). We train 100 experts on the MMLU training set, using the globally fine-tuned model as initialization. We summarize the results in Table 2 and find that TTMM slightly increases accuracy. A broader evaluation other -erpleelyy tasks and non-language domains is an important direction for future work.
but
Ablations. We perform a series of ablations, which we summarize in Appendix D. Notably, we find that (1) larger embedding models need not lead to improved performance of TTMM and that (2) TTMM performs similarly when fixing the number of experts as to when selecting the number of active experts dynamically via thresholding.
Discussion and Future Work
We propose Test-Time Model Merging (TTMM) which scales the mixture of experts paradigm to orders of magnitude more experts than in tfateoof-the-art multitask methods, leading to a substantial improvement in language modeling ability. We find that TTMM can achieve performance gain close to TTT by merging few active experts into single model per query. TTMM achieves this with almost the same test-time cost as inference with the base model, and without increasing train-time FLOPs GPU memory compared to multitask fine- tuning. A limitation TTMM is the additional CPU memory required for storing all experts.
In achieving close to the performance of TTT without nearly any test-time overhead, TTMM opens up several exciting directions for future research. First, TTMM makes it feasible to re-select models online during generation or evaluation based on the growing prefix, which with TTT, is too expensive in most applications. Gururangan et al. (2023) have already shown that online re-selection can work with an ensemble selected from a smaller number of experts. Second, one may further improve the performance of TTMM using tools from the literature on model merging and MoEs aimed avoiding interference between experts and improving the routing of data to experts, e.g., by fine-tuning the router on holdout data. TTMM can further be applied to settings with privileged information by adding or removing experts dynamically based user access. Furthermore, it may be possible to leverage the large number of experts in TTMM for uncertainty estimation, e.g., by tracking also the variance of the merged parameters. Finally, many reasoning language models (called RLMs) such as DeepSeek Ri (Guo et al., 2025) are based on a MoE. architecture, and TTT has been shown to be able to improve RLMs (Zuo et al., 2025). Since TTMM can be considered scalable MoE architecture that approximates TTT, it would be particularly interesting to study whether TTMM can improve reasoning models.
Acknowledgments
This project was supported in part by the European Research Council (ERC) under the Euro- pean Union's Horizon 2020 research and Innovation Program Grant agreement no. 815943, and the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545. Ido Hakimi was supported by an ETH AI Center Postdoctoral fellowship.
10

---

Published as a conference paper at COLM 2025
References
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In ICLR, 2023.
Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot- learning. In ICML, 2025.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Lazy learning, 1997.
Marco Bagatella, Jonas Hübotter, Georg Martius, and Andreas Krause. Active fine-tuning of multi-task policies. In ICML, 2025.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 4(6), 1992.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint ArXio:2005.14166 2020.
Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan-Bo Lin, Mohit Bansal, and Gedas Berta- sius. Dam: Dynamic adapter merging for continual video qa learning, arXiv preprint ar arX1::2403.08555, 2024.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In ICML, 2022.
William S Sleveland. Robust locally weighted regression and smoothing scatterplots, Journal of the American statistical association, 74(368), 1979.
William S Cleveland and Susan Devlin. Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American statistical association, 83(403), 1988.
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Moham- mad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experss language models. arXiv preprint ar arXiv:2401.06060 2024.
Karan Dalal, Daniel Koceja, Gashon Hussein, Jiarui Xu, Yue Zhao, Youjin Song, Shihao Han, Ka Chun Cheung, Jan Kautz, Carlos Guestrin, et al. One-minute video generation with test-time training. arXiv preprint ar arXiv:2504.05228, 2025.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, 2000.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. In NeurIPS, 2021.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In NeurIPS, 2018.
11

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arviv preprint ar Xix::2407.21783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint Xri:::2501 129488 2025.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A Smith, and Luke Zettlemoyer. Scaling expert language models with unsupervised domain discovery. ar Xrv preprint ar Xix::2330 14177, 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang, Retrieval augmented language model pre-training. In ICML, 2020.
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. In ICLR, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Jonas Hübotter, Lenart Treven, Yarden As, and Andreas Krause, Transductive active learning: Theory and applications. In NewrIPS, 2024.
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test-time: Active fine-tuning of llms. In ICLR, 2025.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, 2018.
Anil K Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters, 31(8), 2010.
Vidit Jain and Erik Learned-Miller. Online domain adaptation of a pre-trained cascade of classifiers. In CVPR, 2011.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. In ACL,, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
Aecheon Jung, Seunghwan Lee, Dongyoon Han, and Sungeun Hong. Why train everything? tint a single layer for multi-task model merging. arXiv preprint arXio:2412.19998, 2024.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gener- alization through memorization: Nearest neighbor language models. In ICLR, 2020.
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of neural sequence models. In ICML, 2018
Ben Krause, Emmanuel Kahembwe, lain Murray, and Steve Renals. Dynamic evaluation of transformer language models. arx preprint ar axXio:1904.08375 2019.
Kunfeng Lai, Zhenheng Tang, Xinglin Pan, Peijie Dong, Xiang Liu, Haolan Chen, Li Shen, Bo Li, and Xiaowen Chu. Mediator: Memory-efficien Ilm merging with less parameter conflicts and uncertainty based routing. arXiv preprint arXiv:2502.04411 2025.
12

---

Published as a conference paper at COLM 2025
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensivee nlp tasks. In NeurIPS, 2019.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettlemoyer. Branch-train--mrge Embarrassingly parallel training of expert language models. arxio preprint ar Xriv:2208.03300 2022.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In ACI, 2021.
David Lowe and D Broomhead. Multivariable functional interpolation and adaptive net- works. Complex systems, 2(3), 1988.
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin- merging: Dynamic integration of modular expertise in model merging. In NeurIPS, 2024.
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurIPS, 2022.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ilms: Model fusion at test-time via perplexity optimization. In COLM, 2024.
John Moody and Christian Darken. Fast learning in networks of locally-tuned processing units. Neural computation, 122), 1989.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1,, 1964.
Changdae Oh, Yixuan Li, Kyungwoo Song, Sangdoo Yun, and Dongyoon Han. Dawin: Training-free dynamic weight interpolation for robust adaptation. In ICLR, 2025.
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathematical statistics, 33(3), 1962.
A Paszke. Pytorch: An imperative style, high-ppeformance deep learning library. arviv preprint 1r1000191,.01703, 2019.
Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. Mingle: Mixtures of null-space gated low-rank experts for test-time continual model merging. Xiv preprint ar dxXiv:2555.18833 2025.
Nils Reimers and Tryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In 1JCNLP, 2019. 2019.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of a Density Function. The Annals of Mathematical Statistics, 27(3), 1956.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir
Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture- of-experts layer. In ICLR, 2017.
Assaf Shocher, Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In CVPR, 2018.
Toby Simonds and Akira Yoshiyama. Ladder: Self-improving Ilms through recursive problem decomposition. arXio preprint arXio:2503.00735, 2025.
13

---

Published as a conference paper at COLM 2025
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurlPS, 2020.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert iims into mixture-of-experts Ilm. arXiv preprint ar 3rXi:22003.08166 2024.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint ar Xiv:::240 000:22077.0660, 2024.
Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In ICML,, 2024a.
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter efficient multi-task model fusion with partial linearization. In ICLR, 2024b.
Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arxiv preprint ar xrXio:250.09552, 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In IČLR, 2023.
Geoffrey S Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A, 1964.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Wikimedia Foundation. Wikimedia downloads, 2025. URL httts::///ump:p: wwkimedia. org.
Andrew Gordon Wilson. Deep learning is not so mysterious or different. arXiv preprint a/Xit:2503.02113, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. In NeurIPS, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arxiv preprint arXiv:2412.15115, 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arxiv preprint Sttt::2008076666 2024b.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning, In ICLR, 2024c.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models. ar Xiv preprint XrXi::2412.1990 19048, 2024a.
14

---

Published as a conference paper at COLM 2025
Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar Talupuru, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, Ahmet Üstün, et al. Bam! just like that: Simple and efficient parameter upcycling for mixture of experts. In NeurIPS, 2024b.
Qizhen Zhang, Prajjwal Bhargava, Chloe Bi, Chris X Cai, Jakob Foerster, Jeremy Fu, Punit Singh Koura, Ruan Silva, Sheng Shen, Emily Dinan, et al. Bts: Harmonizing specialized experts into generalist Ilm. arv preprint ar Xr10:2550200005, 2025.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experss with expert choice routing, In NeurIPS, 2022.
Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint ar 2rxi::2504.1600 2025.
15

---

Published as a conference paper at COLM 2025
Appendices
A Additional Related Work
17
B Equivalence of Cross-Attention and RBF Kernel
17
Proof of Informal Proposition 3.1
17
D Ablations
19
D.1 Fixing the Number of Active Experts
19
D.2 Different Embedding Models
19
D.3 Evaluation Alternative Merging Coefficients
19
Experiment Details
21
F Qualitative Examples
22
16

---

Published as a conference paper at COLM 2025
name in prr_param_namees
As (I
d[name]["A"] for in lora_dicts])
Bs torch. stack
d[name]["B"] for in lora_dicts])
delta torch. einsum) "k srir k o", Bs, As)
Figure 7: We merge LoRA adapters using Pylorch's einsum, before applying the low-rank updates to the model parameters.
A Additional Related Work
Retrieval-Augmennt Generation. An alternative to specializing model parameters at test- time (i.e., TTT) is instead specialize predictions by conditioning an autoregressive model directly on prompt-related data. This is called -trreval-augmeeneed generation (RAG, Lewis et al., 2019; Khandelwal et al., 2020; Guu al., 2020; Borgeaud et al., 2022) and motivated by the ability of LLMs to learn from context (Brown et al., 2020; Wei et al., 2022). Similarly to TTT, RAG leads to expensive inference since all local data needs to be processed at test-time. TTMM is more efficient by compressing local information into expert LoRA adapters at train-time. RAG is commonly used to endow LLMs with privileged information. Similarly, TTMM can include (& remove) information simply by adding (or removing) local experts.
B Equivalence of Cross-Attention and RBF Kernel
Assuming that all embeddings are normalized, we have
exp (-1 1 11 -0:12 = exp ( [ 1
p a exp ( 1
(5
2B
B
C Proof of Informal Proposition 3.1
Setup. Let fxx 0) be a neural network parameterized by E IRF and evaluated on in- put x X. We have loss function L(0;z) E R for data point = = (x,y) E X' x y. Let DC X' y be dataset of input-output pairs. Given a prompt x* X' and any N 1, we consider the set of N nearest neighbors to & (x*) in D (with respect to the input embedding):
D ,N,,, C D.
Let D' be any dataset of size M > 1:
D' {z),--. ...,... M C.
We first analyze single-step gradient descent with step size lj > from the same initializa-
tion 0:
1 M
0₂- 0 ---1
11
N Vo L(; 2;)
0' -
Vo C(:; z/)
M
Assumptions.
(A1) Lipschitz-smoothnes the loss. There constant G > such that for all z = (x, (x',y') and all 0:
v C(; 2)
V L(;; z')l < G G|Ix-x' x'1l.
3hhis supervised data may be constructed via self-supervision, in language modeling.
17

---

Published as a conference paper at COLM 2025
(A2) Network Lipschitz in parameters. There is constant 0 such that for all 0, 8' and all x:
(xx(); - f(x;0'| I L || 0'|l.
Proposition C.1. Suppose (A1) and (A2) hold, and let 0 be any initialization. For any prompt x² E X and N 2 1, let 0x be the model trained via single-step gradient descent with step size > 0 on Dx Let o' be the expert model trained on any D _ DD Then, if D' Dx + Ø
[1(xx=0x=) -(xx))))))=))))) S (LGG(iam((,x)) + diam(D')
(2)
Proof. We have
N VO ((0:z))
M E V ((0:2))
0
0'
11
1=1
j=1
Va L((; zi)
M Σ Va ((012))
M
Vo [(:22))) ]
Ve ((;
Thus,
1
M
1
11
0'11 - < 11
Vo ((0:2))
V. 2(0,211
(triangle inequality)
M
N
M
S G
Σ
x
x111-
(Assumption A1)
Let x be any point in D'
DDD xt
We have
1
M
1
N
G
Σ
Σ
x; - x x/11
M
N
M
S G
|| - x|| + 11x- x411)
(triangle inequality)
s G((diam (Dx,) + diimmD'))
Finally,
< 11 0'll - ()LG(diam(Dpx) + diam(D')).
(Assumption A2)
To complete the proof of Informal Proposition 3.1, we simply observe that the bound | - 0(() < (d((mam()))) + diam(D')) can be applied recursively from the shared initialization ((0) for T steps of gradient descent via the relation
|0 [++ 0(((+1) || < ||0(t (000 G diam(D')).
Unrolling this recursion proves Informal Proposition 3.1 with an additional factor T com- pared to Proposition C.1.
18

---

Published as a conference paper at COLM 2025
Model
# Active Experts
Wikipedia
GitHub (Python)
TTMM
Fixed
TTMM
Fixed
1
7.669
7.669
2.552
2.552
Llama-3.2-1B
3
7.571
7.380
2.519
2.505
10
7.510
7.509
2.492
2.491
1
7.166
7.166
2.330
2.331
Qwen2.5-11.5
7.101
7.104
2.304
2.295
10
7.080
7.080
2.287
2.287
Table 3: Perplexity comparison (lower is better) between TTMM (dynamic expert routing) and fixed expert selection for Llama and Qwen across different numbers of active experts.
Model
Humanities
Social Sciences
STEM
Other
Overall
Fine-tuned
44.80
54.24
40.79
54.43
48.10
all all-mpnet-base-v2
45.95
54.76
40.56
54.72
48.61
45.46
55.31
41.77
55.26
48.96
45.27
54.76
40.91
54.65
48.45
Table 4: MMLU accuracy (in %) comparing TTMM with different embedding models against the fine-tuned baseline. Bold indicates the highest accuracy.
D Ablations
D.1 Fixing the Number of Active Experts
In Table 3, we ablate the choice of selecting the number of active experts dynamically per prompt depending on the threshold (cf. Algorithm 2) rather than specifying fixed sparsity N. On our evaluated corpora, we find no significant difference in performance.
D.2 Different Embedding Models
In Table 4, we evaluate different embedding models on MMLU and find that TTMM outperforms the base model regardless of the choice of embedding model. Notably, the largest Qwen2-1.5B embedding model performs worse than the 100M MPNet model. A possible reason is that finding decent clustering might be significantly easier than directly solving questions, since the former only requires a broad understanding of the meaning of some keywords while the latter requires some ability of reasoning. Stella (Zhang et al., 2024a) performs the best among the tested embedding models.
D.3 Evaluation of Alternative Merging Coefficients
SIFT: Accounting for redundancy between experts. We evaluate weighting active experts according to their estimated uncertainty reduction about the response to the prompt, as proposed by Hübotter et al. (2025). Building transductive active learning (Hübotter et al., 2024), Hübotter et al. (2025) approximate the expected reduction in uncertainty about the prompt when conditioning uncertainty after seeing the i-th data point.
the i-th data point by of_ - of where of estimates the
We use the same uncertainty estimates of to weight the experts in TTMM, but instead of conditioning on individual data points represented by their embeddings, we condition on expert models represented by their centroids. For given prompt, we first retrieve the N
19

---

Published as a conference paper at COLM 2025
50
100
active expert
TTMM
Wikipedia
GitHub (Python)
2.58
7.64
2.56
Perplexity
2.54
7.62
2.52
7.60
2.50
11
10"
10a
10a
10a
10
10
10a
10a
10a
Lambda (A)
Lambda (A)
Figure 8: We evaluate weighting experts according to SIFT's uncertainty estimates, which take into account redundancy between clusters. Here, À is the hyperparameter SIFT. As À - co, SIFT selects only the centroid closest to the prompt embedding, and as À - 0, SIFT weights all clusters uniformly.
nearest centroids and then reweight them according to:
k 1-1- o2
Wx k k
k
=1 1-11 0
0
Given that embeddings are normalized, we have oj = 1.
Intuitively, these weights account for redundancy between clusters. Suppose we duplicate cluster m times, then the above weights downweigh each of the m duplicates by a factor 1/m. The cross-attention mechanism TTMM, on the other hand, would treat each duplicate clus- ter as if it were a distinct cluster, and hence increase the weight of the duplicated cluster by mI.
We evaluate SIFT in Figure 8. We find that when we have control over the created experts (e.g., by clustering the data) in TTMM, SIFT yields only negligible improvmenet over the cross-attention mechanism. This likely stems from the fact that the centroids the clusters are already well-separated, and hence the cross-attention mechanism is able to effectively weight the experts. We expect that in settings where we have less control over the experts, Seightiig could provide more significant improvement.
Weighting experts by their logit-entropy. Oh et al. (2025) propose DaWin, which weights experts by their logit-entropy. Let (x(x*) ) be the distribution over logits of expert for input x* with H (ppx(x+) denoting its entropy. Then, the weight of expert is given by
DaWin:
wk sparse-softmax, (-1H((P((x))))
Note that these weights are equivalent to the weights obtained by the cross-attention mecha- nism TTMM, up to the use of (x(px(x") (x*)) to measure similarity as opposed to & ((x*). As opposed to the inner product, which can be evaluated with very little overhead, comput- ing the entropy requires a separate forward pass with each expert. This makes it as expensive as ensembling experts at test-time. In Figure 9, compare the logit-entropy w weighting to the inner product weighting of TTMM, and do not find a performance difference.
Ablation against summarizing each expert by its clusters' centroid. As suggested in Section 3.2, evaluate whether summarizing each expert by its clusters' centroid meaningfully degenerates the performance of TTMM. We find that this appears not to be the case, shown in Table 5.
20

---

Published as a conference paper at COLM 2025
7.66
active
Inner product (TTMM)
7.64
Logib entropy (DaWin)
Perplexity
7.62
7.60
0.05
0.10
0.15
0.20
Temperature
Figure 9: We evaluate ensembling models with inner product weighting (like in TTMM) and logit-entropy weighting (like DaWin). We evaluate on Wikipedia with Llama-3.2-1B as base model. Solid lines are with sparsity 0.1, translucent lines are with t 0.01.
Selection criterion
Perplexity (lower is better)
Summarized by centroid
14" 2x11 ExcD, 11 (xx)11
7.669
Not summarized by centroid
7.633
Table 5: We evaluate the selection of experts, without approximating each expert by its clusters' centroid as suggested in Section 3.2. The evaluation on the Wikipedia corpus with Llama-3.2-1B as base model, K 1000 total experts and single active expert.
E Experiment Details
Parameter
Value
Number of clusters K
100
Learning rate
2e-4
LoRA rank
64
LoRA alpha
16
Adam epsilon
1e-8
Adam beta
(0.9, 0.999)
Weight decay
0.01
Batch size
4
Sparsity
0.01
Table 6: Hyperparameters.
Dataset
Source
Wikipedia
GitHub (filtered to Python)
Table Datasets.
All models were trained on the first 1024 tokens of each training example. Testing was con- ducted on the full input length, with the context size limited to 24 tokens. For each dataset, the test sets were constructed by selecting single random example from each cluster's holdout set, ensuring that the evaluation approximately covers the data of all clusters.
21

---

Published as a conference paper at COLM 2025
50
7.9
Evalucatio prefixx length
Perplexity
40
7.8
7.7
7.6
10
20
Query prefix length
Figure 10: On Wikipedia and with Llama-3.2-11 as base model, we evaluate different configurations of query prefix length (the number of tokens used as query) and evaluation prefix length (the number of tokens ignored for evaluation). Around 40 to 50 tokens, the performance gain of including more tokens in the query begins to diminish.
The hyperparameters for TTT match those used to train the expert models, except for learning rate of 5e-4. The models were trained for five epochs and the best checkpoint was selected for evaluation.
Unless noted otherwise, all ablations are with Llama-3.2-1B as the base model. Further, unless noted otherwise, we use "number of active experts" upper bound to the mean number of active experts across test examples.
Dataset size for expert training. Both the Wikipedia and the GitHub (Python) corpora contain roughly 7M documents. Therefore, since each train document is at most 1024 tokens long, each expert is trained on at most =7M tokens we train 1000 experts. This is relatively small dataset size for training experts, compared to datasets used for training common MoE or multitask models.
Preventing Information Leakage. The TTT setting (and also TTMM) requires specification of prompt to obtain task-specific model. In language modeling datasets that do not have the natural structure of prompt and response, it would be unfair use the response as prompt. Using the response prompt would invalidate the evaluation, as the test data is leaked to the model before its evaluation. To prevent this, we split individual data points into prefix (prompt) and suffix (response) similar to Hardt & Sun (2024), the prefix for obtaining the task-specific model, and evaluate the model on the suffix. Table summarizes the choice of prefix per dataset. When not dynamically re-selecting models before the generation of each token (or after every N tokens), the prefix must be sufficiently informative for the suffix. On Wikipedia, we find this to be around 50 tokens (cf., Figure 10).
Prefix length (in tokens)
Wikipedia GitHub (Python)
50
200
Table 8: Overview of prefix lengths used per evaluation dataset.
F
Qualitative Examples
Clustering: To investigate the clustering, we use an LLM to summarize each clus- ter's content with title based on 10 data points of each cluster. Tables 9 and 10 list some of those cluster titles.
2. Expert Selection: Table 11 shows examples of selected experts for different queries.
22

---

Published as a conference paper at COLM 2025
3. Model Generation: We find that expert models generate distinctly different text. Table 12 gives examples of generations from empty strings, while Table 13 shows examples of generations from prompts. Within their subject of expertise, expert models tend to hallucinate less or at later stage than the base model.
Cluster number
Cluster title
118
Aviation History and Aviation-Related Topics
119
Historical Figures and Notable Americans
120
Historical Figures and Events in the United States
612
Honey Bees and their Scientific Classification
613
Small Mammals and Insects
614
Insect Taxonomy and Systematics
968
Swiss Geography and Mountains
969
Rivers of Germany
970
Historical Places and Landmarks in Europe
971
German Municipalities and Towns
Table 9: Wikipedia cluster titles
Cluster number
Cluster title
17
Kivy GUI Framework
18
Tkinter GUT Framework Development
19
Multi-Touch Plugin for Alignak
427
Hidden Markov Model Implementation
428
Sparse Matrix Decomposition and Optimization
429
Logistic Regression with L2 Regularization 3D Printing and CAD Design
513
514
3D Mesh Generation and Manipulation Tools
979
Database Schema Definition and Migration Tools MongoDB Database Schema Management
980
Table 10: GitHub (Python) cluster titles
Query input
Closest expert
2nd-closest expert (281) NBA and WNBA Teams and Seasons
3rd-closest expert (311) Sports and Athletics
Basketball is great sport
(309) Women's
Basketball
Competitions and Leagues
Albert Einstein was an astounding physicist
(169) Historical Physicists and Scientists with Nobel prizes
(104) Tufts University Alumni and Faculty in Science
(88) British Scientists and Mathematicians
UNESCO World Heritage Site
(716) Cities and Urban Landmarks
(752) Asian Landmarks and Cultural Sites
(906) Places in South Africa
Table 11: Example Wikipedia expert selection
23

---

Published as a conference paper at COLM 2025
Model
Generated text
Base Model
Ex 1.
Question: What is the greatest common factor of 30 and 15? Answer:
Ex 2.
Question: Design a Python function snippet to Check High Handling personal items: Sanitizing Frequently Touched Items for Analysis for Beginners. Implement if/ else or switch/can case statements to handle different conditions related to the Reliability. Provide detailed comments explaining your control flow and the reasoning behind each decision.
Expert (1):
Ex 1.
Politicians and Political Parties from Southeast Asia
The 2018 South Korean bleelections was held in South Korea on 9 January 2018. The by-elections were held to fill two vacant seats in the National Assembly. The first by-election was held for the vacant seat the constituency Busan 1st District, and the second for the vacant seat in Seoul 6th District.
Ex 2.
The National Congress of Australia's First Peoples (NCAFP) is organisation that advocates for the interests of Aboriginal and Torres Strait Islander Australians the Australian Parliament. It a member of the Council of Australian Traditional Owner's 's Organisations (CATOO).
Expert (280):
Ex The 2017 was the New England Patriots' 48th in the National Football League (NFL), their 58th overall, their 51st playing home games at Gillette Stadium and their sixth under head coach Bill Belichick. The Patriots entered the season as the defending Super Bowl champions and were looking to defend their AFC East division title and make their third consecutive Super Bowl.
American Football Teams and Players
Ex
The 2006 Houston Texans season was the franchise's 7th season in the National Football League (NFL) and the 1st under head coach Gary Kubiak. They improved playoffs for the first time since 2003. They won the AFC South for the first time since 2002 and defeated the Indianapolis Colts in the Wild Card round 31-17.
their 4-12 record from 2005 and made the
Table 12: Generation from empty string inputs
24

---

Published as a conference paper at COLM 2025
Model
Generated text
Base Model
Ex 1.
The politics in the US, the UK and elsewhere have been dominated by the "Brexit" vote to leave the European Union. While many people in the UK and elsewhere have been unhappy with the outcome, it is worth remembering that the vote was very close. The referendum was held June 23rd, 2016, and the result was 52% in favour of leaving and 48% favour of staying.
Ex 2.
Super Bowl 50: 5 Things to Know The 2016 Super Bowl will be played on February 7, 2016. It will be the 50th Super Bowl and the second to be played in the new 20,000 seat Levi's Stadium Santa Clara, California. Here are five things to know about the Super Bowl 50: 1. The Super Bowl is played on the first Sunday in February each year. The last Super Bowl was played on February 7, 2015.
Expert (1):
Ex 1.
Politicians and Political Parties
The politics in the Cook Islands are characterised by multi-party system and electoral calendar dominated by the 2018 general election. The Cook Islands has had democratic political system since 1965, with the Cook Islands being the first country in the Pacific to hold elections for the Parliament. Ex 2.
from Southeast Asia
Super Bowl XLIV was an American football game that was held on Sunday, February 7, 2010, at Superdome, New Orleans, Louisiana. The 2010 Super Bowl was the 45th edition of the American Football League Championship Game and the 46th overall Super Bowl.
Expert (280):
Ex 1.
American Football Teams and Players
The politics in the 1990 NEL season involved the 1990 NFL Draft, the 1990 Pro Bowl, and the 1990 NFL season ending controversy over the 1990 NFC Championship Game. The 1990 season was the first NFL season in which the league was divided into two conferences, with the NFC and AFC, and the first to end in controversy. Ex 2.
Super Bowl LIV was an American football game played to determine the champion of the National Football League (NFL) for the 2019 season. The American Football Conference (AFC) champion Kansas
City Chiefs defeated the National Football Conference (NFC) champion San Francisco 49ers, 31-20, on February 2020, at Hard Rock Stadium Miami Gardens, Florida.
Table 13: Generation from input strings, input is in bold
25


Reviews:
[{"review_id": "yFMy6zgOjy", "reviewer": "Reviewer_7eGv", "summary": "This paper introduces a new method for test time adaption of language models (TTMM) but where new LoRA adapters are trained based on training data distribution grouped into clusters. At test time, based on the prompt, their method introduces a technique to select the relevant adapters based on prompt embedding and then weight merge the adaptors. This final adaptor merged model is used for inference and evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper identifies correctly that test time adaptation (TTT) on the granularity level of each prompt is prohibitively expensive for each prompt. That finetuning based on each prompt is not pragmatic is a valid problem to solve.\n\nTo that end:\n\n1. The introduced method is comparatively simple to understand and implement which is an advantage.\n2. The notion of clustering is relevant as this allows to overcome the issue of extreme finegrained task finetuning.\n3. The results show that the performance degradation is relatively small compared to TTT method.", "reasons_to_reject": "The main concern I have with the conclusions is that there is no discussion on the Pareto frontier of the latency and the performance degradation compared to TTT.\n\nIf the advantage claim of the paper's approach is in its latency with \"test time\" adaptation, then we need to have notion of apples-apples comparison of inference times.\n\nFor example, Figure 4 shows the inference times for TTMM. But looking at the figure 4b, it is not clear to me why this method is faster than TTT. For example, at 100 experts, the method takes 1 sec (100ms) to generate 20 tokens. If I do not consider distributed inference, for a 3B model, that is pretty high, almost 10x for the 10 experts case.\n\nSo we need to see a tradeoff between how the performance degrades with less number of experts vs the gain in time. That would also allow the method to select the sweet spot.\n\nA second question on the method on which I see no discussion is the concept of \"experts\". It seems that there is a disconnect between how the experts are selected based on an embedding model and how the weights are merged. Since there is no training of the gating or router mechanism here, the weights for merging are selected through a dot product that comes from the embedding model. Why is that a good weighting mechanism when the model has never adapted to those weights before.", "questions_to_authors": "1. Can you clarify how the number of clusters were selected - it seems like the perplexity keeps improving with more experts/clusters and it goes on till 1000 experts. However, as I mentioned above, it is not likely a singular dimension that selects the number of clusters, the test time latency should also be a factor for selecting K, but I do not see a discussion on that.", "extra_scores": {}}, {"review_id": "uopstevg9F", "reviewer": "Reviewer_Ltqr", "summary": "This paper proposes Test-Time Model Merging (TTMM), a test-time optimization method for large language models. It works by creating a large number of adapter models from the training data and dynamically selecting and composing these models based on the input during inference to achieve task-specific tuning. The theoretical analysis shows that TTMM approximates test-time training (TTT) under appropriate conditions, and it demonstrates the ability to construct specialized models with significantly less cost (100 times faster) compared to TTT.\n\nEDIT after rebuttal: I raised my reccomendation to 7.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This method significantly reduces the real-time overhead during test-time while improving the performance (perplexity) of language models. This is a desirable characteristic for applications, and the method could be particularly useful for applications that require real-time performance.\n  * However, the cost is not entirely eliminated; it requires separate storage to keep the LoRAs. Therefore, the title \"Essentially Free\" seems to be an overstatement.", "reasons_to_reject": "* The paper evaluates the proposed method based on perplexity but does not mention downstream tasks at all. While perplexity might suffice for purely theoretical results, discussing the impact on actual task performance is essential when considering real-world applications of language models. Although the authors mention this in Future Work (final section), it's unclear if there was a specific reason why the results were not included in this paper. It's a very good study I think, but this point is quite disappointing.\n* As mentioned in the experiments, the method's scaling seems to have limitations with the current approach, as increasing the number of experts to 1000 did not yield further improvement.\n* The performance of this method is likely significantly affected by the training data clustering method, but the analysis of the clustering method itself is very limited. Technically, this is an essential hyperparameter for applying this method and requires deeper analysis.", "questions_to_authors": "* The theoretical analysis in Section 3.2 includes unrealistic approximations such as single-step optimization. While understandable for analysis efficiency, is it possible to replace these with assumptions closer to actual training settings? Also, it seems that the effect of the reduction due to applying LoRA in this method is not sufficiently discussed. Is there any consideration of whether synthesizing a large number of trained LoRAs can maintain the same model expressiveness as tuning the original parameter space directly?\n* As mentioned in \"Reasons To Reject,\" the discussion about clustering itself seems limited. How sensitive is the proposed method to other clustering algorithms or changes in hyperparameters within the algorithm? If experimental results are unavailable, is it possible to provide some qualitative observations?\n* The embedding used for selecting LoRAs is thought to have a direct impact on model performance, but the paper only mentions results using all-mpnet-base-v2. Please provide justification for this choice. Also, can you discuss how the behavior changes specifically with different embeddings? Relatedly, can you mention any guidelines for selecting embeddings?\n* The conclusion in the paper states that \"TTMM with 10 active experts achieves close to the accuracy of TTT,\" and while this is broadly true, Table 1 suggests a different relationship between TTMM and TTT for Wikipedia and GitHub. Specifically, the order of perplexity is reversed. This could be seen as reflecting the methodological characteristics of TTMM, but is there any discussion on this point?\n* Regarding the point that performance did not improve with an increased number of experts, you only state that \"additional experts do not capture any meaningful new information\". Is there a more detailed analysis or consideration of methods to avoid this issue?\n* You mention that model merging interference is an important issue, but you do not seem to deeply consider it in this study. Is there anything you can mention regarding this?\n* The process of determining the prefix length for GitHub is not clear. Is it possible to provide any information on this?", "extra_scores": {}}, {"review_id": "O3i8yL1phv", "reviewer": "Reviewer_tuLa", "summary": "This work proposes Test-time model merging (TTMM) as an approximation of test-time training with better test-time efficiency. Precisely, TTMM first conducts train-time clustering to cluster the training dataset. For each cluster, the authors adopt a small LoRA to learn k experts. The centroid embeddings are calculated to represent cluster-specific features for the cross-attention weight during test-time model merging. TTMM is evaluated mainly on Wikipedia and GitHub (python) via perplexity.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The idea of TTMM is sound, and the experiments also verify the effectiveness of TTMM on Wikipedia and GitHub (python) in perplexity. It is likely to be a practical paradigm for test-time approaches.\n2) This work is well-written and easy to follow.\n3) The model designs and motivations are well clarified.", "reasons_to_reject": "1) The authors merely evaluate TTMM on two datasets Wikipedia and GitHub (python) via perplexity. It is suggested that more tasks are needed to comprehensively verify the effectiveness of TTMM. Moreover, Perplexity is adopted for evaluation. Other metrics in practical downstream benchmarks should be considered to answer whether the proposed TTMM could function well in diverse real-world tasks.\n2) Some MoE efforts do adopt more experts recently (e.g., DeepSeek-V3 adopts 256 routed experts). Will this setting cover some of the advantages of TTMM proposed in this work? It is interesting to verify its effectiveness on MoE-based architectures. Moreover, evaluations on larger LLMs will make the conclusion more solid.\n3) From the results, it seems that the selections of (a) the number of experts, and (b) the number of activated experts are non-trivial and diverse in different datasets and tasks. The authors are suggested to give a discussion on the wiser parameter selection.", "questions_to_authors": "1) The real overall run time (including train-time and test-time costs) for TTT and TTMM of the current evaluation setting should be given.\n2) In Figure 1, it seems that TTMM performs consistently worse than TTT, while in Table 1, TTMM outperforms TTT on two datasets. The authors could give more discussions on these results.", "extra_scores": {"ethics_concerns_details": "NA"}}]

Author rebuttal:

Meta-review: This work presents a new method for efficient test time adaptation. Instead of training one adapter per tasks, the proposed method scales up the number of experts, but then efficiently merges for less overhead during inference. 

The reviewers agree that the proposed method is simple to understand and implement. It provides a practical solution for test time adaptation, which is otherwise unpractical in many real-world settings. 

The reviewers note some limitations with current work, namely lack of evals on downstream tasks, and some better discussion of tradeoff between number of experts and latency. The authors provided additional evidence of both counts during rebuttal, so I am providing a positive recommendation for this work.

**Pros**

- Method is simple to understand and implement (7eGv,tuLa)

- Practical version of test-time adaptation with minimal performance loss compared to per prompt tuning (7eGv,Ltqr,tuLa)

- writing well motivates the proposed approach, and it is easy to follow (tuLa)

**Cons**

- performance trade-off between number of experts and increase in latency is not properly explained (7eGv)

  - clarified in rebuttal

- unclear how experts are selected (7eGv,tuLa)

  - clarified in rebuttal

- the paper slightly over-claims benefits of TTMM by calling it "essentially free". compared to TTT, TTMM requires more memory to keep adapters (Ltqr, 7eGv)

- no downstream metrics (Ltqr, tuLa)

  - fixed in rebuttal 

- limitations in how much TTMM can scale (Ltqr)

- unclear how it compares with recent models with more experts (tuLa)
Decision: Accept