R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Authors: Zhongyang Li
Ziyue Li
Tianyi Zhou
Venue: ICML 2025 poster (2025)
Keywords: mixture-of-experts
test-time optimization
multimodal models
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: A test-time method that dynamically adjusts Mixture-of-Experts routing weights to boost multimodal model performance without any retraining

Abstract: In large multimodal models (LMMs), the perception of non-language modalities (e.g., visual representations) is usually not on par with the large language models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on challenging downstream tasks. 
This weakness has been recently mitigated by replacing the vision encoder with a mixture-of-experts (MoE), which provides rich, multi-granularity, and diverse representations required by different downstream tasks. The performance of multimodal MoE largely depends on its router, which reweights and mixes the representations of different experts for each input. However, we find that the end-to-end trained router does not always produce the optimal routing weights for every test sample. To bridge the gap, we propose a novel and efficient method ''**R**e-**R**outing in **T**est-**T**ime (R2-T2)'' that locally optimizes the vector of routing weights in test-time by moving it toward those vectors of the correctly predicted samples in a neighborhood of the test sample. We propose three R2-T2 strategies with different optimization objectives and neighbor-search spaces. R2-T2 consistently and significantly improves state-of-the-art LMMs' performance on challenging multimodal benchmarks of diverse tasks, without training any parameters in the base model. Our code can be accessed here.

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Zhongyang Li'
Ziyue Li Tianyi Zhou
R2-T2 (MaA)-BB)
ShareGPT4V-7B
LLaVA-NeXT-13B Improvement by R2-T2
Abstract
MoAl-7B
Ann-GGmmin-HD-3B
5QA-IMG
In large multimodal models (LMMs), the percep- tion of non-language modalities (e.g., visual rep- resentations) is usually not on par with the large language models (LLMs)' powerful reasoning ca- pabilities. deterring LMMs performance on chal- lenging downstream tasks. This weakness has been recently mitigated by replacing the vision encoder with a mixture-of-experts (MoE), which provides rich, multi-granularity, and diverse repre- sentations required by diverse downstream tasks. The performance of multimodal MoE largely de- pends on its router, which reweights and mixes the representations of different experts for each in- put. However, we find that the end-to-end trained router does not always produce the optimal rout- ing weights for every test sample. To bridge the gap, we propose novel and efficient method "Re- Routing in Test-Time (R2-T2)" that locally opti- mizes the vector of routing weights in test-time by moving it toward those vectors of the correctly predicted samples in a neighborhood of the test sample. We propose three R2-T2 strategies with different optimization objectives and neighbor- search spaces. R2-T2 consistently and signifi- cantly improves state-of-the-arr LMMs' perfor- mance on challenging multimodal benchmarks of diverse tasks, without training any parameters in the base model. Our code can be accessed here.
86.3
A12D
MME-P
05.0
L714.0
00.4
673
TextvQA
78.6
67.8 666
08
75.3
862
MMBench
60.2
63
574
993
70.2
71.2
662
GQA
77.8
CVBench
Figure I. R2-T2 applied to MoAI-7B compared against 7/8/13B VLMs on 8 benchmarks. R2-T2 significantly enhances perfor- mance of the 7B base MoE model, surpassing a recent 13B VLM.
perts, which provides finer-grained expertise and skills, yet selecting a constant number of experts for each input (Lep- ikhin et al., 2020). In MoE, the sparse selection of experts is achieved through a router, which determines the weight of each candidate expert based on the input only experts with nonzero weights are selected (Fedus et al., 2022). MoE then aggregates the outputs of the selected experts according to their weights. Hence, the router and its produced routing weights play important roles in MoE's inference cost and output quality.
1 Introduction
As the most widely studied LMM, many vision language models (VLM) adopt architecture composed of a vision encoder and an LLM (Zhu et al., 2023), which are both pre-trained and then aligned by further finetuning so the L.M can include the vision encoder's output in its input as additional tokens. The alignment is usually obtained through a lightweight projection layer or Q-former (a Trans- former model) converting the vision encoder's output to LLM tokens. Despite the broad usage of this architecture, the capability of a vision encoder is usually much more lim- ited than the LLMs (i.c., the "modality imbalance") (Schrodi et al., 2024). So the visual features cannot cover all the in- formation required by different reasoning tasks performed by L.Mss. Moreover, the alignment module may lead to
Mixture-of- offtpre-ff-Exeerts (MoE) have achieved remarkable suc- cess in scaling up the size and capacity of large language and multimodal models (LLMs and LMMs) (Shazeer et al., 2017) without (significantly) increasing the inference cost. Specifically, it allows us to increase the total number of ex-
'Department of Computer Science, Johns Hopkins University, Baltimore, USA 'Deparmeent of Computer Science, University of Maryland, College Park, USA. Correspondence to: Tianyi Zhou <tianyiuumd.coe
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expeee
Before re-routing
After re-routing
routing weighes
LANG
routing Laux
LUX
L Lave
weights
LLANG
LAUX Lemu
R2-T2
ISELS
Answer: (A)
experts
Answer: (B)
experts
ANN Question Simularity 0.4220
Question Similarity 0.4469
Queation Simulanit:: 0.4150
Question:
Considering the relative positions of the chair (annotated by the box) and the teunis racket in the image provided, where is the chair (annotated by
Question: What
Question: number things large objects the left side of the red matte thing or objects that are the small red rubber ball? Answer:
Question: number of things
the thing that both the right side of the large red matte ball and the big blue objecr? Answer: brown
both on the left side of the big
hox) located with respect the tenmis racket?? Seleet from the following choices. (A) above (R) helow
red shiny object and in front of the rubber Answer:
Figure 2. An example how R2-T2 optimizes the routing weights, Given the test sample, it finds ANN in the reference set correctly predicted samples with similar questions. In the example, the test sample requires reasoning about positional relationships. R2-T2 identifies relevant KNN samples, adjusting the top-1 expert from LaNo (aligning visual features with language) to Laxx (aligning visual features with auxiliary computer vision features). This expert shift is crucial correcting the initial wrong answer.
information bottleneck from the visual perception to the reasoning (Yao et al., 2024).
Test-fime without training any model parameters.
Since routing weights encode the choices of experts with essential knowledge and key skills acquired by the input task, and motivated by the assumption that knowledge and skills are usually transferable across different tasks, we posit that the routing weights of successful tasks can provide crit- ical clues for optimizing the routing weights of new task. Specifically, leverage the similarity in task embedding space, which may reflect the knowledge or skill sharing be- tween tasks, and modify the routing weight vector of test task by imitating its nearby successful tasks. While the task embedding space, optimization objective, and the number of update steps can vary and their design choices may re- sult different performance, this innovative mechanism of optimizing routing weights "re-routing" in test-time (R2- T2) focuses to correct the mistakes made by the routers in existing multimodal MoF, e.g., extracting object detection features for a task mainly depending on the text information in an input image, and thus turns various failed cases into success. Rather than finetuning the whole model, R2-T2 training-fre and aims to maximize the potential of MoE in the reasoning tasks by LMMs.
Recent advances in LMMs replace a single vision encoder with mixture of encoders (Lin et al., 2024; Lee et al., 2025; Zong et al., 2024; Shi et al., 2024), which turns out to be an effective and low-cost approach to mitigate modality imbal- ance and alignment bottleneck. In multimodal MoE, each expert is encoder mixer of sensory inputs that fo- cuses specific type of features, e.g., object classes, text in images, spatial relations, dense captions, segmentation, etc., so the LLM can select the information acquired by any given downstream task from the concatenated or fused features from the MoE, through a router that trained in an end-to-end manner produce the weights of all the experts adaptive the input task.
Although multimodal MoE achieves remarkable success in enhancing the performance of existing LMMs, the choice of experts or the routing weights for individual instances are not always optimal due to the limitations the router's design and the diversity of potential downstream tasks com- pared the tasks used to train the router. The suboptimality of routing substantially limits the performance and general- ization of multimodal MoE on unscen tasks. As illustrated in Figure 2, the base model initially selects a sub-optimal expert (e.g., laANa) for spatial reasoning task, leading to incorrect predictions. This has been verified on recent mul- timodal MoE models: as shown Table 2, compared to the original routing weights of base models, the optimal (ora- cle) routing weights improves the accuracy by ' 10% most evaluated LMM benchmarks. To avoid the expensive cost of re- re-training a router on a much larger dataset, in this paper, we investigate how to improve the routing weights in
Following the above idea, explored several novel strate- gies for test-time routing weight optimization. They all modify the routing weights of a test task/sample based on a representative set of tasks/samples on which the multi- modal MoE achieves correct or high-quality outputs. While the oracle routing weights are achieved by minimizing the test sample's loss, for practical approach, we propose replace the oracle loss with surrogate, i.e., weighted average of losses of nearby reference samples, and apply multiple steps of "neighborhood gradient descent (NGD)"
2

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
to minimize the surrogate. In addition, we investigate kernel regression and mode finding, which do not require gradient descent. The former moves the routing weights to a kernel- weighted sum of nearby reference tasks' routing weights in a task embedding space, while the latter moves the routing weights to the nearest mode on the distribution of reference tasks' routing weights. Evaluating these strategies on two recent multimodal MoE models across eight challenging benchmarks, we find that R2-T2 significantly outperforms models twice its size, as shown in Figure 1. Our analysis reveals that NGD progressively refines routing, increasing correct predictions while mitigating the original router's s over-reliance on a single expert. Case studies confirm that test-time re-routing enhances domain-specific reasoning, demonstrating R2-T2's ability to adapt multimodal MoE models without additional training, unlocking greater gener- alization and robustness.
of large-scale neural networks (Shazeer et al., 2017). By dynamically selecting a subset of specialized expert modules for each input (Li et al., 2023b), MoE reduces computational overhead while maintaining high expressive power (Shazeer et al., 2017; Zoph et al., 2022). In the context of LLMs, MoE has been shown to improve both training efficiency and generalization across tasks (Artetxe & Schwenk, 2019). Works such as Switch Transform- ers (Fedus et al., 2022) and GShard (Lepikhin et al., 2020) have demonstrated the effectiveness of MoE in scaling up model capacity without prohibitive increases in training costs. In multimodal settings, MoE has been explored to address the modality alignment problem (Goyal et al., 2021), where different experts handle distinct modalities or specific tasks. However, the optimal utilization of experts heavily relies on the effectiveness of routing mechanisms, which remains an active area of research.
Our main contributions can be summarized below:
Routers and Routing Strategies are the cornerstone of any MoE-based architecture, responsible for determining which experts are activated for each input (Li & Zhou, 2024; 2025). Traditional routers, such as softmax gating func- tions (Shazeer al., 2017), compute a weighted combina- tion of experts based on input embeddings. Despite their simplicity, these routing strategies often face challenges in achieving optimal expert assignment (Lepikhin et al., 2020; Zoph et al., 2022), particularly in unseen or highly diverse test scenarios. Recent works have proposed ad- vanced routing strategies, including routing via reinforce- ment learning (Rosenbaum et al., 2017), early-exit (Li et al., 2023c), and task-specific allocation (Shi et al., 2024). How- ever, these approaches typically focus on training-time opti- mization, leaving test-time adaptability largely unexplored. R2-T2 introduces an efficient method to refine routing weights dynamically during inference, ensuring better align- ment with task-specific requirements and improving overall model robustness across diverse multimodal benchmarks.
We proposed a novel problem of R2-T2 that bridges significant performance gap on multimodal MoE.
We developed three practical R2-T2 strategies that shed several critical insights into expert re-routing,
Our R2-T2 considerably advances the performance of multimodal MoE several recent benchmarks of challenging tasks for LMMs.
2 Related Work
Large Multimodel Models has emerged as a powerful paradigm for integrating language and non-language modalities, such as images (Radford et al., 2021), audio (Ao et al., 2021), and video (Zellers et al., 2021), to perform complex reasoning tasks. Recent advancements have been driven by the fusion of pretrained LLMs with multimodal encoders (Peng et al., 2023; Tsimpoukelli et al., 2021; Alayrac et al., 2022), enabling the models to process and generate cross-modal content effectively. Works such as Flamingo (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023a) demonstrated the potential of aligning vision and language modalities through carefully designed bridging modules. However, these models often fall short in richness or align- ment with the reasoning capabilities of L.MSs (Bubeck et al., 2023; Bommasani et al., 2021). To address this, techniques have been proposed, such as contrastive pretraining (Rad- ford al., 2021: Yuan al., 2021) and feature fusion mechanisms (Lu et al., 2019). Yet, efficiently capturing diverse modal interactions across different tasks remains bottleneck (Baltrušaitis et al., 2018), highlighting the need for more adaptive mechanisms in multimodal reasoning.
Test-Time Optimization has been explored by adapting models dynamically during inference to improve general- ization (Li et al., 2025). For example, (Wang et al., 2022) propose test-time adaptation, which fine-tunes model pa- rameters on test data distributions using entropy minimiza- tion or self-supervised learning. Similarly, (Sun et al., 2020) introduce test-time training, where models are up- dated via auxiliary tasks (e.g., rotation prediction) during inference. However, these methods require modifying the base model's parameters, leading to significant computa- tional overhead and potential instability when deployed on resource-conster constraiee systems. Unlike prior test-time opti- mization methods that update model weights, R2-T2 solely optimizes the routing weights of a frozen MoE model with- out retraining any model parameters.
Mixture-of-Expers has become a prominent architec- tural choice to enhance the scalability and efficiency
3

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-ENeee Expoeftssxstssss
Routing weights of neighbors Neighbors' gradient descent direction
Routing weights of the test sample in re-routing
Routing weights after re re-routing WWeighted average of neighbors' routing weights
Re-routing direction
Cr + (1
a)p
o)FF (1
f
(1 o/F
V
T3
T2
22
222
T3
(a) Neighborhood Gradient Descent
(b) Kernel Regression
(c) Mode Finding
Figure 3. Illustration of R2-T2' test-time re-routing mechanism with three strategies. (a) Neighborhood Gradient Descent: Optimizes r using gradients derived from neighbors' loss functions (V.4, V2 and V,l3 for the nearest neighbors), weighted by their similarity the test sample. (b) Kernel Regression: Estimates as a weighted average of neighbors' routing weights (f), and further optimizes it through binary search between f and initial weights r find the optimal coefficient a. (c) Mode Finding: Iteratively updates r through weighted interpolation between curreent weights and the local average in routing weight space, shifting towards the densest region.
Test- Re-Routing
3.1 Gradient Descent
MoE trains router to reweight experts for each input. How- ever, such an end-to-end trained router may not always pro- duce optimal weights for challenging or out-of-distribution samples at test-time, whereas sub-optimal weights can dras- tically degrade the performance of MoE on diverse down- stream tasks. The importance of routing weights has been broadly demonstrated on eight benchmarks in our experi- ments: The large performance gap between the base model (using the router's routing weights) and the oracle (using the optimal routing weights) in Table 2 implies the potential merits of optimizing the routing weights in the test-time.
The gradient descent method uses the gradient of an ob- jective function L(r) to update r for multiple steps until convergence or when certain stopping criteria have been fulfilled. In every step, we apply
r r
(1)
where À is learning rate determined by a scheduler. We discuss the two choices of L(r) in the following.
Oracle (upper bound) assumes that we know the ground truth label y for x, which is a cheating setting that can provide an upper bound of the gradient descent method. In this setting,
To address this problem, Test-Time Re-Routing (R2-T2) introduces a dynamic test-time re-routing mechanism that adapts the routing weights for each test sample based on similar samples in reference set-a set of samples which the MoE's outputs are correct preferred. Specifi- cally, given a reference set of samples {(,,, b-) and their corresponding routing weights {r:) on which the model makes correct prediction (i.e., f( r;) = y:). for a new test sample x, the goal of R2-T2 to find a better routing weight vector r for x that leads a more accurate and higher-quality output fx; r).
(Tr) ([f(z,r),
(2)
where el. is the loss function (c.g., cross-entropy L2 loss) measuring the discrepancy between the model output f(z, r) and the ground truth y. Although this is not appli- cable in real scenarios, it serves as a performance ceiling to reveal the degradation caused by sub-optimal routing weights and evaluate the effectiveness of other methods.
Neighborhood Gradient Descent (NGD) is practical approach that uses the loss functions of the nearest neighbors of x in the reference set to estimate the gradient of r, i.e..
In the following, we will introduce three core strategies, illustrated in Figure 3, to optimize based on the neighbors of x the reference set, Nxx).. according to a similarity metric. These strategies are developed with different optimization objectives (e.g., loss surrogate, regression, mode finetuning, etc.) and neighbor-search spaces (c.g., routing weights, task embedding, etc.).
L(r) EieN(e)) K( 2)) ([f r).
(3)
EvEN(e) K(#;,2))
By incorporating loss information from the neighborhood of xr, NGD enables label-free, test-time adaptation mecha- nism. This effectively aligns with the successful routing patterns in the reference set. This ensures that r exploits the routing for relevant reference examples without requiring access to the oracle loss.

---

R2-T2: Re-Routing in Test-Time for Multimodal casture-of-Experrs
3.2 Kernel Regression
either kNN e-ball, i.e.,
Kernel regression predicts by the weighted average of the neighbors' routing weights (ri)ieN(x)) i...
((xx) _ arg
min Σ Ed(x,,
(9)
AC2", Sk SEA
M(x) - {i [n] d(z;, S c).
(10)
A _ EiEN(a) K (I;,
EieN(a) ((x,,))
(4)
Embedding Instead of directly applying an existing kernel function K ((-)) and distance metric d(..-) the raw in- puts 2, and 2, we can replace z and with their embedding E (x) and E(r,), where E(-) is pre-trained embedding model applied to the task description of each sample.
where -(.-)) is kernel function, e.g., Gaussian kernel, Matern kernel, etc. In the experiments, found that di- rectly setting f-1 f already brings non-trivial improvement.
However, f does not take the router- prooured initial into account and may not fully capture the nuanced dependencies required for optimal performance. To further optimize r, we conduct a binary search on the straight line between r and f:
Table I. Summary of reference and evaluation benchmarks, If the reference dataset contains more than 5,000 samples, we randomly
select 5,000 to ensure balanced evaluation.
Task Type
Reference
Size
Evaluation
Size
VQA-V2
5,000
MMBench
2.374
General
Visual7W
5,000
MME-P
2,114
- or + (1 (1.
(5)
Visual
COCO-QA
5,000
CVBench
2,638
Understanding
CLEVR
5,000
GQA
1,590
The search goal is find the optimal a minimizing the objective L(r). i.c.,
Knowledge-
A-OKVQA
5,000
SQA-IMG
2,017
Based
TQA
5,000
A12D
3,087
Reasoning
MathVista
5,000
a" E mig Lar + (1 - a))).
(6)
Optical Character Recognition
ST-VQA
5,000
TextVQA
5,734
DocVQA
5,000
This refinement step balances the kernel regression estimate with the router's s original routing weights. It includes as a special case (when a = 0) and can further enhance the accuracy and robustness of the model's predictions.
4
Experiments
4.1 Experimental Setting
Models We evaluate two multimodal MoE models: MoAI (Lee et al., 2025) and MoVA (Zong et al., 2024), each leveraging specialized experts for vision-language tasks. MoAl has six experts: (1) Visual Experts process auxiliary CV features (IAux), align visuals with language (IIAN)), and capture spatial relationships (IæLL)): (2) Language Experts integrate external knowledge (Laux), link language visu- als (LIma), and maintain coherence (Lezur). Further details about MoAI experts are provided in Appendix A. MoVA in- cludes seven experts, incorporating SAM (Zou et al., 2024) to enhance the vision encoder with specialized knowledge.
3.3 Mode Finding (Meanshift)
Mode finding aims to move r towards the highest density region of the distribution p(r) for the reference routing weights {ri)/- - It applies the following update for multiple steps until convergence.
r or + (1- (
where a controls the step size and F the weighted average routing weights defined below (different from f).
Reference datasets and evaluation benchmarks Our eval- uation covers three task categories: general visual under- standing, knowledge-based reasoning, and optical character recognition. Table I summarizes the reference datasets and evaluation benchmarks, including their dataset sizes. See Appendix B for details.
F A EIENG) K(rar) EieNTo K(ri,r)
Unlike kernel regression, mode finding identifies the densest region in the routing weight space (so the kernel K(:--) and neighborhood. (-) are applied to r instead of xt), represent- ing the most consistent configurations among nearby refer- ence samples. This makes it effective for capturing the dom- inating patterns in the local distribution of routing weights.
Evaluations We adopt standard evaluation protocols for each benchmark. For MME-P, performance is assessed using two metrics:(1) Accuracy, measuring the correct- ness of single question per image, and (2) Accuracy+, requiring both questions per image to be answered cor- rectly.The final score is the sum of these two metrics, with a maximum of 2,000 (Fu et al., 2024). For other benchmarks, accuracy is the primary metric (Yin et al., 2023). We compute the mean score across benchmarks
3.4 Neighborhood and Embedding Space
Neighborhood The choices of neighborhood definition and the embedding space in which to apply the kernels are im- portant the final performance. For the former, we can use
5

---

R2-T2: Re-Routing in Test-Time for Multimodal centuure-of-Epeeere
Table Comparison of three R2-T2 methods (INN with l: 5) on MoVA and MoAl (base models). Oracle has access to the ground truths and provides upper bound. NGD significantly improves base models and performs the best.
Method
MMBench
MME-P
SQA-IMG
A12D
TextvQa
GQA
CVBench
CVBench
MoVA (base model)
74.3
1579.2
74.4
74.9
76.4
64.8
61.6
62.3
Mode Finding
75.2
1587.1
74.9
75.8
77.3
65.7
62.5
63.2
Kerne Regression
77.9
1610.6
76.4
78.5
79.9
68.3
65.2
65.9
NGD
81.2
1645.3
79.1
81.8
83.2
71.5
68.3
68.9
Oracle (upper bound)
87.6
1735.4
87.3
88.4
89.5
76.2
72.5
73.2
MoAl (base model)
79.3
1714.0
83.5
78.6
67.8
70.2
71.2
59.3
Mode Finding
80.8
1725.2
84.1
79.8
66.5
71.4
70.0
60.1
Kernel Regression
83.7
1756.7
86.2
82.6
71.2
74.5
74.6
64.5
NGD
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
Oracle (upper hound)
92.1
1860.2
93.8
91.2
79.6
83.2
84.0
76.8
as #benchmark (Sinal + Smmp-)), where Storal is the sum of all benchmark scores except MME-P, and Smmp-e is the normal- ized MME-P score.
behind larger models-R2-T2 achieves substantial perfor- mance gains and outperforms 7/8/13/34 competitors across most benchmarks through effective test-time re-routing. In addition, applying R2-T2 to MoAI-7B results in a signifi- cant performance boost, establishing it as highly competitive against larger models. These results highlight the effective- ness of R2-T2 in unlocking the potential of smaller models, enabling them to match or even surpass the performance of significantly larger VLMs.
Baselines R2-T2 introduces test-time re-routing, a problem not addressed in prior work. To assess its effectiveness, we compare it against multiple R2-T2 variants and base models. Additionally, we benchmark R2-T2 against state-of-the-art VLMS across scales, as shown in Table 3.
We use fixed hyperparameters across all benchmarks with- out per-task tuning. determined via experiments on small- scale benchmarks independent of our evaluation datasets. See Appendix for details.
Inference efficiency trade-off While R2-T2 introduces additional operations beyond the base model's inference pipeline, achieves near-oracle performance with moderate computational overhead (Table 4). To ensure hardware- independent comparison, measure computational costs in FLOPSs. The base model requires 9.9T FLOPs per case. Mode finding adds only 1.8T FLOPs, leading to a 1.5% accu- racy gain. Kernel regression and R2-T2 require 6-7x more FLOPs due to loss computations over five neighbors, yet R2- T2 (KNN, NGD) achieves the highest accuracy improvement (+5.9%) while maintaining competitive efficiency.
4.2 Main Results
Comparison of different R2-T2 methods Tables 2 sum- marizes the performance of R2-T2 methods on the Mo and MoAI models across eight benchmarks. Among all evaluated methods, &NN Neighborhood Gradient Descent (NGD) emerges as the most effective, delivering significant improvements over the pretrained base models. For MoAI- 7B, R2-T2 enhances performance significantly, achieving +6.9% on MMBench, a +66. 1-oint increase on MME-P, and a +6.8% gain on TeA.VQA. Similarly, on MoVA-BB, yields notable improvements of +5.9% on MMBench, +71.5 points on MME-P, and +5.7% on TeXtVQA. These consistent gains across diverse benchmarks highlight the ability of R2-T2 to optimize routing weights effectively, enabling better utilization of expert modules for improved model performance. Notably, ANN NGD achieves results close the Oracle upper bound, which relies on ground truth labels during test-time and is thus infeasible in practice. Our method, without accessing labels, captures 70-80% of the potential improvement, demonstrating its effectiveness.
4.3 Ablation Study
We analyze how each component contributes to the per- formance and robustness of ANN NGD, with all studies conducted on MoAI. Results averaged across 8 test benchmarks detailed in Section 4.1, with individual results and further analysis provided in Appendix D.7.
Neighborhood selection compare two strategies: e-ball (ra- dius e = 0.2 to 0.8) and ANN (k- = 3 to 20), as shown in Table 5. The results demonstrate that kNN with consistently achieves better performance across most tasks, outperforming both smaller neighborhoods that may lack sufficient context and larger ones that could introduce noise. While e-ball shows stable performance across different ra- dius, suffers from inherent limitations: a fixed radius threshold may yield too few neighbors in sparse regions or excessive neighbors in dense regions, leading to inconsistent performance. The kNN approach provides more reliable
5
Comparison with state-of-the-art VLMs In Table 3, we compare our approach with state-of-the-ar VLMs of var- ious sizes (7B, 8B, 13B, 34B) across benchmarks. When applied to the pretrained MoVA-7B whhhhic initially lags
6

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Exxe -fepeests
Table Comparison R2-T2 (kNN, NGD) with state-of-the-ar ssioo-languagge models on cight benchmarks.
VLM
MMBench
MME-P
SQA-IMG
A12D
TextVQa
GQA
CVBench
CVBench 3D
7B Models
InstructB1.IP-7B (Dai 2023)
36.0
60.5
50.1
56.7
Qwen-V V1.-BB (Bai et al., 2023)
67.1
62.3
63.8
59.4
Qwen- Owen-Vi--Chat-78 (Bai et al., 2023)
60.6
1488.0
68.2
57.7
61.5
mPP.UG-OW1-7B (Ye et al., 2023)
46.6
967.0
58.9
mPL.UG--OW2-7B (Ye al., 2024)
1450.0
68.7
58.2
62.9
ShareGPT4V-7B (Chen al., 2025)
68.8
1567.4
68.4
67.3
65.8
63.4
60.2
57.5
8B Models
Mini-Gemini-HD-8BB (Li et al., 2024)
72.7
1606.0
75.1
73.5
70.2
64.5
62.2
63.0
LLaVA-NeXT-8B (Liu al., 2024)
72.1
1603.7
72.8
71.6
64.6
65.2
62.2
65.3
Cambrian -8B (Tong et al., 2024)
75.9
1647.1
74.4
73.0
68.7
64.6
72.3
65.0
13B Models
BLIP2-13B (Li al., 2023a)
28.8
1294.0
61.0
42.5
InstruciBLIP-133B (Dai et al., 2023)
1213.0
63.1
50.7
Mini-Gemini-HD-1333 (Li et al., 2024)
68.6
1597.0
71.9
70.1
70.2
63.7
53.6
67.3
LLaVA-NeXT-13B (Liu et al., 2024)
70.0
1575.0
73.5
70.0
67.1
65.4
62.7
65.7
Cambrian1-13B (Tong 2024)
75.7
1610.4
79.3
73.6
72.8
64.3
72.5
71.8
34B Models
Mini-Geemini-HD-34BB (Li ct al., 2024)
80.6
1659.0
77.7
80.5
74.1
65.8
71.5
79.2
LLaVA-NeXT-34B (Liu et al., 2024)
79.3
1633.2
81.8
74.9
69.5
67.1
73.0
74.8
Cambrian1-344 (Tong al., 2024)
81.4
1689.3
85.6
79.7
76.7
65.8
74.0
79.7
Ours
MoVA-7B
74.3
1579.2
74.4
74.9
76.4
64.8
61.6
62.3
R2-T2 (MoVA-7B)
81.2
1645.3
79.1
81.8
83.2
71.5
68.3
68.9
MoAI-7B
79.3
1714
83.5
78.6
67.8
70.2
71.2
59.3
R2-T2 (MoAI-7B)
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
Table FLOPs of different methods (I:NN with 1: == 5) on MM- Bench using MoAI-7B as the base model.
to its ability to effectively capture similarity relationships in high-dimensional embedding spaces while being less affected by the curse of dimensionality (Cristianini, 2000).
Inference FLOPs (T) Accuracy
Method
steps
per case
(%)
Rase Model (MoAT-7R)
79.3
Embedding model directly impacts the neighborhood qual- ity. which in turn influences the gradient updates. In Table 7, we compare four embedding models. The results show that NV-Embed- V2 achieves consistent improvements of 3.2% over Sentence- indicating ability provide more discriminative feature representations that better capture semantic relationships between samples.
Mode Finding
10
10.7
80.8
Kemnel Regression
R2-T2 (KNN, NGD)
10
67.5
85.2
Oracle (upper bound)
Table Ablation study R2-T2 (&NN, NGD) with different
choices of neighborhood on MoAl.
e-ball Parameter
ANN
Gradient descent steps significantly affect both conver- gence and performance. Experiments with 5, 10, 20, and 50 steps assess the trade-off between cost and accuracy. As seen in Table 8, increasing the step count from 5 to 10 sig- nificantly improves performance (76.6 - 80.7), indicating that more iterations enhance optimization. Beyond 10 steps, performance saturates (80.5 at 20 steps, 80.7 at 50), sug- gesting diminishing returns. Thus, 10 steps offer the best balance between performance and efficiency.
Avg.
Parameter
Avg.
0.2
76.5
k 3
78.6
0.4
77.9
k 5
80.7
0.6
78.9
A 10
79.4
0.8
77.7
A 20
76.6
and generally superior results. This suggests that maintain- ing a fixed number neighbors not only ensures consistent computational cost but also provides sufficient information for effective test-time re-routing.
4.4 Case Studies
Kernel choice is critical for determining how similarity is modeled in high-dimensional spaces, which directly affects gradient updates in NGD. In Table 6, we compare four different kernel functions. The results consistently show that the Gaussian kernel outperforms other kernel functions across all tasks, with up to a 4.4% accuracy improvement over the linear kernel. Its superior performance may due
Accuracy Transition Analysis Figure illustrates the tran- sition of predictions as NGD progresses over ten steps. Dur- ing Step 0 to Step 4, 17.22% of incorrect predictions are corrected, and by Step 10, a total of 28.12% incorrect predictions have been converted to correct ones. Meanwhile,

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experr
Table Ablation study of R2-T2 (kNN,
Table 7. Ablation study of R2-T2 (ANN,
Table Ablation study R2-T2 (ANN,
NGD) with kernel choices on MoAl.
NGD) with embedding models MoAL.
NGD) with NGD steps MoAI.
Kernel
Avg.
Emhedding Model
Avg.
#Step
Avg.
Linear (Cortes, 1995)
76.3
Sentence- Bert (Reimers, 2019)
77.5
76.6
Polynomial (Cortes, 1995)
77.7
Stella-En-I --eEn------- (Kusupati al., 2022)
78.5
10
80.7
Matern (Williams Rasmussen, 2006)
78.7
Cte-Qwen2-7B
2023c)
78.7
20
80.5
Gaussian (Williams Rasmussen, 2006)
80.7
NV-Embed-V2 V (Loc et al., 2024)
80.7
50
80.7
LANG
LIMG
SELE
AUX
AUX
Top-1 expert before re-routing (1696 & 254
(a) Top-1 expert after re-routing (1950
Top-1 expert before re-routing (39 L 649 X)
(b) Top-1 expert after re-routing (688 X)
Figure Top-1 expert transitions to correctiincoreett preditions on CVBench 2ADD after re-routing. For transition to correct predictions in (a), the main transition patterns include LaNg LIMG LAUX and LAUX. For the transition to the incorrect predictions in (b), the main transition pattern include LANGG to LAUX, LImG and LAUX.
become more balanced, showing that R2-T2 effectively di- versifies expert selection. Furthermore, transition patterns differ between correctly and incorrectly predicted samples. In correct cases (Figure (a)), re-routing typically shifts LaNG to LIMG- In incorrect cases (Figure (b)), transitions often involve ANG to LAUX This may indicate occasional mismatches in the re-weighted routing. Crucially, the num- ber of cases shifting from correct to incorrect is significantly lower than those transitioning from incorrect to correct. The overall improvements outweigh potential misclassifications. validating R2-T2 as an effective optimization strategy.
Correct
65.8%
69.1%
71.1%
73.9%
x- 10.83%
0.52%
6.43%
4.78%
x-
X 6-/33399
0.81%%
.x0..9%
Incorred
34.2%
30.9%
28.9%
27.8%
27.1%
26.1%
10
Step
Figure 5. Transition between correct and incorrect predictions CVBench 20/3 during NGD steps of R2-T2 from Step 0 to 10. NGD keeps turning more incorrect predictions to correct. only 2.31% % correct predictions become incorrect through- out the optimization process. As the optimization converges in later steps, the routing weight changes become smaller, reducing the number of prediction shifts.
Example Case: Spatial Reasoning Improvement Figure demonstrates how R2-T2 rectifies spatial reasoning failure. The test question asks, "where is the chair located with re- spect to the tennis racket?". Initially, the model selects LANG (language- aligned visual expert), which prioritizes textual alignment but fails to capture positional relationships. R2- T2 addresses this by retrieving nearest neighbors from the reference set with similar spatial queries. By dynamically adjusting routing weights, R2-T2 elevates L to the top-1 position. Laux integrates features from open- world object detection (Lee et al., 2025; Minderer et al., 2023), enabling more precise interpretation of spatial layouts.
Expert Shift Patterns Figure 4 illustrates top-1 expert tran- sitions before and after re-routing, where (a) shows transi- tions leading to correct predictions and (b) those leading to incorrect predictions. The original router over-relies on LANG- limiting model adaptability. After re-routing, many samples shift from LANg to LIMG LAux, and LAUX leading to improved accuracy. This indicates that the pretrained router excessively favors laang, preventing optimal expert utilization. Notably, samples that were initially correct be- fore re-routing exhibited more balanced expert distribu- tion, whereas those initially incorrect depended heavily on I LANG- After routingigg, expert distributions in both cases
Additional transition pattern cases and details are provided in Appendix D.8 and for further insights.
8

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expeer
5 Conclusions
international conference on computer vision, pp. 4291- 4301, 2019.
We introduce R2-T2, a novel test-time re-routing method that enhances multimodal coxture-of-Expertt (MoE) mod- els without additional training. By dynamically adjust- ing routing weights based reference samples, R2-T2 corrects suboptimal expert selection, improving model generalization. We propose and evaluate three strate- gies-Neighhoorrood Gradient Descent, Kernel Regres- sion, and Mode demoonstatnnn their effectiveness across multiple multimodal benchmarks. R2-T2 consistently outperforms the base MoE model and rivals oracle-based optimization methods, highlighting the potential of test adaptation for more efficient and adaptive expert utilization.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint MoXiv:2108.0755000 2021.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E.,, Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. /2712, 2023.
Chen, L., Li, J., Dong. X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference Computer Vision, pp. 370-387. Springer, 2025.
Impact Statement
This work contributes to advancing the field of Machine Learning by introducing novel insights and methodologies. While our research primarily focuses on technical improve- ments, its broader societal impact depends downstream applications. We do not identify any immediate ethical or societal concerns that require explicit discussion.
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., and Gird- har, R. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
References
Cortes, C. Support- networks. Machine Learning, 1995.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A.. Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few- shot learning. Advances in neural information processing systems, 35:237 23736,,, 2022.
Cristianini, N. An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, 2000.
Dai, W., Li, J., Li, D.,, Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general- purpose vision- lisionnlanguage models with instruction tuning. arxiv 2023. arXiv preprint arXiv:2305.0663 2, 2023.
Ao, J., Wang, R., Zhou, L., Wang, C., Ren, S., Wu, Y., Liu, S., Ko, T., Li, Q., Zhang, Y., al. SpeechtS: Unified- modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021.
Du, Y., Li, C., Guo, R., Cui, C., Liu, W., Zhou, J., Lu, B., Yang, Y., Liu, Q. Hu, X., et al. Pp-oerv2: Bag of tricks for ultra lightweight ocr system. arXiv preprint arXiv:2109.08. 034,03314,4444 2021.
Artetxe, M. and Schwenk, H. Massively multilingual sen- tence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the association for computa- tional linguistics, 7:597-610, 2019.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J. Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. 1(2): 3, 2023.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., Wu, Y., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https: //arxiv. abs 22006 100006.33944
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal machine learning: A survey and taxonomy. IEEE trans- actions pattern analysis and machine intelligence, 41 (2):423 443, 2018
Goyal, A., Didolkar, A., Lamb, A., Badola, K., Ke, N. R., Rahaman, N., Binas, J., Blundell, C., Mozer, M., and Bengio, Y. Coordination among neural mod- ules through a shared global workspace. arXiv preprint arXiv:2 197, 2021.
Biten, A. F. Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D. Scene text vi- sual question answering. Proceedings of the IEEE/CVF

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Li, Y., Zhang, Y., Wang, C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemint Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 2024.
Li, Z. and Zhou, T. Your mixture-of-expert Ilm is se- credy an embedding model for free. arXiv preprint arXiv:2410.0083 2024.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700- 6709, 2019.
Li, Z. and Zhou, T. Sparser nixture-of-adapters with cross- layer generalization. In Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Asso- ciation for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pp. 3988 4002, 2025.
Johnson, J., Hariharan, B., Van Der Maaten, L.., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Li, Z.., Ren, K., Jiang, X., Shen, Y., Zhang, H., and Li, D. Simple: Specialized model-samee sample matching for domain generalization. In The Eleventh International Conference on Learning Representations, 2023b.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Li, Z., Ren, K., Yang. Y., Jiang, X., Yang. Y., and Li. D. Towards inference efficient deep ensemble learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 8711-8719, 2023c.
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., and Hajishirzi, H. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 4999- 5007, 2017.
Li, Z., Li, Z., and Zhou, T. C3po: Critical layer, core-expert, collaborative pathway optimization for test-time expert re-mixing. arXiv preprint arXiv:2504.07960 2025.
Liang. Y., Li, M., Fan, C.,, Li, Z., Nguyen, D., Cobbina, K., Bhardwaj. S., Chen, J., Liu, F,, and Zhou, T. Colorbench: Can vims see and understand the colorful world? com- prehensive benchmark for color perception, reasoning, and robustness. arXiv preprint wrXiv:2504.105/1 2025.
Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., et al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30245 2022.
Lin, X. V., Shrivastava, A., Luo, L., Iyer, S., Lewis, M., Ghosh, G., Zettlemoyer, L., and Aghajanyan, A. Moma: Efficient carly-fusion pre-training with mixture of modality-aware experts. arXiv preprint arX1v:2407.21770.. 2024.
Lee, B.-K., Park, B., Won Kim, C.,, and Man Ro, Y. Moai: Mixture of all intelligence for large language and vision models. In European Conference Computer Vision, pp. 273-302. Springer, 2025.
Lee, C., Roy, R., Xu, M., Raiman, J., Shoeybi, M., Catan- zaro, B., and Ping, W. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
preprint 4rXiv:2405.1744
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z.,, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2025.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O.,, Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint ar 2006 166666600.6668...... 2020.
Ji, Li, D., Savarese, S., and Hoi, S. 2lip-2: Bootstrapping language-imen image pre- treiirain with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742, PMLR, 2023a.
Lu, J., Yang, J., Batra, D., and Parikh, D. Hierarchical question-image co-attention for visual question answer- ing. Advances neural information processing systems, 29, 2016.
10

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pre- training task-agnostic visiolinguistic representations for aaion-and-languaaee tasks. Advances in neural informa- tion processing systems, 32, 2019.
Rosenbaum, C., Klinger, T., and Riemer, M. Routing
works: Adaptive selection of non-linear functions for multi-task learning, arXiv preprint arXiv:1711.01239, 2017.
Lu, P., Mishra, S., Xia, T., Qiu, L.,, Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 15:2507-2521, 2022.
Schrodi, S., Hoffmann, D. T., Argus, M., Fischer, V., and Brox, Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- representation learning. arXiv preprint arXiv:2404.07983, 2024.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:23 2023.
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual ques- tion answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Ma, W., Chen, H., Zhang, G., de Melo, C. M., Chen, J., and Yuille, A. 3dsrbench: A comprehensive 3d spatial reasoning benchmark. arXiv preprint arXiv:2412.07825, 2024.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q. Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1701. 701.06338 2017.
Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprini arXiv:2203.10244, 2022.
Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y. Shi, H., et al. Eagle: Exploring the design space for multi- modal ilms with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Mathew, M., Karatzas, D., and Jawahar, C. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200- 220000222,,2222 2021.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Minderer, M., Gritsenko, A., and Houlsby, N. Scaling open- vocabulary object detection. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Sys- tems, volume 36, pp. 72983-73007. Curran Associates, Inc., 2023.
Sun, Y. Wang. X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts, 2020. URL https://arxiss org/ abs 1909 13231.
Minderer, M., Gritsenko, A., and Houlsby, N. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36, 2024.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal Ilms. arXiv preprint arXiv:2406. 16860, 2024.
Peng. Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F. Multimodal few-shot learn- ing with frozen language models. Advances in Neural Information Processing Systems, 34:200-212 212, 2021.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763, PMLR, 2021.
Wang. Q., Fink, O., Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Reimers, N. Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:198..10884
Williams, C. K. and Rasmussen, C. E. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
2019.

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expers
Wu, H., Zhang, Z., Zhang, E., Chen, C., Liao, L.., Wang. A., Li, C., Sun, W., Yan, Q., Zhai, G., et al. Q-bench: A benchmark for general-purpont foundation models low-level vision. arXiv preprint arXiv:2309. 14181, 2023.
Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., and Liu, Y. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.12 13046. 2024.
Yang, J., Ang, Y. Z.., Guo, Z.., Zhou, K., Zhang, W., and Liu, Z. Panoptic scene graph generation. In European Conference on Computer Vision, pp. 178-196. Springer, 2022.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. Designing effective sparse expert models. arXiv preprint SXXiv:2222009000 2(3): 2022.
Yao, L., Li, L., Ren, S., Wang, L., Liu, Y., Sun, X., and Hou, L. Deco: Decoupling token compression from semantic abstraction in multimodal large language models. arXiv preprint Xiv:22405.20985 2024.
Zou, X., Yang. J., Zhang, H., Li, Li, L., Wang, J., Wang, L., Gao, J., and Lee, Y.J. Segment everything everywhere all at once. Advances in Neural Information Processing Systems, 36, 2024.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint ar S3Vv:230414472. 2023.
Ye. Q. Xu, H.. Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q. Zhang, J.. and Huang, F. owl2: Revolution- izing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Yuan, X., Lin, Z., Kuen, J., Zhang, J., Wang, Y., Maire, M., Kale, A., and Faieta, Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Yue, X., Ni, Y., Zhang, K., Zheng. T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., al. Mmmu: A massive multi- dialii-disiilieee multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farbadi, A.. and Choi, Y. Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems, 34:23634 23651, 2021.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10055 2023.
Zhu, Y., Groth, O., Bernstein, M., and Fei-Fei, L. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004, 2016.
12

---

R2-T2: Re-Routing in Test-Time for Multimodal Eexture-of-Experts
A
MoAI Experts
MoAl integrates specialized computer vision models and expert modules to achieve comprehensive scene understanding:
External CV Models: Four computer vision models provide complementary capabilities: (I) panoptic segmentation (Cheng al., 2022) for object identification and localization, (2) open- woen-world object detection (Minderer al., 2024) for recognizing diverse objects beyond predefined categories, (3) scene graph generation (Yang et al., 2022) for understanding object relationships, and (4) optical character recognition (OCR) (Du et al., 2021) for text understanding. These models provide auxiliary information that enhances MoAl's visual perception.
Cross-Modal Capabilities: The expert modules are designed to facilitate effective cross cross-modal interactions:
Visual Experts: LAx connects visual features with structured CV outputs through cross cross-attention, LANG aligns visual representations with language semantics, while SELF maintains spatial awareness through self-attention.
Language Experts: LAux integrates verbalized CV outputs with language understanding, Lime grounds language in visual context, and LEELF ensures coherent text generation.
The combination of specialized CV models and cross-modal experts enables MoAI to bridge the gap between detailed visual perception and high-level language understanding. This architecture is particularly effective for tasks requiring both fine-grained visual analysis and natural language reasoning.
B Evaluation Benchmarks and Reference Datasets
We conduct evaluations using a diverse set of reference datasets and task-specific benchmarks (Liang et al., 2025). For general visual understanding, we use four reference datasets: VQA-V2 (Goyal al., 2017), Visual7W (Zhu et al., 2016), CLEVR (Johnson et al., 2017), and COCO-QA (Lu et al., 2016). For knowledge-basee reasoning, which requires leveraging external knowledge, we include A-OKVQA (Schwenk et al., 2022), TQA (Kembhavi et al., 2017) and Math Vista (Luu et al., 2023). For optical character recognition (OCR), we employ ST-VQA (Biten et al., 2019), DocVQA (Mathew et al., 2021). To ensure a balanced evaluation, we randomly sample 5,000 instances from datasets exceeding this size.
Correspondingly, we evaluate on task-specific benchmarks. For general visual understanding. these include MMBench (Liu al., 2025), MME-P (Fu et al., 2024), CVBench (Tong et al., 2024), and GQA (Hudson Manning, 2019). For knowledge-based reasoning, we evaluate SQA-IMG (Lu et al., 2022) and AI2D (Kembhavi et al., 2016). TextVQA (Singh et al., 2019) is evaluated for OCR.
Reference Datasets
VQA-V2 (Goyal al., 2017): Focuses on open-ended visual question answering, requiring models to answer questions about images. Tasks include object recognition, attribute identification, and scene understanding. Contains 1.IM questions across +00K+ COCO images, with balanced annotations to reduce language bias.
Visual7W (Zhu et al., 2016) Specializes in 7-type visual QA ("what," "where," "when," "who," "why," "how," and "which"), emphasizing grounding answers in image regions (e.g.. "Where the cat?" with bounding box annotations). It includes 327K QA pairs, challenging models on spatial reasoning and causal explanations.
CLEVR (Johnson al., 2017): A synthetic benchmark for compositional visual reasoning. Tasks involve counting objects, comparing attributes, and logical operations (e.g., "Are there more red cubes than blue spheres?"). Contains 100K rendered 3D images and 853K questions, designed to test systematic generalization.
COCO-QA (Lu et al., 2016): Automatically generates QA pairs from COCO image captions for basic visual understanding. Questions fall into four categories: object, number, color, and location (e.g. "What color is the car?"). Includes 117K QA pairs, serving as lightweight evaluation for object-centric reasoning.
A-OKVQA (Schwenk et al., 2022): Requires commonsense and external knowledge for visual QA (e.g.. "Why is the person wearing a helmet?"). Distinguishes between direct perception ("What is this?") and hoowledge-augmentee reasoning. Contains 25K questions with crowdsourced explanations.
13

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
TQA (Kembhavi al., 2017): A multimodal machine comprehension dataset designed to test reasoning over middle school science curricula. It contains 1,076 lessons with 26,260 questions, combining text, diagrams, and images. Questions require parsing complex scientific concepts and reasoning across multiple modalities, making it more challenging than traditional QA datasets. The dataset is split into training, validation, and test sets, with no content overlap, ensuring robust evaluation of models' ability to integrate and reason over multimodal information.
Vista (Lu et al., 2023): A multimodal math reasoning benchmark combining visual understanding (dia- grams/plots) and textual problem-solving. Contains 6.14 problems testing abilities like geometric reasoning. equation parsing, and chart interpretation. Highlights the stark gap between human performance (91.6% on text-only tasks) and state-of-the-art Al models (58. 98),), particularly in visual-textual integration and multi-step reasoning.
ST-VQA (Biten et al., 2019): Evaluates scene text understanding in visual QA. Questions require reading text in images (c.g.. "What is the store name?"). Includes 23K questions across diverse scenarios (signboards, documents, etc.), with strict answer normalization.
VQA (Mathew et al., 2021): Focuses on document image understanding. Tasks include extracting information from tables, forms, and invoices (e.g., "What is the invoice number?"). Contains 50K questions on 12K document images, testing OCR and layout understanding.
Evaluation Benchmarks
MMBench (Liu et al., 2025): A comprehensive benchmark for multimodal understanding and generation. Tasks span image captioning, visual entailment, and fine-grained attribute QA. Includes 2,374 pairs with hierarchical evaluation dimensions (perception, reasoning, knowledge).
MME-P (Fu et al., 2024): Evaluates multimodal event understanding through paired questions (e.g., before/after event prediction). Contains 2,114 pairs covering temporal, causal, and counterfactual reasoning in video/text contexts.
CVBench 2D/3D (Tong et al., 2024): unified benchmark for 2D and 3D vision tasks. 2D tasks include depth estimation and object detection (1,438 pairs), while 3D tasks focus on point cloud registration and mesh reconstruction (1,200 pairs).
GQA (Hudson & Manning, 2019): Tests compositional reasoning over real-world images. Questions use functional programs (e.g., "Select then compare") ensure compositional validity. Includes 1.590 pairs with explicit scene graph grounding for error analysis.
SQA-IMG (Lu al., 2022): science QA benchmark with diagrammatic reasoning. Questions combine textbook diagrams and textual context (e.g.. "Which process shown in the diagram?"). Contains 2,017 pairs spanning biology, physics, and chemistry.
A12D (Kembhavi et al., 2016): Focuses on diagram interpretation for K-12 science. Tasks include diagram labeling, relation extraction, and multi-step inference (e.g., "What happens after step 3?"). Contains 3,087 pairs with annotated diagram primitives (arrows, labels).
Te (Singh et al., 2019): Requires text-aware visual QA (e.g., answering "What brand?" from text in images). Contains 5,734 pairs with focus OCR-VQA integration, using real-world images with scene text.
C Hyperparameter Choices
To ensure robust and fair evaluation, we fixed set of hyperparameters across all benchmarks. This approach maintains consistency, prevents task-specific optimizations, and allows for an unbiased comparison of performance.
The selected hyperparameters are as follows: cosine annealing schedule with a learning rate ranging from x 10-2 to 1 x 10-5, neighborhood selection is performed using INN with 5, the number of NGD steps is fixed at 10, the Gaussian kernel is used for kernel- bernel-based methods, and NV-Embed-V V2 is adopted as the embedding model. These values are applied uniformly across all evaluated tasks.
14

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Eppers
Hyperparameter Selection Strategy Rather than tuning hyperparameters separately for each benchmark, determined these values through controlled experiments Qhench (Wu al., 2023) that do not overlap with evaluation benchmarks. This ensures that hyperparameter selection general applicability.
selection is independent of the test sets, minimizing the risk of overfitting while maintaining
Additionally, our ablation studies (Section 4.3) confirm the effectiveness of these choices. Variations in key hyperparameters, such as NGD steps and neighborhood size, show that our selected values strike a balance between performance and efficiency, supporting their suitability across diverse benchmarks.
Additional Analysis
D.1 Evaluation on Additional Benchmarks
To further validate the effectiveness of R2-T2, we evaluate on two additional challenging benchmarks: MMMU (Yue et al., 2024) and ChartQA (Masry et al., 2022), as requested by reviewers.
Table 9. Performance of R2-T2 T2 on additional benchmarks MMMU and ChartQA.
Method
MMMU
ChartQA
Base (MoAI-7B)
55.7%
67.4%
R2-T2 (MoAI-7B)
61.3% (+5.6)
71.6% (+4.2)
Table 9 demonstrates that R2-T2 consistently improves performance across diverse multimodal tasks, achieving significant gains of +5.6% MMMU and +4.2% on ChartQA. These results reinforce the robustness and generalizability of our approach across different types of multimodal reasoning tasks.
D.2 Comparison with Additional Baselines
We evaluate R2-T2 against additional baseline methods provide comprehensive comparison, including ensemble- based approaches and retrieval-augmente methods.
D.2.1
ENSEMBLE AND NOISY ROUTING BASELINES
Table 10. Comparison of R2-T2 with ensemble-based baselines on MoAI-7B.
Method
Average
Base (MoAI-7B)
74.5%
Multiple Sampling
74.9%
Noisy Routing Ensemble
75.4%
R2-T2 (MoAl-7B)
80.7% %
As shown in Table 10, ensemble-based methods provide only marginal improvements (0.9%), whereas R2-T2 delivers substantial improvement of +6.2%. These results validate the effectiveness of R2-T2 beyond what additional computation alone achieve.
D.2.2 COMPARISON WITH RAG AND IN-CONTEXT LEARNING
We compare R2-T2 with RAG and ICL approaches using Qwen-VL. as the base model, since MoAI and MoVA do not support interleaved input.
Table 11 shows that RAG improves modestly (61.7% - 64.1% %) and ICL to 62.9%, while R2-T2 boosts MoAI from 74.5% 80,7% demonstrating that leverages the reference set more effectively than traditional retrieval approaches.
15

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expee Exxeerrts
Table 11. Comparison of R2-T2 with RAG and In-Context Learning approaches.
Method
RAG
ICL
0-shot (base)
61.7%
61.7%
I-shot
63.1% (+1.4%)
62.4% (+0.7%)
2-shots
63.6% (+19%)
62.7% (+1.0%)
3-shots
63.9% (+2.2%)
62.8% (+1.1%)
5-shots
64. 1% (+2.4%)
62.9% (+1.2%)
R2-T2 (MoAT-7B)
80.7% (from 74.5%)
D.3 Robustness Analysis
D.3.1 PERFORMANCE WITH LIMITED TASK COVERAGE
We evaluate R2-T2 on 3DSRBench (Ma et al., 2024), a dataset focused on rare spatial reasoning cases, to assess performance when reference samples have limited coverage for certain task types.
Table 12. Performance of R2-T2 on rare spatial reasoning tasks (3DSRBench).
Method
3DSRBench Accuracy
Base (MoAI-7B)
45.2% 49.7 (+4.5%)
R2-T2 (MoAI-7B)
Despite lower reference coverage for spatial reasoning tasks, R2-T2 still delivers a 4.5% improvement, demonstrating robustness even with limited task-specific reference samples.
D.3.2 REFERENCE SET SIZE ANALYSIS
We analyze the impact of reference size on performance by varying the size from 1/200 the full reference set.
Table 13. Impact of reference set size on R2-T2 performance.
Reference Set Size
Average Score
Base (MoAL, 0 reference)
74.5%
R2-T2 (1/200 reference)
74.6%
R2-T2 (1/100 reference)
74.6%
R2-T2 (1/50 reference)
74.8%
R2-T2 (1/10 reference)
77.5%
R2-T2 (1/2 reference)
79.8%
R2-T2 (full reference)
80.7%
Random selection
74.7%
Table 13 shows that even with 1/10th of the original reference set size, R2-T2 provides notable improvement (+2.9%). However, random selection yields minimal improvement (+0.2%), emphasizing the importance of well-curated reference sets.
D.3.3 MISMATCHED REFERENCE SET ANALYSIS
We test R2-T2 with mismatched reference sets to analyze robustness when incorrect reference data is used.
Results show that R2-T2 provides significant gains with task-relevant reference sets but offers minimal improvement when references are mismatched, highlighting the importance appropriate reference selection.
16

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experes Experetsss
Table 14. Performance with mismatched reference (using OCR subset for knowledge-based VQA).
Method
Knowledge-based VQA
Base (MoA1-7B)
83.5%
R2-T2 (Task-relevant reference)
88.3% % (+4.8%)
R2-12 (OCR subset as reference)
83.8% (+0.3%)
D.4 Computational Efficiency Analysis
D.4.1 MEMORY USAGE COMPARISON
We provide detailed GPU memory usage analysis across different embedding models.
Table 15. GPU memory usage and accuracy trade-offss for different embedding models.
Method
GPU Memory Usage
Average Accuracy
Base (MoAl-7B)
18GB
74.5%
R2-T2 em
27GB
80.7%
R2:T2 all_mini_yv
20GB
77.5%
R2-122 Bn-la--n--1.5BV3
22GB
78.5%
R2:T2 Gte Qwen2-7B
31GB
78.7%
Table 15 demonstrates scalable trade-offs between memory usage and accuracy. Smaller embedding models like all mini v6 still provide significant accuracy gains with lower memory overhead.
D.4.2 LATENCY ANALYSIS
We measure inference latency on RTX A6000 to assess the computational overhead of R2-T2.
Table 16. Average running time comparison on RTX A6000.
Method
Avg. Running Time (per case)
Base (MoAI-7B)
7.8s 25.6s (+3.3xx
R2-T2 (MoAl-7B)
While R2-T2 increases latency by 3.3x, the substantial accuracy gains justify the trade-off, and optimizations such as reference set pruning and efficient kNN search can further reduce overhead.
D.5 Task Classification Automation
To improve usability, we explore automated task type selection using lightweight classifier.
Results show that automated task classification achieves comparable performance (80.4% vs 80.7%) while reducing computational overhead, demonstrating the viability of this approach for practical deployment.
D.6 Data Contamination Analysis
We conducted rigorous analysis to ensure no data contamination between reference sets and evaluation benchmarks using two-step screening process:
1.
Question Similarity Check: Computed cosine similarity between evaluation and reference questions using
NV TV_Embed_V> V2. Samples with similarity >0.95 were flagged.
2. Image Similarity Check: For flagged cases, applied CLIP measure image similarity. Only samples where both similarities exceeded 0.95 were classified potential overlaps.
Through this analysis, we found no overlapping samples between reference sets and evaluation benchmarks, confirming that performance gains stem from method effectiveness rather than data leakage.
17

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-xxpee Expeeesss
Table 17. Performance with automated task classification manual selection.
Method
Average Score
R2-T2 (Manual task selection)
80.7% 80.4% (-0.3%)
R2-T2 (Automated classification)
D.7 Ablation Study
We perform ablation study assess the impact of key hyperparameters R2-T2's performance. Table 18 evaluates different learning rate schedules for Gradient Descent, comparing cosine annealing, step decay, and fixed schedules. Full results for the ablation studies discussed in Section 4.3 are presented in Tables 19-22.
Comparison of different learning rate In Table 18, we investigate how different learning rate schedules affect the performance of Gradient Descent. We compare cosine annealing schedule against two fixed (1e-3 and le-4) and step decay schedule. The cosine annealing schedule consistently outperforms all baseline approaches across all benchmarks, achieving improvements of up to 12.7 percentage points over the fixed learning rate (le-3) baseline. These findings suggest that carefully designed learning rate schedules are essential for maximizing the potential R2-T2.
Table 18. Ablation study of R2-T2 (6NN.NGD) with different learning rate schedules.
Schedule
MMBench
MME-P
SQA-IMG
A12D
TextVQa
GQA
CVBench 20
CVBench 30
Fixed (1c-3)
71.8
1671.2
74.3
70.9
60.4
63.1
66.8
57.2
Fixed (le-4)
75.2
1692.5
77.8
74.5
63.9
66.5
69.9
63.3
Step Decay
82.9
1745.4
84.2
81.8
70.5
73.8
73.5
67.2
Cosine
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
Table 19. Ablation study of R2-T2 (&NN, NGD) with different choices of neighborhood on MoAL.
Neighbors
Parameter
MMBench
MME-P
SQA-IMG
A12D
TextVQa
GQA
CVBench
CVBench
0.2
82.4
1733.9
84.8
81.3
69.9
73.1
67.1
66.5
0.4
83.9
1758.4
86.0
83.0
71.5
74.8
68.5
67.3
e-ball
1778.8
87.2
69.6
0.8
83.7
1756.5
85.9
82.5
71.2
74.5
68.3
67.4
&
83.2
1740.9
86.1
83.1
71.3
75.1
75.8
67.4
k 5
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
ANN
& 10
84.0
1761.3
86.8
83.5
72.8
75.3
76.6
68.1
& 20
80.7
1693.6
83.6
80.7
70.5
73.2
73.9
65.7
18

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-ENeee Exppetss
Table 20. Ablation study of R2-T2 (KNN, NGD) with different choices of kernels on MoAL.
Kernel
MMBench
MME-P
SQA-IMG
A12D
TextVQA
GQA
CVBench 2D
CVBench
Linear
82.1
1722.3
84.2
80.8
72.8
72.7
62.1
Polynomial
83.2
1745.5
85.1
81.9
70.4
73.9
74.5
65.2
Matern
83.9
1752.8
85.8
76.3
67.8
Gaussian
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
Table 21. Ablation study of R2-T2 (ANN, NGD) with different embedding models on MoAL.
Embedding Model
MMBench
MME-P
SQA-IMG
AI2D
TextVQA
GQA
CVBench
CVBench
Sentence-Ben
82.8
1748.2
84.2
80.3
70.2
73.8
75.6
66.0
Stella-En-I.5BB.
83.6
1752.5
85.4
82.1
70.8
74.3
76.3
67.5
Gte-Qwert Qwenn-7B--nnttee
84.0
1757.0
86.0
82.7
71.3
74.8
76.1
67.0
NV-Embed-V2 85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
Table 22. Ablation study of R2-T2 (INNN NGD)
with different number of NGD steps.
#Step
MMBench
MME-P
SQA-IMG
AI2D
TextVQa
GQA
CVBench 3D
CVBench
81.3
1705.8
72.2
83,8
1745.2
86.5
83.2
71.8
75.2
76.0
67.6
10 (ours)
85.2
1785.5
88.3
85.0
73.5
77.0
77.9
69.2
20
85.0
1777.8
88.5
84.6
73.7
76.8
77.7
69.0
85.3
1792.0
88.2
84.8
77.1
77.6
69.3
19

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expers
D.8
Case study
Case Study: Transition from LANG to LAUX Figures and 8 illustrate cases where the initial routing incorrectly prioritizes LANG which aligns visual features with language but lacks object-specific recognition capabilities. This results in misidentifications:: in the first case, the model misinterprets the plane number, yielding "728FW" instead of the correct "728TFW"; in the second case, it incorrectly predicts "FRENCH" as the license plate's sate's state instead of the correct "California."
To correct these errors, R2-T2 retrieves three highly relevant reference samples using &NN based on question similarity, Each reference set contains samples with similar question structures, providing more suitable routing adjustment. After incorporating insights from these references, the routing shifts towards LAUX, which enhances object-specific recognition and scene understanding. This re-routing process enables the model to produce the correct answers "728TFW" and "California," demonstrating the effectiveness of R2-T2 in dynamically refining expert selection.
Transition from laNg to TAUX We show one case for this transition in Figure 2 and analyze in Section 4.4. Figure 6 illustrates another case where the initial routing incorrectly prioritizes LANG, which aligns visual features with language but lacks object-specific recognition capabilities. As a result, the model miscounts the number of hats in the image, selecting answer "(C) 2" instead of the correct "(D) 1."
Case Study:
To correct this, R2-T2 retrieves three highly relevant reference samples using kNN based on question similarity. These samples contain similar counting-related queries, allowing for a more effective routing adjustment. After integrating insights from these references, the routing shifts towards LAUX, which specializes in fine-grained object recognition. This re-routing enables the model to correctly identify and count the hats, selecting the correct answer "(D) 1." This case demonstrates the ability of R2-T2 to refine expert selection dynamically, improving numerical reasoning in visual question-answering tasks.
Case Study: Transition from LANG to LIMG Figures 9 and 10 illustrate cases where the initial routing incorrectly prioritizes LANG, which aligns visual features with language but lacks fine- aie-ggained perceptual understanding. This misalignment leads to incorrect predictions: in the first case, the model incorrectly identifies "DVD Player" instead of the correct answer "Speaker" when asked which device is not illuminated; the second case, incorrectly answers "No" instead of "Yes" when asked if the shirt is soft and white.
To correct these errors, R2-T2 retrieves three relevant reference samples using KNN based on question similarity. These samples involve similar queries related to illumination and color perception, guiding more suitable routing adjustment. After incorporating insights from these references, the routing shifts towards LIMG, which specializes in fine-grained visual details. This adjustment enables the model to correctly identify the non-illuminated device and recognize the shirt's color and texture, leading to the correct answers "Speaker" and "Yes."
These cases demonstrate R2-T2' ability dynamically refine expert selection, improving visual perception in multimodal reasoning tasks by leveraging contextual cues from reference samples.
20

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expeee
Before re-routing
After re-routing
routing weights
LANG
routing I
LAUX
LIMG
weights
LAUX LIMG
LAUX
LSKLF
SELF
R2-T2
LgELP
Answer: (C)
X
experts
Answer: (D)
KNN
Question Similarity 0.9826
Question Similarity 0.9806
Question Similarity: 0.9806
Question:
many (A) (B) 0 2 (D)
image?
Question: How many hats the photo?
Question: How many hats are in the picture?
Answer: (D)
Answer:
Question: How many hats are in the photo?
Answer:
Answer:
Figure 6. Example for tramsition from LLNG to LAx using R2-T2. The model initially gives incorrect answer "(Cy2" by relying on LLANu- After &NN retrieval with similar questions about counting hats, re-routes to Laux and correctly answers "(D) 1" for the number of hats in the image.
Before re-routing
After re-routing
routing weights
Lang
routing weights
LXUX
AUX
LAuR
LIMG
LEELF
LAUX
LLANG
LevG
TSELF
R2-T2
TSELE
Answer: 728FW
experts
728TFW
experts
Question Similarity :0.9491
Question Similarity
Question Similarity: 0.8854
Question:
What is the plane number?
Question: What number the plane? Answer: N288SA
Question: planes number? Answer: 423
Question: What is the number of this plane? Answer:
Correct Answer: 728TFW
Figure 7. Example of routing transition from LLAGG LAUX using R2-T2. Initially, the model selects LANG misidentifying the plane number. By retrieving ANN with similar queries, R2-T2 shifts the routing weights towards LAUX. leading to the correct answer.
21

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expeee
Before re-routing
After re-routing
routing weights
routing weights
LaNG
LAUX
AUX
LAUX Live
LANG
TAUX
LIMC
LgELF
R2-T2
SSELE
LySLF
ISELP
Answer: FRENCH X
experts
Answer: California
experts
KNN
Question Similarity :0.8750
Question Similarity 0,8750
Question Similarity: 0.7751
Galliooinia FRENCH
6CT4983
Question:
Question:
Question:
Question: this plate number? Answer: 3XHE872
What state is this plate from?
Where is the license plate from?
Where is the license plate from?
Correct Answer: California
Answer: California
Answer: ORLANDO
Figure 8. Example for transition from LANa to Lax using R2-T2. The model initially gives incorrect answer "FRENCH" by relying on LANG- After &NN retrieval with similar questions, re-routes LAUX and correctly identifies "California" the plate's state.
Before re-routing
After re-routing
routing weights
LLAN
LIMG
routing weights
TAUX
LANG
LIMG
LAUX
SELF
LEELF
LAIX
LEELF
R2-T2
Answer: Player
experts
Speaker
experts
KNN Question Similarity 0.7585
Question Similarity 0.7890
Question Similarity: 0.6662
Question: What light is illuminated? Answer: Landing light
Question:
Question: What is illuminated?
Which kind of device isn't Illuminated
Answer: pool
Question: What is illuminated in the background? Answer: A Wall
Answer:Speaker
Figure 9. Example of routing transition from llang to LisG using R2-T2. Initially, the model selects LlANG, leading to the incorrect prediction "DVD Player" when asked which device illuminated. By retrieving &NN samples with similar Ullumination-rllan queries, R2-T2 T2 shifts the routing weights towards LIMG. enabling the correct answer "Speaker."
22

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of Expertts
Before re-routing
After re-routing
routing weights
LANG
routing weights
Lpsa
LEML
LAUx LAUXLLANG
AUX
SE
LSELP
R2-T2
Answer: No
Answer: Yes
experts
ANN Question Similarity 0.6581
Question Similarity 0.6621
Question Similarity: 0.6581
SS
Question: What
Question: What the color of shirt?
Question:
shirt?
Question: What the color of shirt?
Is the shirt soft and white?
Answer: Black
Answer:
Answer:Yes
Answer: Peach
Figure 10. Example of routing transition from lLANG to LIMG using R2-T2. Initially, the model selects llang, Icading to the incorrect prediction "No" when asked if the shirt soft and white. By retrieving &NN samples with similar color-hased queries, R2-T2 shifts the routing weights towards LIMG, allowing the model to correctly answer "Yes."
23

---

R2-T2: Re-Routing in Test-Time for Multimodal Sixture-of-Experts
E
Expert Transition Analysis
To better understand the impact test-time re-routing, we analyze expert transitions across different prediction scenarios. Figures 11-14 illustrate how top-1 expert selections shift before and after re-routing on CVBench2D/ap
Incorrect Predictions
LANG
Laux LAux
LANG
LIMG
LAUX
LAUX
Correct Predictions
Figure Top-1 expert transitions incorrect correct predictions on CVBench an/3 after re-routing, For transitions to incorrect
20/3D
predictions, the main patterns include transitions from LaNg to JIMG LAUX and TAUX
Correct Predictions
LANG
LMMG
Laux
LANG
LAUX
LIMG
Incorrect Predictions
Figure 12. Top-1 I expert transitions from correct to incorrect predictions on CVBench after re- ro-ronting. The visualization shows
primary transitions from LaNG Laux and LIMG, demonstrating how correct predictions can shift to incorrect outcomes through these pathways.
Correct Predictions
LANNG
LAUX
LIMG ISELFI
LANG
LAUX
LIMG
LAUX
Correct Predictions
Figure 13.. Top-1 expert transitions from correct correct predictions CVBench after re-routing. The main transition patterns demonstrate consistent routing from LLANG through Laux to LImG and Luxx showing stable pathways for maintaining correct predictions.
24

---

R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of Expertss
Incorrect Predictions
LANG
I
Laux
LIMGJAUX
LANG
Incorrect Predictions
LAUX
LIMG
LAUX
Figure 14. Top-1 I expert transitions from incorrect incorrect predictions CVBench after re-routing. The visualization reveals persistent incorrect prediction patterns, with transitions primarily flowing from LLAG through LAUX to LIMG and LAUX, with additional ISLLF routing observed.
25


Reviews:
[{"review_id": "BmL8n44Q1T", "reviewer": "Reviewer_HT7U", "summary": "The authors introduce test-time re-routing (R2-T2) for vision-language MoEs. R2-T2 adapts the routing weights for each test sample based on similar samples in a reference set. In particular, three various strategies with different optimization objectives and neighbor-search spaces have been proposed, with \"Neighborhood Gradient Descent\" (NGD) being the most performant approach. R2-T2 is evaluated on a variety of vision language benchmarks.\n\n## update after rebuttal:\nThe authors have addressed most of my concerns related to comparison to other baselines and have clarified the computational cost requirements and hence I raise my rating to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer HT7U\n\nThank you for your detailed feedback! We address your comments below.\n\n> **Q1: Significant increase in computational and memory costs.** \n\nR2-T2 does not require significant increase in computations and memory because it only optimizes a low-dimensional routing weight vector (e.g., 6-dim for MoAI). Except NGD, the other two R2-T2 strategies are gradient-free and do not require backpropagation. Hence, compared to PEFT (LoRA, prompt, prefix tuning), re-training the MoE model or routers, R2-T2 is much more efficient. Moreover, R2-T2 significantly improves model accuracy, especially on challenging downstream tasks, making the modest additional cost worthwhile. \n\nWe can further reduce R2-T2's cost by: \n- The embedding vectors can be pre-computed and cached (less than 1GB), which substantially reduces the runtime overhead during deployment. \n- We use fast similiarty search tools such as faiss to search the neighbors and compute similarity efficiently. \n- Our approach is flexible, allowing users to adjust the kNN neighborhood size and the number of iterative steps for a better trade-off between accuracy gains and computational cost effectively.\n\n> **Q2: The method has only been adapted for a single-level routing decision, lacking the complexity of SOTA MoE models with independent routing across multiple layers.** \n\nOur focus is on visual-centric tasks, where the primary bottleneck of most VLMs is the limited capability of a single visual encoder. In MoE VLMs such as MoAI and MoVA, MoE is applied only at the visual encoder, by replacing it with <10 experts, making a single-level routing decision sufficient for effective feature extraction. Extending to multi-layer routing—common in LLM MoE—would introduce much more computational overhead without addressing the core challenge of constrained visual expert capacity.\n\n> **Q3: The framework assumes access to a large-scale reference set, making comparisons in Table 3 unfair.** \n\nBecause MoAI and MoVA do not support interleaved input and ICL/RAG, we use Qwen-VL as the base model. \n1.For RAG, we retrieve similar reference samples based on embedding similarity and use them as few-shot demonstrations. \n2.For ICL, we randomly choose the same task demonstrations with correct answer in reference set as few-shot demonstrations.\n\n|                | RAG             | ICL |\n|----------------|-----------------|---------------------|\n| 0-shot (base)  | 61.7%           | 61.7%               |\n| 1-shot         | 63.1% (+1.4%)    | 62.4% (+0.7%)        |\n| 2-shots        | 63.6% (+1.9%)    | 62.7% (+1.0%)        |\n| 3-shots        | 63.9% (+2.2%)    | 62.8% (+1.1%)        |\n| 5-shots        | 64.1% (+2.4%)    | 62.9% (+1.2%)        |\n\nRAG improves only modestly (61.7% → 64.1%) and ICL to 62.9%, while R2-T2 boosts MoAI from 74.5% to 80.7%, showing it leverages the reference set more effectively.\n\n\n> **Q4: Please evaluate additional baselines, such as ensemble voting, noisy router sampling, few-shot prompting, or a simple RAG approach.**\n\nWe evaluated these baselines:\n\n- Ensemble Voting – Enabled dropout during inference and performed 10 forward passes per sample, aggregating predictions via majority voting.\n- Noisy Routing Ensemble – Added Gaussian noise to the router’s output, ran 10 forward passes with different noise realizations, and applied majority voting.\n\nFew-Shot Prompting – Please see **Q3**\n\nRAG-style Retrieval – Please See **Q3**\n\n| | Average |\n|---|---|\n| Base (MoAI) | 74.5% |\n| Multiple Sampling | 74.9% (+0.4%) |\n| Noisy Routing Ensemble | 75.4% (+0.9%) |\n| R2-T2 (MoAI) | 80.7% (+6.2%) |\n\nAs shown, ensemble-based methods provide only marginal gains (≤0.9%), whereas R2-T2 delivers a substantial improvement of +6.2%. These results validate the effectiveness of R2-T2 beyond what additional compute alone can achieve.\n\n> **Q5: The user must specify the task type to select the appropriate reference set. Can this process be automated using a classifier?** \n\nAutomating task type selection could improve usability and efficiency, so we conducted experiments to explore this approach. We pre-annotated our reference set with three task types (visual understanding, knowledge reasoning, OCR) and extracted 4096-dimensional task embeddings using NV_Embed_V2. Then, we trained a lightweight logistic regression classifier using embedding of reference set, which only introduces $10^4$ – $10^5$ FLOPs per inference. Given a test sample, the model predicts its task type and selects the corresponding reference subset for test-time adaptation.\n\nOur results show that this automated selection reduces computational overhead while maintaining accuracy, with only a marginal drop from 80.7% to 80.4%. These findings demonstrate that task classification is a viable strategy for improving efficiency with minimal impact on performance.", "claims_and_evidence": "The authors backup the claims about their methods by implementing R2-T2 on two VLM MoE models: MoAI and MoVA and report results on several standard benchmark.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no theoretical claims or proofs in this paper.", "experimental_designs_or_analyses": "Please refer to the weaknesses section on fairness of comparisons.", "relation_to_broader_scientific_literature": "The paper claims a test-time re-routing procedure for MoEs, and builds the method on top of two models, MoAI and MoVA. These MoEs, differ from conventional MoE architectures for transformers and operate at the larger module/model selection level. The routing in these models are limited to a single routing decision and does not have the complexities for dynamic routing in standard MoE architectures with independent routing decisions per layer leading to $\\binom{N}{K}^{L}$ pathways inside the architecture.", "essential_references_not_discussed": "The paper cites related works in the MoE literature, such as GShard and Switch transformers. However, it does not discuss how the method can be extended beyond the single layer routing decisions in the models considered.", "other_strengths_and_weaknesses": "The idea of using re-routing in test-time for MoEs is interesting and shows the potential of improving MoE performance solely based on adjusting the routing weights. However, there are a few weaknesses which limit the applicability of the method in practice and the fairness of the evaluations:\n\n- Significant increase in computational and memory costs: One of the major motivations for using MoEs is their sparse selection mechanism which reduces the amount of activated parameters per sample. The proposed approach significantly increases the FLOPs (~7X) defeating the original purpose of using MoEs. The method also requires relying on external embedding models and retrieving KNN samples from thousands of reference points with correct predictions, followed by iterative updating of routing weights based on neighbor gradients.\n- The method has only been adapted for a single level routing decision which lacks the complexities of current SotA MoE models leveraging independent routing decisions throughout the network enabling numerous pathways in the architectures compared to the very limited number of pathways possible in MoAI and MoVA architectures. Extending the solution to work with standard MoE architectures will have higher impact on the field. The authors need to comment on the added memory and compute costs for scaling to such MoEs.\n- Evaluation with access to reference samples from the same domain: The framework assumes access to a very large scale reference set with a massive number of 5000 samples per set. The compiled reference set for general visual understanding alone is 20k samples. R2-T2 retrieves nearest neighbors from this large reference set with similar queries. Apart from the requirement to store the embedding of these samples and the expensive knn operation, the method assumes the user should manually select the type of task they are solving and hence perform the knn only in the selected subset. Additionally access to these relevant information at test time, makes it unfair to compare against all other methods in Table 3, which do not have access to such information. For example, it is known that access to few shot examples significantly boosts the performance of models compared to zero-shot evaluation. While all the methods in Table 3, effectively perform 0-shot evaluation, the current method assumes access to thousands of samples from the same task.\n- While the method could be framed as an Inference-Time compute approach,  in which we allow for extra compute during inference (thinking harder) to increase accuracy, it can only be validated as such if the compared methods could also gain from additional compute. Some simple baselines include, 1) using multiple sampling from the base network and performing majority voting, 2) Performing multiple noisy sampling from the router followed by majority voting of independent predictions, 3) providing the compared models in Table 3 access to example correct responses as few shot samples, 4) Implementing a simple RAG approach to collect the most relevant samples to the correct prompt, followed by few-shot responses, etc.", "other_comments_or_suggestions": "No", "questions_for_authors": "At present, the user needs to specify the nature of the task they want to solve to use the relevant reference set embedding for test-time adaptation. Is it possible to automatically determine the nature of the task using a classifier?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8W5vfZ9K19", "reviewer": "Reviewer_bdGh", "summary": "The paper introduces R2-T2, a test-time re-routing method for multimodal Mixture-of-Experts (MoE) models. The core idea is to optimize routing weights during inference by leveraging reference samples with correct predictions, addressing suboptimal routing in pretrained MoE models. The method is training-free and computationally efficient. Experiments on MoAI and MoVA models across eight benchmarks demonstrate significant performance gains over base models, even surpassing larger VLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer bdGh\n\nThank you for your detailed feedback! We address your comments below.\n\n> **Q1: Possible data contamination—subsampled reference datasets (e.g., VQA-V2, MathVista) may overlap with evaluation benchmarks (e.g., MMBench, TextVQA), potentially inflating results. How was this addressed?** \n\nWe appreciate the reviewer’s concern and have conducted a rigorous analysis to ensure no data contamination. Our process follows a two-step screening approach:\n\n1. Question Similarity Check – We first computed cosine similarity between evaluation benchmark questions and reference set questions using the NV_Embed_V2 embedding model. Samples with a similarity score >0.95 were flagged for further inspection.\n2. Image Similarity Check – For flagged cases, we applied CLIP to measure image similarity. Only samples where both question similarity and image similarity exceeded 0.95 were classified as potential overlaps.\n\nThrough this analysis, we found no overlapping samples between the reference set and evaluation benchmarks. These results confirm that our performance gains are not due to data leakage but stem from the effectiveness of our proposed method.\n\n> **Q2: Impact of reference set choice—how does the method perform when reference samples have limited coverage for certain task types (e.g., rare spatial reasoning)? What is the effect of reference set size (e.g., 1K vs. 10K samples)?** \n\nWe appreciate this insightful question and have conducted further experiments to analyze these factors.\n\n1. Limited Task Coverage: We evaluated R2-T2 on 3DSRBench, a dataset focused on rare spatial reasoning cases. Despite lower reference coverage for these tasks, R2-T2 still delivers a 4.5% improvement over the base model, demonstrating its robustness.\n\n| | 3DSRBench Accuracy |\n|---|---|\n| Base (MoAI) | 45.2 |\n| R2-T2 (MoAI) | 49.7 (+4.5%) |\n\n2. Reference Set Size: We varied the reference set size and observed that even when reduced to 1/10th of its original size (e.g., 1K instead of 10K samples), R2-T2 still provides a notable boost (+2.9%). However, when reference samples are randomly selected (without ensuring task relevance), the improvement is minimal (+0.3%), underscoring the importance of well-curated reference sets.\n\n| | Average |\n|---|---|\n| Base (MoAI) | 74.5% |\n| 1/10 Reference set size | 77.4% |\n| Random choose | 74.7% |\n| R2-T2 (MoAI) | 80.7% |\n\nThese results confirm that while reference set quality and coverage affect performance, R2-T2 remains effective even with smaller but carefully selected reference sets.\n\n> **Q3:  More evaluation results are needed on important benchmarks, such as MMMU and ChartQA.** \n\nThank you for the suggestion. We have evaluated R2-T2 on MMMU and ChartQA, with results summarized below:\n\n| | MMMU | ChartQA |\n|---|---|---|\n| Base (MoAI) | 55.7% | 67.4% |\n| R2-T2 (MoAI) | 61.3% (+5.6) | 71.6% (+4.2) |\n\nThese results demonstrate that R2-T2 consistently improves performance across diverse multimodal tasks, reinforcing its robustness and generalizability.\n\n> **Q4: While the improvements presented are compelling, the significant increase in FLOPs shown in Table 4 raises concerns. Additionally, a comparison of latency is needed.** \n\nPlease refer to **Reviewer HT7U Q1** regarding computational cost. To directly address latency, we conducted experiments on an RTX A6000:\n\n| | Avg. Running Time (per case) |\n|---|---|\n| Base (MoAI) | 7.8s |\n| R2-T2 (MoAI) | 25.6s (+3.3×) |\n\nWhile R2-T2 increases latency by 3.3×, we believe the substantial accuracy gains justify the trade-off. Moreover, optimizations such as reference set pruning and efficient kNN search can further reduce overhead.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The method makes sense for the problem or application at hand.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "Yes, all is reviewed", "relation_to_broader_scientific_literature": "The work connects to MoE routing, test-time optimization, and multimodal LLMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The motivation is reasonable. This paper reveals suboptimal routing during inference due to the fixed, pretrained router and grounds this in empirical evidence. \n\n2. The proposed method is training-free and computationally efficient, requiring no model parameter updates and avoiding the costs of retraining or fine-tuning. The three adopted strategies are lightweight:\n\n3. Performance improvements across diverse tasks on strong baselines, including MoAI and MoVA, are significant. \n\n4. The paper is well-written and easy to understand.\n\nWeaknesses:\n\n1. Potential data contamination. The paper uses subsampled reference datasets (e.g., 5,000 samples from VQA-V2 and MathVista) but does not clarify whether these overlap with the evaluation benchmarks (e.g., MMBench, TextVQA). For instance, TextVQA is one of the source datasets of MathVista. If test samples from TextVQA are included in the reference set of MathVista, performance gains could be artificially inflated. A discussion on how contamination was avoided is critical for validity.\n\n2. Choice of reference set. Table 1 summarizes the adopted reference and evaluation benchmarks. Does the method degrade when reference samples lack coverage of certain task types (e.g., rare spatial reasoning cases)? How does model performance change when using different reference sets and reference set sizes (e.g., 1K vs. 10K samples)?\n\n3. More evaluation results are needed on important benchmarks, such as MMMU and ChartQA.\n\n4. While the improvements presented are compelling, the significant increase in FLOPs shown in Table 4 raises concerns. Additionally, a comparison of latency is needed.", "other_comments_or_suggestions": "Please see the Weakness section.", "questions_for_authors": "Please see the Weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I2KSokuQak", "reviewer": "Reviewer_1y8A", "summary": "This paper proposes R2-T2, a test-time re-routing method designed to enhance multimodal mixture-of-experts (MoE) models without retraining. It addresses the limitation of suboptimal routing weights produced by pretrained routers, which often fail on complex or out-of-distribution tasks. R2-T2 introduces three strategies: Neighborhood Gradient Descent, Kernel Regression, and Mode Finding, that dynamically update routing weights for each input by referencing “successful” samples, thereby improving expert selection. Extensive experiments on eight diverse benchmarks demonstrate that R2-T2 significantly outperforms baseline MoE models, even approaching an oracle routing upper bound. Notably, it boosts smaller models to rival larger-scale vision-language models, highlighting its cost-effectiveness and scalability. The findings suggest that a well-tuned, training-free test-time adjustment can maximize MoE potential in multimodal reasoning tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 1y8A\n\nThank you for your detailed feedback! We address your comments below.\n\n> **Q1: Generalization to OOD samples—The method relies on a reference set with similar questions. How does it perform on truly out-of-distribution (OOD) cases?** \n\nThank you for your question. Our approach is designed as a zero-shot reference method. We deliberately do not use any benchmark data for reference, meaning that our test questions is out-of-distribution (OOD) relative to the reference set. And there is no overlap between Benchmarks and Reference Set (Please see **Reviewer bdGh Q1**) \n\nFor the questions are not similar with reference samples: Our Mode Finding (section 3.3 in paper) strategy does not strictly rely on identical questions but rather on the proximity of routing weights in the expert space. Even for literally different questions, if their routing weights are close to those of reference samples, it indicates that similar experts are needed. Therefore, as long as the underlying expert requirements are similar, our method generalizes well even to not similar cases.\n\n>**Q2: Computational overhead and scalability—Larger reference sets may introduce significant computational costs. Have the authors analyzed this in detail?** \n\nThank you for your question. Please refer to **Reviewer 8cuF Q5** for your concern about computational overhead and scalability. \n\n\n>**Q3: Potential mismatches—If a reference question closely overlaps with a test question but belongs to a different type, how does the method avoid incorrect matches?**\n\nThank you for this insightful question. In theory, two questions might have similar surface forms even if they are of different types. \n\nHowever, in practice, VLM questions tend to be short and less complex compared to those in LLMs, and our tasks come with sufficiently detailed descriptions that help disambiguate their semantic intent. This means that even if a few words are similar, the overall context captured in the embedding still reflects the true task type. \n\nFurthermore, in scenarios where subtle differences might lead to confusion, our method can be augmented with a chain-of-thought (CoT) prompt. In this variant, we prompt the model to outline a few high-level reasoning steps, and then we use the resulting CoT output as the embedding for neighbor search. This additional step helps ensure that the underlying reasoning process—and not just the superficial wording—is captured, further reducing the chance of mismatches.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There is no theoretical claims for the paper.", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n* The proposed method adjusts routing weights dynamically during inference, enabling performance gains without retraining the model.\n* Consistent gains bring smaller models’ performance close to or even surpassing that of LVLMs.\n* R2-T2 offers three optimization techniques, giving practitioners multiple avenues to customize the method for specific tasks.\n\nWeaknesses:\n*Reliance on the Reference Set: The method heavily relies on the reference set at test time, raising concerns about generalization and practical utility. Moreover, insufficient experiments have been conducted to explore these issues:\n\n1. Model generalization to OOD samples: The method assumes that the reference set contains questions similar to the test question. However, for LLMs and LMMs, one of the toughest scenarios is out-of-distribution (OOD) questions. While the method could perform well when similar questions are in the reference set, it may fail if they are not.\n2. Computational overhead and scalability: Because the method needs a sufficient number of samples in the reference set to find close matches, a larger set could introduce significant computational costs. The authors should investigate this more thoroughly.\n\n* If a question in the reference set closely overlaps with the test question but is of a different type, changing only a few words might not greatly reduce the measured similarity. It remains unclear how the method avoids such mismatches.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See Weaknesses Part", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0xQ4NE9M1", "reviewer": "Reviewer_8cuF", "summary": "The paper introduces R2-T2, a method designed to optimize routing weights in multimodal Mixture-of-Experts (MoE) models during test time. R2-T2 maintains a reference set comprising samples for which the MoE model's outputs are either correct or preferred for each task. When presented with a sample from a new task, R2-T2 first identifies its neighborhood within the reference set by leveraging embeddings generated from a separate embedding model. The neighboring samples are then utilized to predict the routing weights, employing one of three techniques: gradient descent, kernel regression, or mode finding. Evaluated across multiple benchmarks, R2-T2 demonstrates superior performance compared to its MoE backbone.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 8cuF\n\nThank you for your detailed feedback! We address your comments below.\n\n>**Q1: Concern about reference set construction: Academic benchmarks use predefined sample types, but real-world scenarios may not allow task type selection before inference. It might become necessary to store multiple types of reference sets simultaneously, increasing storage and computational demands.**\n\n\n1. Storage Cost: We appreciate this concern. However, our reference sets store only lightweight metadata—question text and routing weights—totaling just 3.24 MB in Parquet format, while images are loaded dynamically from Huggingface. \n2. Memory Cost:As shown in Reviewer **HT7U Q1**, the additional memory and computational overhead remain minimal. \n3. Scalability: Using a lightweight classifier to select a relevant subset of the reference set for each test sample, ensuring that optimization is performed on a much smaller and targeted set. (Please see **Reviewer HT7U Q5**)\n\n\n\n>**Q2: A filtering mechanism could be required to identify the appropriate reference set before applying R2-T2, adding complexity to the framework.**\n\nThank you for your question. A similar point was addressed in **Reviewer HT7U Q5** regarding training a classifier. Please refer to that response for details.\n\n>**Q3: Additionally, the effectiveness of R2-T2 heavily depends on the choice of neighborhood within the reference set, with only similar samples contributing meaningfully to the optimization process. What happens if no sufficiently similar samples exist in the reference set? What if an incorrect reference set is chosen (e.g., using OCR data for knowledge-based VQA)? Would R2-T2 still maintain its strong performance under these conditions?**\n\nThank you for raising this important point. Please refer to **Reviewer bdGh Q2** for no sufficiently reference samples. As for mismatched reference sets, we conducted experiments using an OCR-based subset (ST-VQA & DocVQA) as the reference for R2-T2 on SQA-IMG (knowledge-based VQA), yielding the following results: \n\n|                                       | knowledge-based VQA |\n| ------------------------------------- | ------------------- |\n| Base (MoAI)                           | 83.5                |\n| R2-T2 (MoAI)                          | 88.3                |\n| R2-T2 (Using OCR subset as reference) | 83.8                |\n\nThese results highlight that R2-T2 provides significant gains (from 83.5 to 88.3) with a task-relevant reference set but offers minimal improvement (83.8) when the reference is mismatched, emphasizing the critical role of reference set selection.\n\n>**Q4: Efficiency considerations: The framework introduces additional storage costs, particularly due to the embedding model's memory overhead (e.g., 7B parameters).**\n\nThank you for raising this important point regarding storage and memory overhead. In addition to FLOPs, we now provide GPU memory usage comparisons (see table below).\n\n| | GPU Memory Usage | Average Accuracy |\n|---|---|---|\n| Base (MoAI) | 18GB | 74.5% |\n| R2-T2 (MoAI) with nv_embed_v2 | 27GB | 80.7% |\n| R2-T2 (MoAI) with all_mini_v6 | 20GB | 77.5% |\n| R2-T2 (MoAI) with Stella-En-1.5B-V5 | 22GB | 78.5% |\n| R2-T2 (MoAI) with Gte-Qwen2-7B | 31GB | 78.7% |\n\nOur R2-T2 framework with the nv_embed_v2 model requires 27GB of GPU memory and achieves 80.7% accuracy, while the base MoAI model uses 18GB at 74.5% accuracy. Importantly, smaller embedding models like all_mini_v6 (20GB total, 77.5%) or Stella-En-1.5B-V5 (22GB total, 78.5%) still provide significant accuracy gains with lower memory overhead. This demonstrates a scalable trade-off: users can select larger embeddings for maximum performance or opt for smaller models to balance efficiency and accuracy.\n\n>**Q5: Scalability—The reference sets could grow significantly, especially when handling diverse or unpredictable tasks.**\n\nThank you for your insightful question. \nWe address scalability by \n1. Using a lightweight classifier to select a relevant subset of the reference set for each test sample, ensuring that optimization is performed on a much smaller and targeted set. (Please see **Reviewer HT7U Q5**)\n2. We employ parquet compression techniques and caching strategies to reduce storage and memory consumption effectively. (Please see **Q1**)\n3. By leveraging efficient similarity search frameworks like FAISS, we significantly improve the computation speed of neighbor retrieval even when the overall reference set is large. (Please see **Reviewer HT7U Q1**)\n\n>**Q6: Similarities to RAG & in-context learning—The method shares similarities with RAG, and could in-context learning achieve comparable performance?**\n\nThank you for your question. Please refer to **Reviewer HT7U Q3**, where we discuss both RAG-related considerations and the role of in-context learning in comparison to our approach.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design appears to fine, more questions in the weaknesses/questions section later.", "relation_to_broader_scientific_literature": "It should be of significant interest to the broader scientific community, as the paper addresses the test-time operations of large MoE models—a topic that warrants greater attention in today's research landscape.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n* The paper is well-written, clear, and easy to follow.\n* It addresses the optimization of test-time routing weights in MoE models, a timely and important research direction with significant practical implications.\n* The performance improvements achieved by R2-T2 over its MoE backbone are impressive, demonstrating the potential of this framework for real-world applications.\n\nWeaknesses:\n\n* Concerns about the construction and use of the reference set:\n  * The evaluation primarily focuses on academic benchmarks, where reference sets are constructed using samples of similar types. However, in more complex real-world scenarios, it may not always be feasible to predefine the most suitable task type before inference begins. In such cases:\n      * It might become necessary to store multiple types of reference sets simultaneously, increasing storage and computational demands.\n      * A filtering mechanism could be required to identify the appropriate reference set before applying R2-T2, adding complexity to the framework.\n      * These factors raise questions about the practical feasibility of the method in real-world settings where task diversity and unpredictability are common.\n  * Additionally, the effectiveness of R2-T2 heavily depends on the choice of neighborhood within the reference set, with only similar samples contributing meaningfully to the optimization process. This raises concerns:\n      * What happens if no sufficiently similar samples exist in the reference set?\n      * What if an incorrect reference set is chosen (e.g., using OCR data for knowledge-based VQA)? Would R2-T2 still maintain its strong performance under these conditions?\n\n* Efficiency considerations:\n  * While the authors use FLOPs to measure efficiency, they do not account for the additional storage costs associated with the framework. These include:\n      * The embedding model (7B parameters), which adds significant memory overhead.\n      * The reference sets themselves, which could grow substantially larger than indicated in the paper, especially when accommodating diverse or unpredictable tasks.\n      * Given these factors, the overall efficiency of the method may be less favorable than suggested, potentially limiting its scalability and practicality.", "other_comments_or_suggestions": "NA", "questions_for_authors": "* The method also shares a lot similarities to RAG methods, which are not discussed in the paper. I wonder what's the authors' take on this matter. \n* Additionally, given the selected neighborhood, could in-context learning achieve performance comparable at least to mode finding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduce a novel test-time re-routing (R2-T2) for vision-language MoE models. The proposed approach adapts the routing weights for each test sample based on similar samples in a reference set. Reviewers remains positive in the comments. The authors are engaged in the rebuttal period an addressed most of the concerns from the reviewers. The authors should still address the remaining concerns in the next version.
Decision: Accept (poster)