R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts
Authors: name: Zhongyang Li, name: Ziyue Li, name: Tianyi Zhou

In large multimodal models (LMMs), the perception of non-language modalities
(e.g., visual representations) is usually not on par with the large language
models (LLMs)' powerful reasoning capabilities, deterring LMMs' performance on
challenging downstream tasks. This weakness has been recently mitigated by
replacing the vision encoder with a mixture-of-experts (MoE), which provides
rich, multi-granularity, and diverse representations required by diverse
downstream tasks. The performance of multimodal MoE largely depends on its
router, which reweights and mixes the representations of different experts for
each input. However, we find that the end-to-end trained router does not always
produce the optimal routing weights for every test sample. To bridge the gap,
we propose a novel and efficient method "Re-Routing in Test-Time(R2-T2) that
locally optimizes the vector of routing weights in test-time by moving it
toward those vectors of the correctly predicted samples in a neighborhood of
the test sample. We propose three R2-T2 strategies with different optimization
objectives and neighbor-search spaces. R2-T2 consistently and greatly improves
state-of-the-art LMMs' performance on challenging benchmarks of diverse tasks,
without training any base-model parameters.