Title: Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, LLM, LLM routing, LLM aggregation, LLM ensemble

Abstract: Open-source Large Language Models (LLMs) increasingly specialize by domain (e.g., math, code, general reasoning), motivating systems that leverage complementary strengths across models. Prior multi-LLM approaches either (i) \emph{route} a query to one or a few experts and generate independently, (ii) \emph{aggregate} outputs from each model via costly multi-turn exchanges, or (iii) \emph{fuse} weights into a single model—typically requiring architectural homogeneity. We introduce \emph{Mixture of Thoughts} (MoT), a simple method for \emph{latent-level collaboration} among heterogeneous experts under a global routing scheme. For each query, a lightweight router selects top-$K$ experts and designates a primary expert; uniformly placed \emph{interaction layers} project hidden states into a shared latent space where the primary expert performs cross-attention over its active (selected) peers. Pre-trained experts remain frozen; only the router and the lightweight interaction layers are trained with a novel joint training objective that improves both the expert selection and inter-expert collaboration. Across five in-distribution (ID) and three out-of-distribution (OOD) benchmarks, MoT surpasses the current routing and aggregation-based state-of-the-art, \textsc{Avengers}, by +0.38\% and +2.92\%, respectively. Further, MoT significantly outperforms the best-performing single model. It achieves this with single-pass inference, runtime comparable to routing baselines, and none of the overheads of iterative aggregation. MoT offers a simple latent-space mechanism for combining heterogeneous LLMs, a practical step toward broader multi-LLM collaboration. Our code is publicly available at \url{https://anonymous.4open.science/r/mot-5B4B/}.

TL;DR: Multi-LLM collaboration via latent-space interactions