Title: Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online Learning

Mixture of Experts
Abstract: We explore the use of expert-guided bandit learning, which we refer to as online mixture-of-experts (OMoE). In this setting, given a context, a candidate committee of experts must determine how to aggregate their outputs to achieve optimal results in terms of aggregate accuracy. We propose two algorithms to address this problem. The first algorithm combines aggregate voting with UCB-driven successive elimination, efficiently pruning suboptimal exploration actions. The second algorithm employs an online weighted-majority-voting mechanism, leveraging the respective voting power of each expert proportional to their predictive power. We derive theoretical guarantees for the regret properties in the bandit setting under ideal circumstances, and empirical results are provided accordingly. As a modern study on applications, these methods are applied to the online fine-tuning of a set of expert large language models (LLMs), where after each response, the generative LLM dynamically reweighs its set of experts and/or selects the optimal committee of experts to generate the most accurate response. Our results introduce new methodologies and no-regret guarantees for combining multiple experts to improve on the performance of the an aggregate model overall.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents two settings for decision making with expert sets. The first setting does this by finding the optimal egalitarian expert committee (OEC), i.e., a set of experts whose majority vote outperforms any individual expert. While the experts in the OEC all receive one vote, the second setting proposes finding the optimal weighted voting committee -- where the votes of the experts are weighted by some measure of their competencies. The paper provides optimal solutions in both scenarios when the competencies of the experts are known. It then discusses how these settings can be used for the online Mixture of Experts (MOE) problem, with a new notion of regret in terms of the best set of experts (as opposed to the best individual expert, in standard no regret algorithms for MOE). Two algorithms are proposed corresponding to each setting -- Successive Expert Elimination (SEE) where the OEC is found by eliminating experts with estimated low competencies and $\\theta$-weighted Majority Voting ($\\theta$-WMV) where the voting weights are found using optimistic estimates of competencies. Regret bounds for both the algorithms are derived. Finally, empirical results are presented for online fine-tuning of LLM experts on real datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Clarity: The paper is well-written and easy to understand. The key insights for the SEE and $\\theta$-WMV algorithms are also presented well. My only qualm is with the discussion of the experiments and empirical results, as mentioned in more detail under weaknesses.\n\n2. Well-motivated problem: The problem with the existing notion of regret (w.r.t the best expert) for online Mixture of Experts is well-motivated and the two settings, the egalitarian expert committee and majority voting with a set of experts, are novel in the context of the MOE problem and are discussed in enough detail.\n\n3. The proposed algorithms seem useful in practice for a large enough number of experts (up to 20 in the experiments, Table 1), while providing meaninful regret guarantees (theorems 1-2).\n\n4. Moreover, the proposed algorithms report lower cumulative regret values when compared to baselines in both model-based (with Bernoulli experts) and data-derived settings (with LLM experts) (Table 1).\n\n**Weaknesses**\n\n1. Rushed experiments section: Section 4 (Empirical Study) could use some more details about\n    - The baselines that the proposed algorithms are compared against: These are mentioned in Figure 3 and Table 1 with the results but not discussed at all in the main text (what are they? what the key differences when compared to the proposed algorithms?)\n    - A brief discussion of the results: The results summary paragraph (Line 312 onwards) merely mentions the setup very briefly. The captions for Figure 3 and Table 1 do not discuss the implications of the results either. It would be very helpful to discuss the results in at least a couple of lines to give some more context. It would also be interesting to see what the egalitarian committees found empirically look like in the LLM setup (are the optimal committees found?) and what the weights found for the majority voting approach indicate (which LLMs are weighted higher and why?).\n\n2. Figure 3, with the main comparison of cumulative regret across techniques, is very difficult to parse. The colors are too similar to be differentiated on the graph. I would suggest increasing the sizes of these graphs, using a more diverse color pallette for distinguishing algorithms, and briefly discussing the key takeaway in the caption.\n\nOverall, I am pleased with the clarity of most of the main text (except the experiments section). I also find the expert set notion of regret and the use of top-k bandits with the two proposed algorithms in this setting original. I think that the problem is well-motivated and the application to MOE indicates that this work is indeed significant. This should encourage future work on no regret algorithms with expert sets and MOE. Please see the Questions section below for other questions and suggestions to further improve the paper.\n\nQUESTIONS:\nQuestions:\n1. How should one choose between the two proposed algorithms in practice? Are there scenarios when one should be prefered over the other?\n2. What is the difference between the \"average scoring function\" (Line 70) and the \"scoring-aggregation function\" (Line 81)?\n3. Line 105: $X_i$ be created conditioned on $c_i(x)$, which is the output prediction. Should it instead not be conditioned on the prediction being correct / incorrect?\n4. Should the condition on $\\theta$ in Equation 2.9 be two-sided (as on line 176)?\n5. Algorithm 1, line 5: \"in descending order\" of what? \n\nOther minor comments:\n1. The abstract reads \"We propose three algorithms ...\" (line 4) but I only see two algorithms discussed in the paper (SEE and $\\theta$-WMV)\n2. Line 44: \"provides convergence best collective\" -> \"provides convergence to best ...\"\n3. Lines 47-48: \"LLM's\" -> \"LLMs\"\n4. Lines 222-223: The sentence \"Recent work on online ... is key\" doesn't read right.\n5. Line 235: \"difference between to expected ...\" doesn't read right.\n6. Line 323: \"amount of experts\" -> \"number of experts\"\n\nI am willing to increase my overall score if the questions/comments mentioned above are addressed and Section 4 (empirical study) is updated with enough detail (see weaknesses for specifics).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called Online Mixture of Experts (OMoE) for aggregating multiple expert predictions in real time to achieve superior collective decision-making, as opposed to selecting a single best expert. The authors introduce two main algorithms: (1) Successive Expert Elimination (SEE) for egalitarian voting, and (2) $\\theta$-Weighted Majority Voting ($\\theta$-WMV) for scenarios where experts should be weighted by predictive quality. The key contribution lies in combining techniques from online learning and social choice theory to provide provable no-regret guarantees. The framework is motivated by applications such as online aggregation of large language models (LLMs), where leveraging the \"wisdom of the crowd\" may outperform any single model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- **Original Problem Formulation:**\n    \n    The paper targets a gap in standard online learning frameworks like EXP4, which are designed to identify the single best expert. In contrast, OMoE explicitly models **group decision quality** and allows the learner to benefit from aggregating multiple expert outputs. This is a compelling direction, especially in settings like LLM ensemble inference.\n    \n- **Sound Algorithm Design and Analysis:**\n    - **SEE** leverages a bandit-based successive elimination strategy for pruning low-performing experts under an egalitarian majority-voting regime.\n    - **$\\theta$-WMV** introduces a more expressive model where experts are weighted by their predictive power. The use of a UCB framework with mixed-integer programming to compute optimal voting weights is clever, albeit potentially computationally intensive.\n        \n        Both algorithms are analyzed rigorously, with regret bounds stated clearly and supported by PAC-style analysis.\n        \n- **Bridging Online Learning and Social Choice Theory:**\n    \n    The connection to optimal voting rules and top-K ordinal aggregation offers an intellectually satisfying perspective and could inspire future work across disciplines.\n    \n\n**Weaknesses:**\n\n1. **Forced Fit with MoE Terminology:**\n    \n    Although framed as a mixture of experts problem, the setting is arguably closer to **ensemble LLM decision aggregation** than traditional MoE architectures (which involve conditional routing and sparsity). The terminology may overpromise or mislead readers familiar with MoE in deep learning.\n    \n2. **High Inference Overhead:**\n    \n    Since the proposed algorithms require querying all (or most) experts at each round, the approach may be **infeasible in latency- or cost-sensitive settings**, particularly when each expert is a large model like an LLM. This undermines the practicality of the method in online LLM inference tasks.\n    \n3. **Weak Modeling of Real-World Feedback:**\n    \n    The formulation assumes access to a scoring function GG that reflects correctness of the experts’ outputs. However, in real-world LLM applications (e.g., open-ended QA), determining whether an output is \"correct\" is often **ambiguous or costly**. This introduces challenges in reward signal design, supervision cost, and label fidelity.\n    \n4. **Catalogue Assumption Limits Generality:**\n    \n    The reliance on a predefined catalogue of possible outputs restricts scalability to **structured prediction tasks** with limited output space. In more realistic NLP tasks—like summarization, image captioning, or generative QA—this assumption is too restrictive and may lead to poor performance or inapplicability.\n    \n\n**Writing and Presentation Issues:**\n\n- The **abstract mentions three algorithms**, though only two are actually proposed. This may confuse readers and should be corrected.\n- The notation is at times **inconsistent or overloaded**, e.g., the symbol $P$ appears with different meanings in different parts of the paper. A careful notation pass is recommended.\n\nQUESTIONS:\nPlease comment on or discuss the above weakness parts.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an online learning framework for expert aggregation via majority or weighted voting, aiming to outperform single expert through collective decision-making. It introduces two algorithms, Successive Expert Elimination (SEE) and $\\theta$-Weighted Majority Voting ($\\theta$-WMV), with theoretical regret guarantees. Experiments on synthetic data and multi-LLM tasks demonstrate improved performance over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1) The paper introduces a new perspective on expert selection via voting-based aggregation, moving beyond traditional online single expert selection.\n2) The proposed algorithms are accompanied by formal regret analyses.\n\nWeaknesses\n1) The problem studied is not aligned with standard Mixture-of-Experts (MoE) models used in modern LLMs (e.g., DeepSeek-MoE, Mixtral). The paper’s claim that MoE selects only a single expert is inaccurate. This mismatch could mislead readers and should be explicitly clarified.\n2) The citations for \"online MoE\" primarily cover general bandit algorithms (e.g., EXP4), without discussing more directly relevant work such as combinatorial bandits, Top-K selection, or their applications to expert or LLM ensemble selection, which makes it difficult to assess the significance.\n3) While theoretically sound, both proposed algorithms adapt standard techniques like UCB-based elimination without introducing fundamentally new algorithmic ideas.\n4) The experiments are restricted to synthetic Bernoulli simulations and simple multi-LLM voting tasks. The rationale for including baselines like Zooming and “Comb. UCB” (not properly referenced) is unclear. The empirical section does not thoroughly analyze why the proposed methods outperform these baselines.\n\nQUESTIONS:\n1) In Section 2 (line 66), the scoring function is defined independently of the context $x$. Does it imply the optimal aggregated output is the same across all queries? This is counterintuitive in settings like LLMs, where the optimal outputs should be context-dependent.\n\n2) How is the scoring function defined in practice, particularly for open-ended tasks?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper is concerned with merging output of experts, which are typically LLMs, in the on-line mode. If I understood the setup correctly, the predictions are rankings on finite sets and we want to compete against optimal weighted committees of the experts. The paper proposes two algorithms, one working through expert elimination and assigning weights to experts. Regret bounds are proven for the algorithms. The papers refers to them as 'no-regret', which, I presume, stands to mean $o(T)$ regret. There is empirical evaluation using LLMs.

STRENGTHS AND WEAKNESSES:
This is an interesting and practical results.

It may be just me, but I found the organisation of the paper rather confusing. Section 2 is quite detailed, but as far as was able to ascertain, it deals with batch setup and was included mainly for motivation. The original contribution of the paper is in Section 3, and it is rather terse. I would like to clarify a few points and see a clearer comparison with the extensive machinery of prediction with expert advice.

QUESTIONS:
Are we still talking about rankings or 0-1 predictions in Section 3? If it is about ranking, is the number of categories fixed or bounded? Do we count only one top vote from an expert, or do other choices matter?

My ultimate goal is to understand how the result of the paper compares with well-known Weighted Majority of Littlestone and Warmuth, 1994.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03