Title: Principled Model Routing for Unknown Mixtures of Source Domains
Venue: NeurIPS 2025
Accepted: yes
Keywords: model routing

domain adaptation

theory
Abstract: The rapid proliferation of domain-specialized machine learning models
presents a challenge: while individual models excel in specific
domains, their performance varies significantly across diverse
applications. This makes selecting the optimal model when faced with
an unknown mixture of tasks, especially with limited or no data
to estimate the mixture, a difficult problem.  We address this
challenge by formulating it as a multiple-source domain adaptation
(MSA) problem. We introduce a novel, scalable algorithm that
effectively routes each input to the best-suited model from a pool of
available models.  Our approach provides a strong performance
guarantee: remarkably, for any mixture domain, the accuracy achieved by the best
source model is maintained.  This guarantee is established through a
theoretical bound on the regret for new domains, expressed as a convex
combination of the best regrets in the source domains, plus a
concentration term that diminishes as the amount of source data
increases.  While our primary contributions are theoretical and
algorithmic, we also present empirical results demonstrating the
effectiveness of our approach.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of routing inputs to the best-suited machine learning model from a set of domain-specialized models when the target distribution is an unknown mixture of source domains. It frames the problem as one of multiple-source domain adaptation (MSDA) and introduces a theoretically grounded algorithm to address it. Two algorithmic variants are proposed, and some empirical results are collected to demonstrate the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem is relevant and well-motivated, with clear distinctions from prior MoE and model routing works.\n- The proposed method has a strong theoretical foundation, building on MSA theory to derive novel regret bounds that hold under reasonable assumptions. Although I did not do a step-by-step review for the proofs in the appendix, the arguments used in the proofs are correct and the obtained results are compatible with the existing prior work in DA, so I think it is safe to assume that they are at least mostly correct.\n- The paper is technically high-quality, well written, well organized, and presents sufficient details and intuitions in most parts.\n\nWeaknesses:\n- The main weakness of the paper is its limited experimental scope. Although the authors claim to be mostly interested in making theoretical and algorithmic contributions, a more robust set of experiments could have been conducted.\n- The evaluation should also be conducted in a practical usage scenario, e.g. where the learned router is used to select the best model for the given input and then a single prediction is generated from the chosen model and its quality is evaluated. I am willing to raise my score if such an experiment is added (or if the authors can convince me it would bring no additional value).\n- The chosen reward metric (BLEU) has many known flaws and is no-longer the most reliable metric for virtually any NLP problem. It is therefore hard to understand its choice in this work.\n- The distinction between OLO and OLLO oracles should be made more explicit, e.g. by providing the explicit equations for both.\n- The choice of LLMs as the base models to which apply the proposed approach is a bit questionable. State of the art LLMs can handle several domains without much need for specialization / fine-tuning. This is not a major concern though, since the proposed approach is completely agnostic to the type of models it is being applied to.\n\nQUESTIONS:\n- Speaking of Option A, the authors claim that \"this approach introduces additional bias when there is high variance in the expert scores for a given input\" (lines 139 and 140). Can you explain the source/cause of this additional bias?\n- On lines 171 and 172, the authors mention including a KL regularization term, to make the predicted distribution closer to \"a uniform domain distribution or a given domain prior\". Did you include this regularization in your experiments? Did you measure its impact on the results? Have you used any other prior besides the uniform?\n\nLIMITATIONS:\nThe limitations are mentioned and acnowledged, although they are spread throughout the paper. I think the paper would benefit from having a dedicated section with a summary of those limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
The paper addresses the problem of routing inputs to the best model from a pool of domain experts when the test-time distribution is an unknown mixture of the source domains. The authors formulate the problem as a multiple-source domain adaptation (MSA) problem and propose an algorithm based on minimax-regret optimization. 

They provide theoretical guarantees and validate the approach empirically.

STRENGTHS AND WEAKNESSES:
## Strengths
- Presentation: The paper is exceptionally well-written, clear, and logically structured.
- Theoretically Grounded: The work is built on a solid theoretical foundation. The minimax-regret formulation seem sound

I appreciate the note on applying the framework to discriminative models.

## Weaknesses
- None of the paper as far as I am aware. My primary limitation as a reviewer, detailed below, is in assessing the novelty and significance

I have spent considerable time on this paper. Its quality is apparent. The formalism, the derivations, and the theoretical analysis all appear to be correct. However, the specific field of multiple-source domain adaptation is outside my expertise. While I can attest to quality of the writing, I cannot comment on the novelty or significance of the work.

This appears to be a high-quality paper, and I recommend accepting it. My confidence is, however, low.

QUESTIONS:
I refrain from posing questions due to my low confidence, and apologize for this to the AC and the other reviewers.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission investigates the model routing problem for testing data that are unknown mixtures of source data domains. The authors formulate this challenge as a multi-source domain adaptation problem and propose a novel and scalable routing algorithm that can achieve regret bounds tightly linked to the online learning oracles. Strong theoretical guarantees ensure the robustness of the proposed model routing strategy. Empirical results on LLM instruction tasks also demonstrate its advantage over the baseline of uniform routing.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The studied model routing or selection problem is of great significance in improving model performance in complex real-world deployment scenarios.\n\n2. Both the perspective of taking model routing as multi-source domain adaptation and the proposed routing algorithm are novel.\n\n3. The presentation is clear, and the theoretical analysis is very solid.\n\n## Weaknesses\n\n1. Although I acknowledge that the primary contributions of the paper are theoretical and algorithmic, current empirical results are not extensible. First, it only demonstrates the advantage of the proposed routing algorithm over the vanilla uniform routing strategy. It would enhance the empirical justification by providing results of advanced model selection or routing baselines. Second, since multi-source domain adaptation is widely studied in vision tasks [Hoffman et al., 2021],  adding empirical results of vision tasks would also enhance the justification.\n\n2. The efficiency analysis of the algorithm is not discussed. Whether efficiency may be a concern depends on the problem setting. What is the sample size of the target domain? Is target access and evaluation offline or online? What if the target domain is not a trivial mixture of source domains, i.e., the target domain consists of various out-of-distribution data?\n\nQUESTIONS:\nCheck the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
The paper introduces an algorithm for predictive model routing in the presence of unknown mixtures of source domains, framed as a multiple-source domain adaptation problem. The authors leverage minimax regret optimization and online learning oracles to design a routing function that selects the best model for a given input among several domain-specialized models. The approach is supported by theoretical regret bounds  and evaluated empirically on the MixInstruct benchmark using LLMs.

STRENGTHS AND WEAKNESSES:
strengths:
- The paper provides detailed, formal regret bounds for both online learning oracles and online Log-Loss oracle -based algorithms grounded in established multiple-source domain adaptation literature.
- The paper openly discusses limitations of assumptions 

weakness:
- LLMs typically have prior knowledge of application tasks and their relative frequencies (contrary to the paper's assumption on unknown domain mixtures
- The proposed framework targets robustness to arbitrary mixtures. In practice, adversarial or drastically shifting mixtures are rare, and the central technical contributions seem tailored to an artificially difficult setting rather than one current generative ML systems face.
- The saddle-point approach and dueling no-regret learners follow closely from the cited literature, the problem framing and regret decomposition rely directly on classic MSA theory. The paper primarily applies these known tools to this contrived setting without introducing theoretical innovations beyond careful integration.

QUESTIONS:
I suggest analyzing routing failures in existing LLMs to motivate scenarios where predictive, input-only routing with unknown mixtures is practically impactful.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 1
DATE: 2025-07-07