CONCUR: A Framework for Continual Constrained and Unconstrained Routing
Authors: Peter Baile Chen
Weiyue Li
Dan Roth
Mike Cafarella
Samuel Madden
Jacob Andreas
Venue: ICLR 2026 Poster (2026)
Keywords: model routing
continual routing
constrained routing
unconstrained routing
Primary area: foundation or frontier models, including LLMs

Abstract: AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial.
Most prior methods build the routing framework by training a *single* model across *all* strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization.
Prior models also typically use a *single* input representation, limiting their ability to capture the full complexity of the routing problem and leading to sub-optimal routing decisions.
To address these gaps, we propose CONCUR, a **con**tinual routing framework that supports both **c**onstrained and **u**nconstrained **r**outing (i.e., routing with or without a budget).
Our *modular* design trains a separate predictor model for each strategy, enabling seamless incorporation of new strategies with low additional training cost.
Our predictors also leverage *multiple* representations of both tasks and computation strategies to better capture overall problem complexity.
Experiments on both in-distribution and out-of-distribution, knowledge- and reasoning-intensive tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, while also reducing training cost in the continual setting.

Published as a conference paper at ICLR 2026
CONCUR: A FRAMEWORK FOR CONTINUAL CON- STRAINED AND UNCONSTRAINED ROUTING
Peter Baile Chen' Weiyue Li
Dan Roth
Michael Cafarella' Samuel Madden' Jacob Andreas
MIT Harvard University 3Oracle Al and the University of Pennsylvania
Correspondence: peterbcemit.edu
ABSTRACT
AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an ef- fective routing system that maps tasks to the appropriate strategies is crucial. Most prior methods build the routing framework by training a single model across all strategies, which demands full retraining whenever new strategies appear and leads to high overhead. Attempts at such continual routing, however, often face difficulties with generalization. Prior models also typically use a single input representation, limiting their ability capture the full complexity of the routing problem and leading to sub-optimal routing decisions. To address these gaps, we propose CONCUR, continual routing framework that supports both constrained and unconstrained routing (i.e., routing with or without a budget). Our modular design trains separate predictor model for each strategy, enabling seamless in- corporation of new strategies with low additional training cost. Our predictors also leverage multiple representations of both tasks and computation strategies better capture overall problem complexity. Experiments on both in in-disrrbuution and out- of-distribution, knowledge- and reasoning-intent intennvvveee tasks show that our method outperforms the best single strategy and strong existing routing techniques with higher end-to-end accuracy and lower inference cost in both continual and non- continual settings, while also reducing training cost in the continual setting [T]
INTRODUCTION
AI tasks vary in difficulty, and thus optimally served by different computation strategies, such as selecting appropriate models (small or large language models) and decoding methods (with or without chain-of-thoughe reasoning (Wei et al.) 2022)). Effective routing ensures tasks paired
with the most suitable strategies to help improve overall accuracy and reduce runtime and costs.
Prior routing work (Zhu et al.) 2025 Ong al. 2024 Ding al. 2024 Lu al. 2024 Hari &
Thomson 2023; Chen et al. 2024 Feng et al. 2225 Pan el al. [2055] Zhuang et al. 2024]] Liu et al.
2024| Sakota al. [2024] Mohammadshahi et al. [224 Nguyen al. [2224] Damani et al. (024]
typically employs a fixed set of computation strategies, relying on a single model trained jointly on data from all strategies. However, this monolithic design limits generalization
continual settings
where routers need to quickly adapt previously unseen strategies. In practice, continual routing crucial, as better and increasingly efficient models and decoding methods are constantly emerg- ing, and not incorporating them promptly risks missing potential gains in accuracy and reductions in computational cost. However, whenever a novel strategy appears, existing approaches require retraining the model from scratch using data that covers both previous and new strategies.
Although some recent efforts attempt to move toward a continual setting, they raise significant con- cerns about generalizability. For example, Waan et al. (2025) adopts modular design by training separate router models for different computation strategies. However, since these router architec- tures differ and are specifically tailored to individual strategies, extending them unseen strategies remains non non-trivial. Jitkrittum et al. 2025) introduces zero-shot router based on model feature vec-
Data and code are available ht tttppp://aat pattss///petrbaat ii.bithub. / concur/

---

Published as a conference paper at ICLR 2026
Taok reerlppion
prodicter each stretepy prodictt
Modular design:
performance taak
scouncy
osd
Semwsk papoue
Constrained
Computation strategy
Mouil dearptin
Oven
12
optinal stratgg
80
Unconstrained
descrgee piacs
Tesi-ged''s bilker
LannE sile mos.
Though C1TTT
Ovi bogg
Figure I: CONCUR learns one predictor per computation strategy that uses multiple input repre- sentations to support continual routing and better routing decisions under both continual and non- continual settings.
tors, enabling generalization unseen models without retraining. However, their method depends on predefined prompts, which limits its adaptability to varied prompts and tasks.
Besides efficiency concerns in the continual setring, prior work also raises concerns about end-to- end performance (accuracy and inference cost) in both continual and non-continual settings. In principle, models should adopt flexible parameterizations that allow them combine both general- purpose and task- or strategy-specific signals, thereby capturing richer information about the routing problem and enabling high-quality routing decisions. However, prior work typically relies on highly restricted parameterizations; for example, Ong et al. (2024) and Zhu et al. (2025) parameterize only task-level representations, while Zhuang et al. (2024) and Pan et al. (2025) ) restrict themselves to single parameterizatton of computation strategies and input tasks, respectively. Such limited designs
may reduce expressivity and constrain the quality of routing decisions.
Motivated by these issues, we propose generalizable routing framework applicable to both contin- ual and non-continual, as well as constrained and unconstrained, settings, as illustrated in Figure [ Our framework adopts modular design, where separate predictor models are trained for each com- putation strategy, using both general-purpon and task-specific representations of input tasks and computation strategies to estimate accuracy and efficiency. These estimates are then used to for- mulate constrained and unconstrained routing as optimization problems, which can subsequently be solved to determine the optimal routing decisions.
In contrast to prior approaches that rely on a single model trained on data from all strategies, our modular predictor design makes continual routing far more practical. New strategies be incor- porated simply by training an additional predictor, without retraining existing ones, thus avoiding costly overhead. Moreover, unlike prior continual routing efforts that lack generalizability, either by tailoring router architectures to specific strategies or by relying on fixed prompts, our predic- tors share the same model architectures, and our method imposes no restrictions on prompts or task diversity.
In addition, rather than restricting to a single representation as in prior work, our architecture in- corporates multiple representations of both input tasks and computation strategies to capture richer information about the routing problem, enabling more accurate routing decisions and improved end- to-end performance in both continual and non-continual settings.
In summary, we present CONCUR, a framework for continual constrained and unconstrained routing. CONCUR trains modular predictors for accuracy and efficiency that draw on both general- purpose and task-specific representations of input tasks and computation strategies, and integrates these estimates with specific routing algorithms to address constrained and unconstrained rout- ing. We evaluated CONCUR on diverse benchmarks (including multi-hop QA, general reasoning multiple-choice tasks, and math problems) across both in- and -uttof-diitribution settings. Results show that CONCUR consistently outperforms the best single strategy baseline and existing rout- ing methods, achieving higher end-to-end accuracy and lower inference cost in both continual and non-continual settings, as well as improved training efficiency in the continual setting.
2 METHODOLOGY
As outlined in Section [] our goal is to build a routing framework that supports continual settings and improves end-to-end performance in both continual and non-continual settings. The core ideas be-
2

---

Published as a conference paper at ICLR 2026
Accurasy
MLP
MLP (two liner layers)
Cost
Task-spocific repessentations
Embbedding loosup
Fintedding lookup
Liner projecioon
Of-ihw-seel we enieedding modei
Unwer pojection
Entedding lockup
Fmbeerting tokus
Mode
Deouding netroa
Made deszaptio
becddngmmthl
Thos decaiptioo
Model
recodinrretoot
stengray
ky
Figure 2: Overall predictor architecture of CONCUR. For each computation strategy 8j, train two predictor models: one estimates the accuracy of applying 8j to the input task, and the other estimates its cost, using both general-purpose and task-specific representations.
hind our routing models are: (1) we adopt a modular design, training separate model for each strat- egy so that extending to new strategies only requires training additional predictors without touching existing models; and (2) we use multiple input representations better capture the complexity of the routing problem, rather than relying on a single representation.
Concretely, we formulate both constrained and unconstrained routing as optimization problems over accuracy and efficiency. Therefore, predictors are trained to estimate these metrics, which are sub- sequently used to solve the optimization problems. Building on this design, Section 2.1 describes how we train modular predictors to model task difficulty using multiple input representations, and Section 2.2 explains how these predictions drive routing decisions.
2.1 PREDICTORS
We outline the training of predictors designed to estimate the performance of applying a computation strategy to a given user task. This involves characterizing input tasks and strategies, detailing the predictor model architectures, and explaining the training procedure and prediction process.
Characterization. We define a task & as comprising both the question and any related context. While prior work (Ong al. 2024 Zhuang al. 2024] Pan al. 2025) typically considers only the question, task may also include supporting documents, such as in the open-domain QA setting, that provide useful signals for estimating task difficulty. We define computation strategy 8y as a (model, decoding method) pair (m,,d) d,). In our implementation, model denotes the underlying language model (e.g., Qwen2.5-7B-Instrucct while decoding method refers to the decoding algorithm (e.g.. chain-of-thoupee
However, developers can broaden the definition of computation strategies to incorporate additional parameters. Given a set of supported language models M and decoding methods D, the complete set of computation strategies S is the Cartesian product $ = M D.
The performance of applying strategy s; to task t; includes both the accuracy Giy and the inference cost Cij. which FLOPs are preferred over token counts because model sizes vary, and the computational cost of generating single token differs across models. Using FLOPs allows for more standardized com- parison of efficiency across heterogeneous models. However, depending on the deployment envi- ronment (c.g., whether models are quantized), FLOPs can be replaced by other numerical efficiency metrics, including throughput, latency, or energy consumption.
a proxy for efficiency. In our setting, measure the cost using FLOPs.
Architecture. To achieve high- predictions, we train two independently parameterized predictors: one for estimating accuracy and the other for estimating cost. Each predictor incor- porates both general-purpose and task-specific representations of the input task 4 and strategy sj (m,, d,), enabling them to capture both general and task-strategy specific characteristics. For every sy. two such predictors are trained, resulting in a modular design where predictors for dif- be trained independently. This allows new strategies be supported by training
ferent strategies only the corresponding predictors, leaving existing ones untouched and incurring minimal overhead. The overall architecture is shown in Figure 2
General General-purpose purpose representation. We generate a general-purpos representation by passing the textual description of the task and strategy (textual descriptions of the strategies are provided in
3

---

Published as a conference paper at ICLR 2026
Appendix A) through off-the-shell text embedding model R.
g = R(t,), g) = R(my), 87 = R(dj)
Concatenating the three parts gives the general representation Bij = |s gigg g/)] E R3 where ls is the dimension of the encoded representation.
2. Task-specific representation. Task-specific representations are derived using learnable projections and embeddings. The task-specific representation of the input task is derived by linearly projecting its previously defined and decoding method are derived from learned embeddings that map model and decoding method IDs into trainable dense vectors optimized alongside the rest of the model.
representation. Task-specific representations for the model
t = W W m = E my,, dj = EbId, to = WCt() mj = [my). dj = E
where X and X represent X in the accuracy and cost predictor models, respectively. The metric (i.e., accuracy or -ostc-specific linear projection denoted by W, E k while EM and ED are metric-specific embedding lookup tables for models and decoding methods, respectively.
Concatenating the three parts gives the task-specific representations st = [t;; mf; dg] and sig = mf: dğ] E R.sk
MLP, Finally, concatenate the general-pune pprrpsse and task-specific representations and feed them through two linear layers to produce the accuracy and cost predictions, 0.j and Eij.
Øsj fd [[aa) s)]); رن = [[]]
where f is binary classifier and f is regressor for predicting the accuracy and cost, respectively.
We note that once trained, the representations of a strategy 8 remain fixed and thus contribute a will show in Section including these strategy
constant term to the predictions. Nonetheless, representations improves performance over strong routing baselines.
Training. Using the training tasks and their target answers, apply each strategy 8j each task 4 to obtain the ground-truth labels as (by comparing the generated and the target answers) and cij- Due to the modular design with respect each sy. training predictor for given strategy only requires the training data associated with that strategy. The training procedure for the predictors is as follows. The accuracy predictor is binary classifier trained with cross-entropy loss:
Lace = -ai log (ā:z) - (1 - a:j) log (1- - āiz)
where azy is the ground-truth truth accuracy label and â:j is its predicted value. The cost predictor is regressor trained with mean squared error loss:
Leost (cy 22y)
where Cty is the ground-trute truth cost and ég is its predicted value.
Inference. For task t, encode it with the same embedding model used during training and pass this representation, along with the representation of each strategy $j, through the respective accuracy and cost predictors to obtain dij and ézj.
Continual routing. Continual routing refers to a setting where the routing framework is trained on an initial of strategies S and must later incorporate additional strategies S' that become available at later time. Our modular design assigns a separate predictor to each strategy 8j, so incorporating a new strategy s' involves training only its predictors, leaving previously trained models unchanged, making extensions straightforward and efficient. Naturally, our framework also applies to the non- continual setting where the set of strategies remains fixed over time.
2.2 ROUTING
Using the predicted accuracy a,j and cost C₁ of applying strategy s, to task t,, demonstrate how both constrained and unconstrained routing can formulated as optimization problems and solved, leading to the final routing decisions.
4

---

Published as a conference paper at ICLR 2026
Unconstrained routing. For given task t,. unconstrained routing involves choosing the compu- tation strategy that achieves an optimal trade-off between accuracy and cost. This can be framed as the following bi-objective optimization problem, maximizing accuracy while simultaneously mini- mizing cost: maxj Oij. miny Cij-
By introducing weight represent the trade-off between accuracy and cost, the bi-objective op- timization problem can be reformulated as a single-objective problem that maximizes the weighted sum of these Iwo objectives.
max
w asj +1 - w) (-ci))) Σ max(w ( asj + +1 w) (-ci))
(I)
Then, &; is routed to the strategy s that maximizes the weighted sum.
Constrained routing. For a task 4 with an associated cost budget B, constrained routing seeks the computation strategy that delivers the highest possible accuracy without exceeding the budget. This can be expressed as the following optimization problem: maxj,ay dij
However, optimizing each task individually may not yield the best overall result, local optima do not always translate to global optimality. For a batch of n tasks (1, 12,,,,,, In the constrained optimization problem can be reformulated as
max 5.E, Cey
aij
We address this optimization problem by formulating dynamic programming (DP) approach (see Appendix B for details), which has an overall complexity of OOn- (S)- = 0 B|S)). where B represents the budget per task and (S] is the number of computation strategies. Since the budget can kept relatively small through scaling, and the number of computation strategies (i.e., L.MM- decoding pairs) used simultaneously is usually moderate, the DP algorithm can be solved efficiently for a reasonable number of tasks.
Although the above formulation is designed to maximize accuracy under cost constraint, it can be straightforwardly adapted to minimize cost for a given accuracy requirement.
EXPERIMENTS
3.1 EXPERIMENTAL SETUP
Datasets. We select diverse set of tasks covering different skills and output formats: factual multi-hop question answering with short text answers, general reasoning with multiple-choice an- swers, and mathematical problems requiring numerical answers. For each task category, we select two datasets: one in in-distribution for both training and testing, and one out-of- out-of-distribution reserved exclusively for testing. A summary of these datasets and their statistics provided in Table
Table Datasets and their sizes.
In-distribution
Out-of-distribution
Dataset
Test size
Dataset
Test size
Multi-hop QA General reasoning Math problems
2WikiMultiHop (o al.
2020
1000
HotpotQA Yang et al.
2018
500
MMLU (Hendrycks 2021
1000
GPQA (Rein al.
448
GSM8K (Cobbe et al.
2021
1000
SVAMP Patel al. 2021
500
Computation strategies. As described in Section 2.1 the set of computation strategies comprises combinations models and decoding methods. For models, we used Qwen2.5-Instruct models (1.5B, 3B, and 7B) and -lams-et models (3.2-3B and 3. 3.B). For decoding, we consid- ered two common approaches: vanilla, where models directly generate the answer, and chain-of- thought (Wei al. 2022). where models produce intermediate reasoning steps before the final swer. The descriptions and prompts for each strategy are presented in Appendix A and Appendix(C) respectively. In total, this yields five L.MSs and two decoding methods, for total of ten strategies.
5

---

Published as a conference paper at ICLR 2026
Average performance on in-distribution datasets
Average performance on out-of-distribution datasets
Accumar)
Accurary
22.35
30.55
Hound
05.9%
350
27.5
32.3
50.0
355
375
300
Figure 3: Pareto curves for unconstrained routing on both in- and out-of-distribution datasets across various values of w defined in Section 2.2 illustrating the trade-off between accuracy and cost. Full diagrams are available in Appendix
Baselines. We compare our routing framework against the best single strategy without routing and three strong routing baselines. Implementation details are provided in Appendix D
(1) Best single strategy: We use the same model-decoding pair that achieves the highest overall accuracy across all tasks. In this case, Qwen2.5-7BIInstree with chain-of-thought (CoT) decoding selected for its superior accuracy.
(2) RouteLLM (Og et al. 2024): This approach uses single classifier that relies on the single general-purpose task representations to select a strategy. Since it does not take the budget into account, we evaluate it only in unconstrained settings.
(3) EmbedLLM (Zhuang et al. 2024): Originally, this method uses a single model to predict the accuracy of each strategy and picks the one with the highest accuracy. We adapt it by adding additional model to predict the cost. Unlike CONCUR, EmbedLLM only uses a single task-specific representation for computation strategies.
(4) RTR (Pan et al. 2025): : This method employs a single model to jointly predict accuracy and cost for all strategies. In contrast CONCUR, RTR only uses single general-purpose representation for input tasks.
Metrics. We compare method and the baselines based end-to-end performance. Once com- putation strategies are assigned to tasks, we compute the overall accuracy and total inference FLOPs (as a proxy for efficiency and cost) for each approach. Inference FLOPs are calculated using the standard formula from Kaplan et al. (2020) and scaled down by 10 for readability.
In the following subsections, we first evaluate our method against the baselines non-continual un- constrained (Section (322) and constrained (Section[3.3 settings to primarily assess the effectiveness of our model architecture, which leverages multiple representations of both input tasks and compu- tation strategies to improve end-to-end performance. We then examine performance in continual setting (Section 3.4 highlight the impact of our modular design on reducing training cost.
3.2 UNCONSTRAINED ROUTING
Figure shows that our method consistently outperforms the baselines in both in-distribution and out-of-distribution scenarios. Tables[land the maximum accuracy achieved by each method. Among the approaches that surpass the best single strategy, our method generally achieves the high- est accuracy with the lowest FLOPs. This demonstrates that, compared to the best single strategy, routing enables higher accuracy at reduced computational cost. Furthermore, when compared to other routing baselines, our method delivers both superior accuracy and efficiency, highlighting its higher effectiveness router.

---

Published as a conference paper at ICLR 2026
Table 2: Performance for unconstrained routing in-distribution datasets: 2wikiMultiHop.. MMLU, and GSM8k. Gray denotes methods whose average accuracy falls below that of the best single-strategy baseline. Bolded numbers indicate the best performance among the remaining methods.
2WikiMultiHop
MMLU
GSM8k
Average
Acc
FLOPs 4
Acc
FLOPs
Acc
FLOPs
Acc
FLOPs
Best single strategy
57.6
49.63
73.7
44.45
91.6
30.82
74.3
41.63
RouteLLM
41.7
17.90
54.7
3.20
64.1
9.51
53.5
10.20
EmbedLLM
58.4
43.68
72.1
33.68
89.8
25.28
73,4
34.21
RTR
57.9
44.04
73.7
44.45
91.3
33.07
74.3
40.52
CONCUR (Ours)
59.5
38.54
74.4
40.73
91.6
30.23
75.2
36.50
Table 3: Performance for unconstrained routing cittof-distribution datasets: HotpotQA, GPQA, and SVAMP.
HotporQA
GPQA
SVAMP
Average
Acc
FLOPs
Acc
FLOPs
Acc
FLOPs
Acc
FLOPs
Best single strategy
59.8
39.75
35.0
75.18
92.2
23.46
62.3
46. 13
RouteLLM
52.8
6.62
29.5
4.22
79.8
8.18
54.0
6.34
EmbedLLM
57.0
30.15
35.3
69.75
92.0
19.63
39.84
RTR
59.8
36.16
35.0
75.18
92.4
23.65
62.4
45.00
CONCUR (Ours)
60.6
33.62
35.3
73.31
92.0
22.75
62.6
43.23
Table 4: Accuracy gains of constrained routing when transitioning from local optimization (L) to global optimization (G), along with the accuracy achieved by global optimization under varying budget settings. Bolded and underlined numbers represent the performance of our method when it ranks the best and second best, respectively.
2WikiMultiHop
MMLU
GSMSk
Average
AL
ALL G)
G
ALL G
ALL G)
G
Low budget (FLOPs budget 25)
EmbedLLM
+4.4
52.4
+3.2
70.6
+1.5
89.8
+3.0
70.9
RTR
+6.0
53.9
+4.7
73.7
+2.0
90.4
+4.2
72.7
CONCUR (Ours)
+9.7
56.5
+3.8
72.7
+3.7
90.3
+5.7
73.2
High budget (FLOPs budger 40)
EmbedLLM
+4.6
57.6
+2.4
72.6
0.0
90.1
+2.3
73.4
RTR
56.6
+2.9
74.1
+0.2
+2.1
74.0
CONCUR (Ours)
+8.1
59.5
+3.2
74.4
+0.8
91.6
+4.0
75.2
3.3
CONSTRAINED ROUTING
Constrained routing means routing under budget constraints. To test generalizability, we evaluate performance under both low- and high-budget settings. Furthermore, noted in Section 2.2 given the predicted accuracy and cost, the constrained routing problem can be addressed two ways: (1) local optimization, which treats each task independently and allocates the budget evenly across tasks, and (2) global optimization (our approach), which distributes the total budget jointly across all tasks. We also compare the performance when using these two optimization methods.
Examining the accuracy improvement from the local optimization baseline to our global optimiza- tion method, Table 4 shows substantial positive change in both budget settings, demonstrating the effectiveness our global optimization approach in making better routing decisions. Moreover, when comparing the accuracy of different routing methods under global optimization, Table Mable/4ndee indi- cates that our method achieves the highest average accuracy across both settings, highlighting the overall strength of our routing framework.

---

Published as a conference paper at ICLR 2026
Average performance with large models only
Average performance with both large and small models
18.9%
94.55
14.9%
74.5%
Accunary
2..9
Accuary
72.0%
Ented
Briec Mecin
70.99
70.%
E7hy
KThrom
bount
35.
27.5
32.3
2000
250
375
300
0%-3
Figure 4: Performance of different methods under the continual routing with different collections of strategies. X5s denotes method X trained from scratch. XF7r%; denotes method X fine-tuned from its prior version, which was trained from scratch in Setting I, using Y% of the new data.
3.4 CONTINUAL ROUTING
Table 5: Performance for continual routing. Table bolding scheme.
2
provides the definitions of the coloring and
2WikiMultillop
MMLU
GSMSk
Average
Relative training time
Ace
FLOPS 1
Ace
FLOPS
Acc
FLOPs
Acc
FLOPSs
Setting 1: Large models only (Qwen2.5-72) 5-7-hhrrcct and Llama-3. (88--nstruct))
Best single strategy
$7.6
49.63
73.7
44.45
91.6
30.82
74.3
41.63
RouteLLMps
0.18x
54.0
48.77
68.7
17.03
90.0
30.06
70.9
31.96
EmbedLLLMss
1.32x
58.9
49.62
73.3
44.29
91.2
33.44
74.5
42.45
RTREs
3.93x
59.4
51.12
72.1
42.98
91.6
30.82
74.4
41,64
CONCUR (Ours)
1.00x
59.3
50.71
74.5
41.02
91.7
30.78
75.2
40.84
Setting 2: Large and small models (Qwen2.5, Llama-3.1, and Llama-3. 3.2 family)
Best single strategy
57.6
49.63
44.45
91.66
30.82
74.3
41.63
RouteL.MMss
0.20x
41.7
17.90
54.7
3.20
64.1
9.51
$3.5
10.20
RouteLLM, prr(09))
0.14x
41.6
18.21
53.5
2.80
61.5
9.69
52.2
10.23
Routel.LM (7(0)
0.17x
41.6
19.96
52.9
2.90
66.5
10.13
$3.7
11.00
MouteLLM/7((re))))
0,20x
41.5
13.73
53.0
2.91
59.1
9.45
51.2
8.70
RouteLLMp77(1000
0.23x
41.3
16.62
54.6
3.28
62.5
9.33
52.8
9.74
EmbedL
3.08x
58.4
43.68
72.1
33.68
89.8
25.28
73.4
34,21
Embedd.LMyyrra00
1.0xx
56.9
40.12
68.9
34.45
89.6
30.83
71.8
35.13
Embeell..M
1.92x
57.4
36.87
68.3
26.36
89.5
30.64
71.7
31.29
Embedl.
2.82x
55.1
36.19
69.5
32.69
90.2
27.40
71.6
32.09
EmbedLLLpp7(0nns
3.11x
59.2
42.93
70.0
32.59
89.7
26.45
73.0
33.99
RTR
7.66x
57.9
44.04
73.7
44.45
91.3
33.07
74.3
40.52
1.69
39.80
88.2
72.7
31.85
RTRy)(50))
2.84x
47.79
13.53
88.2
13.51
71.9
24.95
RTR (7(755)
3.89x
58.9
47.81
73.6
44.38
91.6
30.82
74.7
41.01
RTR
4.96x
57.6
44.06
73.4
43.19
88.3
17.91
73.1
35.05
CONCUR (Ours)
1.00x
59.5
38.54
74.4
40.73
91.6
30.23
75.2
36.50
We consider the following scenarios where routers must adapt unseen computation strategies. Initially, an organization prioritizes accuracy and selects moderate-to-larg models: Qwen2.5-7B. Instruct and -
Later, to reduce cost and latency without sacrificing aceu- racy, the organization introduces smaller models: Qwen2.5-1.33 1.ee2.5-1.5B-Instrru
Qwen2.5-3B-Insttre and Llama-3.2-33 1.mma--..-3-B-mmmm Instruct.. We refer to the first scenario with only large models as Setting and the second scenario with both large and small models Setting 2.
Since all routing baselines L.M EmbedLLM, and RTR) train a single router model using training data across all computation strategies, the straightforward way incorporate unseen strate- gies to retrain the model from scratch using all available data. However, this can be unnecessarily costly given that the router model has already been trained prior data. To address this, also
8

---

Published as a conference paper at ICLR 2026
evaluated variants of these baselines where the existing router models fine-tuned randomly sampled 25%, 50%, 75%, and 100% the new training data.
In addition to end-to-end performance metrics (accuracy and inference FLOPs), explicitly mea- sure the training time for each method as an indicator of training cost, reflecting how casily each method can adapt unseen strategies.
Figure shows that in both Setting 1 and 2, in terms of end-to-end performance, CONCUR out- performs all baselines. As shown in Table [ in Setting 1, our method achieves the highest average accuracy with the lowest FLOPs among all baselines, demonstrating the effectiveness of our rout- ing framework. In Setting among routing baselines that outperform the best single strategy, our method again achieves the highest accuracy with the lowest FLOPs, while requiring significantly less training time. This highlights the advantage of our modular predictor architecture, which al- lows easy extension to unseen strategies. Importantly, the goal of the organization is to reduce FLOPs while maintaining aceuracy when moving from Setting Setting a target achieved only by the RTR7775%) baseline and our method, with our approach performing substantially better.
4 ANALYSIS
Section 3 highlights that CONCUR outperforms existing approaches. In this section, we present deeper analysis to better understand the sources of this improvement (Section [4.1.. We also per- form an ablation study to examine the effectiveness of using multiple representations for input tasks and strategies (Section 4.2). All analyses were conducted under the unconstrained routing setting
discussed in Section 3.2
4.1 BENEFITS OF ROUTING
Table 6: The table shows (1) the percentage of tasks routed by our framework to different strategies, (2) the performance of tasks using routed strategies by our framework compared to using the best single-strategy baseline transitions from the baseline strategy the routed strategy by our framework, where and indicate correct and incorrect, respectively. Bolded numbers indicate cases where the routed strategy by our framework outperforms the baseline strategy.
1a-2255--B-nntrrue with CoT), and (3) the distribution of task accuracy
Baseline stralegy
Routed strategy
Task accuracy transitions (%)
Tasks routed (%)
Acc
FLOPs 1
Acc
FLOPs
C I-C
1
C-11
2WikiMulillop
QwenBB-CoT (Baseline)
33.6%
82.4
47.7
Qwen2B-vanill
25.3%
22.9
56.9
27.3
7.8
16.2%
11.1%
66.0%
6.7%
LlamaSB-CoT
21.0%
54.8
48.6
56.7
70.5
48.1%
8.6%
36.7%
6.7%
Qwen7B-vanills
17.1%
67.3
43.5
29.9
8.2%
Others
3.0%
36.7
51.8
33.3
21.0
23.3%
10.0%
53.3%
13.3%
MMLU
Qwen7B B-CoT (Baseline)
83.6%
74.2
44.4
Qwen7B-vanile
12.7%
69.3
44.5
78.0
14.4
64.6%
13.4%
17.3%
4.7%
Others
3.7%
78.4
45.3
67.6
48.1
62.2%
5.4%
16.2%
16.2%
GSM8k
Qwen7B-Col (Baseline)
96. 0%
91.6
30.9
92.5
27.9
92.5
13.2
87.5%
5.0%
demonstrates that CONCUR outperforms the best single-strategy baseline (Qwen2.5-7B: Instruct with CoT decoding) in both accuracy and efficiency. In this section, take closer look to understand the source of gain. Specifically, we analyze the routing decisions made by our framework in comparison to the best single-strategy baseline, which routes all tasks to fixed computation strategy. We also analyze how task accuracy varies between the baseline and our framework.
Section 3
As shown in Table our framework assigns most GSM8k tasks to Qwen7B-Con remains similar compared to the best single-strategy baseline. However, for 2WikiMultiHop and MMLU, our framework routes substantial portion of tasks cost-efficient strategies (smaller
performance

---

Published as a conference paper at ICLR 2026
models and/ simpler decoding methods). These alternatives often achieve higher accuracy while significantly reducing FLOPs, explaining the performance gains of our framework.
Additionally, Table 6 shows how tasks distributed according to their accuracy changes when moving from the best single-strategy baseline to our framework, indicating whether they stay cor- rect//ncorrect transition between the two. Most tasks follow one of two patterns: (1) they keep their original correctness but are routed to cheaper strategies, significantly reducing computation cost and improving efficiency; (2) they switch from previously incorrect correct answers, lead- ing to gains in accuracy. A small fraction of tasks change from correct to incorrect, but these losses are minor compared the overall improvements. This again illustrates the source of our gains.
4.2 ABLATION STUDY ON REPRESENTATIONS
As discussed in Section 2.1 our framework considers two types of representations for input tasks and strategies: general-puryy purpose representation and task-specific representation. To verify the contri- bution of each, we evaluate our framework using only the general-purpose representation, only the task-specific representation, and the combination of both. As shown in Table 7 each representation provides valuable information, while their combination consistently delivers the best accuracy. This demonstrates that the representations offer complementary signals and that leveraging both is crucial for realizing the full effectiveness our framework.
Table 7: Performance of CONCUR in unconstrained routing when using different representations for both input tasks and strategies. Bolded and underlined numbers represent the performance of our framework using both representations when it ranks the best and second best, respectively. 2WikiMulttHop
MMLU
GSM8k
Average
Acc
FLOPs +
Acc
FLOPs
Acc
FLOPs
Acc
FLOPs
ceneral-purpose only
57.9
43.82
72.5
25.71
89.5
37.05
73.3
35.53
Task-specific only
58.7
45.69
68.5
29.01
89.8
37.25
72.3
37.32
Both (CONCUR)
59.5
38.54
74.4
40.73
91.6
30.23
75.2
36.50
5 CONCLUSION
This work introduces CONCUR, framework for continual constrained and unconstrained routing. Central to CONCUR are modular predictors that leverage both general-purpose and -asc-fpec representations to estimate strategy's accuracy and cost given task, enabling optimization- based routing and straightforward extension to unseen strategies. Extensive experiments on a diverse set of in-distribution and out-of-distributee tasks show that CONCUR outperforms the best single strategy and existing strong routing methods in both continual and non-continual settings.
ACKNOWLEDGMENTE
We gratefully acknowledge support from the DARPA ASKEM Award HR001 TRR001122000 12220042, the ARPA-H Biomedical Data Fabric project, Liberty Mutual, Jane Street, the MIT MGAIC Consor- tium, the MIT Everest Projects, the MIT-IBM Watson AI Lab, ONR-LARD and the Google Phl) Fellowship in collaboration with MIT. We also thank Amazon, Google, and Intel for their contributions through the MIT Data Systems and AI Lab (DSAIL).
REFERENCES
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. RouterDC: Query- based router by dual contrastive learning for assembling large language models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
Sopenneview.ne//Eooee wtt.:oormmmmmmmuvrrmae jayMi
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Refichiro Nakano, Christopher Hesse, and John
10

---

Published as a conference paper at ICLR 2026
Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Mehul Damani, Idan Shenfeld, Andi Peng, Andreea Bobu, and Jacob Andreas, Learning how hard to think: Input-adaptive allocation of Im computation. arXiv preprint arXiv:2410.04707, 2024.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks
V. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://openreviat
net ftte:/ooooommmmo2222
Tao Feng. Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for LLM selections.
In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview.cr net/ wttt:///ooooommdee00000eee
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent routing of user prompts to large language models. arXiv preprint arXiv:2308. 1601, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Mcasuring massive multitask language understanding. In International Confer- ence Learning Representations, 2021. URL https:: htttp:://opereeview. http:/wwwwwweeeeeeeeeeeee feret/forummee d7KB (015mmm3mmmm 3GmQ.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, Barcelona, Spain (Online), December 2020. International Com- mittee on Computational Linguistics. doi: 10. 10.18863/v1/2000 www.000..olinngminn...m
URL https:
//aclanthology.ccr wrgp:000.0gg/00000oo0o0 coling ma main. 580/
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Uni- versal model routing for efficient Ilm inference. arXiv preprint arXiv:2502.08773, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan. Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL httpt: httpaaxiviv.o.o/ org. hbwgg/bb//000
http://///ariloooo/a//////////// 2001. 08361
Yueyue Liu, Hongyu Zhang, Yuantian Miao, Van-Hoang Le, and Zhiqiang Li. Optllm: Optimal assignment of queries to large language models. In 2024 IEEE International Conference on Web Services (ICWS), pp. 788-798. IEEE, 2024.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 1964-1974, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 10.18653/v 11116553/1/2224.nam 1/201//4.22l2llloom/// 109, https: naacl long. 109/
Taaclanthology.org
Alireza Mohammadshahi,, Arshad Rafiq Shaikh, and Majid Yazdani. Routoo: Learning to route to large language models effectively. arXiv preprint arXiv:2401. 130.9999..3999, 2024.
Quang H Nguyen, Thinh Dao, Duy C Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V Chawla, and Khoa D Doan. Metallm: A high-performant and cost-efficient dynamic framework for wrapping 1lms. arXiv preprint ar arXiv:2407. 10834, 2024.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and lon Stoica. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406. 18665, 2024.
Zhihong Pan, Kai Zhang, Yuze Zhao, and Yupeng Han. Route to reason: Adaptive routing for I1m and reasoning strategy selection. arXiv preprint arXiv: 19435, 2025.

---

Published as a conference paper at ICLR 2026
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tus Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 18618886611222) www.2/l...aaal---../. lnancl-maiin 168. URL attps://ccaanthoooyy.ooooo wwt.oo/g////2 nwacl-aclaaaaa ma iomain... 168/
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. / met/foruu Sidiunndd--i-5581111
Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, pp. 606-615. ACM, March 2024. doi: 10.1145/
3616855.3635825. URL /ttttt////> wwt.:doi..rrggg.. 10. 005......00..111188886888.003330 3635825
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. Mixllm: Dynamic routing mixed large language models. arXiv preprint arXiv:2502. 18482, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ich Tsujii (eds.), Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2369- 2380, Brussels, Belgium, doi: 10. (//65////DD8--.. 1/D18-1159,
2018. Association for Computational Linguistics. logyy org/ 1259/
Yizhang Zhu, Runzhi Jiang, Boyan Li, Nan Tang, and Yuyu Luo. Elliesql: Cost-cfficient text-to-sql with complexity routing. arXiv preprint arXiv:2503.224000 2025.
Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchan- dran. Embedllm: Learning compact representations of large language models. arXiv preprint arXiv:2410.02223 2024.
12

---

Published as a conference paper at ICLR 2026
A
DESCRIPTIONS FOR COMPUTATION STRATEGY
Table 8 lists the description for decoding strategies and Table lists the descriptions for models.
Table 8: Descriptions decoding strategies.
Strategy ID
Description
Van lla
Vanilla prompting retains the original question content without adding any additional prompt information.
CoT
Chain-of-Thin Thouuhhh (CoT) prompting guides the model to articulate step-by-step reasoning process before providing the final answer. This results in longer responses and slower inference, but delivers superior per- formance complex reasoning tasks.
Table Descriptions of models.
Model ID
Description
5--1.. Instruct
(ree2..5-.5SIIIstrree is an ultra-lightweight 1.5 B parameter model designed for minimal-resource environments. is best suited for simple prompts, basic classification, and short text completion, but struggles with nuanced understanding or ad- vanced reasoning tasks.
Owen2.5-3B 5-33--IIttttIturt
Qwen2.5-3B --stuuuntt is a lightweight 3 B parameter model with fast inference and low resource usage. suitable for simple tasks such as basic question answering and short-form text gener- ation, but is limited in handling complex reasoning or multi-step tasks.
Qwen2.5-77 1en22-5-BB-nntruuc Instruct
OWwnn..5-7B-Instruu is a mid-small B parameter model that balances speed and performance. It can handle multi-turn dia- logue, basic code and math tasks, and offers improved language understanding smaller models while maintaining efficient inference.
Allam-- 1mma-33-33R-Tstmnmee
Llama-3.2-33 Bi- is compact 3 B parameter model opti- mised for efficient inference in constrained environments. It han- dles basic instruction following, simple question answering, and short text generation reliably, but lacks the depth for nuanced rea- soning or complex task execution.
Llama-3.
Llama-3. 1-8B-Instruct is a moderately-sized model that offers a strong balance between performance and re- source usage. It supports multi-turn dialogue, intermediate rea- soning, and modest code or math capabilitics, though it may still struggle with deeply intricate or highly technical prompts.
B parameter
13

---

Published as a conference paper at ICLR 2026
B DYNAMIC PROGRAMMING FORMULATION FOR SOLVING CONSTRAINED OPTIMIZATION
To address the constrained optimization in Equation formulate dynamic programming (DP) based solutions. We define ( as the maximum achievable total accuracy when routing the first tasks, subject total cost exceeding b.
Then, we initialize the DP problem as follows
0
i 0
DP
vb [0,
(3)
1-x
otherwise
We define the recurrence relation as follows: for all b E [Bmin, min (Bruas nB)
DP
max DPli 1][b c,Jl + agj
(4)
3.b243
where B = Ei L minj Ckj and B 2k max, Ckj- B and B denote the minimum and maximum total cost required to assign exactly one method to each of the first i tasks. We constrain b within these bounds avoid unnecessary computations. This recurrence reflects the process of updating the maximum cumulative accuracy by considering all computation strategies for task and choosing the one that achieves the highest accuracy without exceeding the budget. Since the budget nB and the cost Cjj can be floating-point numbers, we round them to integers to enable integer-based indexing in the DP array.
Then, the maximum accuracy attainable within the budget nB is maxsçaB DP[n][6]. We apply backtracking to recover the strategy chosen for each task.
C PROMPTS
Tables 10 show the prompts used for all task types (multi-hop QA, general reasoning, and math problems) with different decoding strategies.
Table 10: Prompts for multi-hop QA (blue refers to the vanilla prompt and yellow refers to the CoT prompt).
2WikiMultiHoo and HotpotQA (vanilla & CoT)
System:
You are expert at question answering.
User:
You provided with user question, and information that might be relevant to the user question. Your task is to only output short answer within <ans></ans>
You are provided with a user question, and information that might be relevant to the user question. Please reason step by step before providing the short answer; put your final answer within <ans>< < ans>.
Document title: Mistress (1992 film)
Document content: Robert De Niro is the producer of the film Mistress.
Document title: The Godfather Part II
Document content: Robert De Niro played the role of Vito Corleone in The Godfather Part II.
Here is the user question: In The Godfather Part II, who did the producer of Mistress play?
14

---

Published as a conference paper at ICLR 2026
Table 11: Prompts for general reasoning.
MMLU and GPQA (vanilla & CoT)
System: You are an expert at question answering.
User:
You are provided with a multi-choice question. Your task is to only output an an- (the letter corresponding the answer choice placed inside parentheses) within <ans></ans> (e.g. <ans> (A) </ans>).
You are provided with a multi-choice question. Please reason step by step before providing the final answer, and put your final answer (the letter corresponding to the answer choice placed inside parentheses) within <ans></ans>.
Here is the user question:
Which the following a second messenger that stimulates release of calcium ions into the cytoplasm?
Here are the multiple-choice answers:
(A) Prostaglandins
(B) Inositol triphosphate
(C) Cyclic AMP
(D) Calmodulin
Table 12: Prompts for math problems.
GSM8K and SVAMP (vanilla & CoT)
System:
You are expert at solving math questions.
User:
You are provided with math question. Your task is to only output numerical answer within <ans></ans>,
You are provided with math question. Please step by step before providing numerical answer; put your final answer within <ans></ans>>
Here is the user question:
Tommy fundraising for his charity by selling brownies for $3 slice and cheesecakes for $4 a slice. If Tommy sells 43 brownies and 23 slices of cheesecake, how much money does Tommy raise?
15

---

Published as a conference paper at ICLR 2026
Average parformance In-distribution datasets
Average parformance out-al-diatribution datasets
Accurscy
urcecum
Colicie fund
Calrocte tound
Luarooi menmaa cat
Lianbiis cat
Luarg rntrast cat
ceentin instral
Ovend inttaa, well's
Ceent 1-5 T.Elitttrrs EXT
Ovent Ai.rttttt
13.%
23.5%
(x101t)
Figure 5: Performance of all methods for unconstrained routing on both in- and out-of-distribution datasets across various values of w defined in Section 2.2 illustrating the trade-off between accuracy and cost.
Average parformance large
only
Average performance both large
21.66
71.2%
13.%%
$3.5%
Accurac
Accuracy
Readt
13.5%
52.5%
035%
13.2%
Ented
broea Mevi
Guerd Sclieaaau yell
115%
Lape 1.18 inttuh
33.2%
Quend 184 seturs viil
12.%%
26
23.0%
PLOPs (x10!t)
PLOPS (x101))
Figure 6: Performance of all methods under the continual setting with different collections of strate- gies. Xyy denotes method X trained from scratch. XyTrs% denotes method X fine-tuned from its prior version, which was trained from scratch in Setting 1, using Y% of the new data.
D IMPLEMENTATION DETAILS
We used the fff-the-shelf ALL-MPNET BASE-V? v2 model as the frozen encoder outlined in Sec- tion 2. following the approach Pan al. (2025); Zhuang et al. (2024), which generates repre- sentations of size k 768. Training was conducted on an A100 GPU cluster for up to 100 epochs, using the Adam optimizer with a batch size 32 and an initial learning rate x 10 10-3
E FULL DIAGRAMS
Figures and present the full versions of Figures and respectively, including all methods.
F THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLM was used only to aid writing quality (proofreading and polishing grammar). No ideas, claims, methods, results, references are generated by LLMs. All content decisions and revisions are made by the authors.
-mp.et-teetteeeee -ase-v2
16


Reviews:
[{"review_id": "79JpEJgCew", "reviewer": "Reviewer_d5bV", "summary": "This paper introduces a routing method to choose the best model, maximizing efficiency and accuracy, for a continual learning setup, where the task based routing predictor can be updated to include new model variations (strategies) on the fly.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a timely problem of efficiently routing between a growing variety of models to solve a task with maximum accuracy and efficiency (measured in FLOPs).\n2. The proposed method of learning a model specific predictor without retraining the strategy prediction backbone makes routing to new models seamless as also verified by the experiments in the paper.", "weaknesses": "1. The authors use FLOPs to measure and learn the efficiency while choosing between models. Are the FLOPs measured by counting the number of multiply-add operations? If that is the case, the proposed method will not capture the efficiency achieved by quantized models, which can be much more efficient and still achieve similar accuracies (most models are now released with lightweight quantized version).\n2. It is unclear to me, how the base strategy predictor performs when the number of input strategies increase, i.e, what is the expressivity of this predictor and does it overfit to initial learnt strategies? \n3. Maybe this is a more high level question, but how would routing work in the case where the questions (tasks) are open ended and there does not exist a right answer to measure accuracy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vch7fItjjO", "reviewer": "Reviewer_NPzC", "summary": "This paper proposes CONCUR, a continual routing framework designed to address limitations in existing AI task routing systems, supporting both constrained (with cost budget) and unconstrained (no budget) routing scenarios. CONCUR outperforms baselines in both unconstrained and constrained routing.", "questions": "1.\tHow does replacing the ALL-MPNET-BASE-v2 model with other embedding models affect the accuracy and cost prediction of CONCUR? Is there a minimum embedding quality required for the framework to perform well?\n\n2.\tWhen integrating a completely new strategy with no historical task data (cold start), how does CONCUR perform? Do you have any initialization methods (e.g., transferring knowledge from similar strategies) to improve the cold-start prediction accuracy of new predictors?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe modular predictor design directly solves the retraining overhead issue of existing methods. Integrating new strategies with minimal cost aligns with real-world needs (e.g., frequent updates of language models), making it highly scalable.\n\n2.\tBy combining general-purpose and task-specific representations, CONCUR captures both universal patterns (via general embeddings) and task/strategy-specific nuances (via learnable representations), which is empirically shown to improve routing decisions (e.g., higher accuracy on out-of-distribution tasks).\n\n3.\tSupporting both constrained and unconstrained scenarios broadens its applicability—unconstrained for use cases prioritizing performance balance, and constrained for resource-limited environments (e.g., edge devices with tight FLOP budgets).\n\n4.\tThe study uses diverse datasets (covering different task types and distribution shifts), standardized metrics (accuracy + FLOPs), and comparisons to strong baselines, ensuring the validity and generalizability of results.", "weaknesses": "1.\tThe general-purpose representation relies on a frozen pre-trained embedding model (ALL-MPNET-BASE-v2). The paper does not evaluate how changes to this model (e.g., using a smaller/larger embedding model) affect predictor performance, raising concerns about its robustness to embedding choice.\n2.\tThe unconstrained routing uses a weight to balance accuracy and cost, but the paper does not analyze how different weight values (e.g., weight=0.1 for cost prioritization vs. weight=0.9 for accuracy prioritization) impact real-world utility, nor does it provide guidance on choosing this weight for specific use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CBGEtBOL6F", "reviewer": "Reviewer_dMBg", "summary": "This paper introduces a unified framework for routing among multiple LLMs and decoding strategies under both continual and non-continual settings. The main idea is to train modular predictors that estimate, for each model–decoding pair, its expected accuracy and computational cost on a given task. These predictors combine general text embeddings (from an off-the-shelf embedding model) with task-specific representations to capture both semantic and domain-specific information. Routing decisions are then formulated as an optimization problem that can be solved efficiently. Experimental results show that the proposed framework outperforms existing baselines in terms of both accuracy and efficiency.", "questions": "See the weakness above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses an important and practical problem. Effectively routing among multiple LLMs and decoding strategies under varying cost and performance constraints has become an increasingly relevant and widely discussed topic. This topic is relevant to the growing need for efficient deployment of large reasoning models, especially in settings where computational budgets and task distributions evolve over time. The formulation of both continual and non-continual routing scenarios adds further practical value to the work.\n\n- The proposed framework demonstrates consistent improvements over existing routing baselines across several reasoning and question-answering benchmarks. The experimental results suggest that the modular predictor design and optimization-based decision process lead to better accuracy–efficiency trade-offs, validating the effectiveness of the proposed approach in diverse routing settings.", "weaknesses": "- Although the paper repeatedly discusses routing in both continual and non-continual settings, the problem itself is never formally defined. It remains unclear what exactly constitutes a routing input and output, and how the optimization objective is formulated in mathematical terms. The framework is presented primarily at a conceptual level, which makes it difficult to precisely understand the problem scope and to reproduce the approach in other contexts. As a result, the paper may not be self-contained for readers who are not already familiar with prior routing works such as RouteLLM or RTR, which substantially limits accessibility and clarity.\n\n- The overall framework follows a structure that is already common in related areas such as task planning and model selection. The paper mainly adapts existing ideas to the LLM routing setting, without introducing a fundamentally new algorithmic or theoretical insight. \n\n- The paper lacks a systematic ablation study quantifying the contribution of each component in the proposed framework. Is there any ablation study or experimental analysis addressing this? For example, how would the performance change if only the general-purpose representation or the task-specific representation were used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Qy7RA0WxN", "reviewer": "Reviewer_DE3V", "summary": "This paper introduces CONCUR, a modular framework for continual task routing among diverse computation strategies, addressing the challenge of efficiently mapping AI tasks to optimal computational configurations under both constrained (budget-aware) and unconstrained conditions. Unlike traditional monolithic routing approaches that require retraining a single model whenever new strategies are added, CONCUR proposes a modular design in which each computation strategy has its own predictor model estimating its accuracy and cost on a given task. These predictors rely on both general-purpose and task-specific representations, improving the granularity and flexibility of routing decisions.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The modular predictor design allows new strategies to be added by training only their respective predictors, greatly reducing retraining cost and time compared to monolithic routers.\n2. Combines general-purpose embeddings (from pretrained encoders) with learnable task- and strategy-specific representations, demonstrating superior expressiveness over single-representation baselines.\n3. Comprehensive evaluation and analysis.", "weaknesses": "1. While modularity is beneficial, training two predictors (accuracy + cost) per strategy may become cumbersome when the number of strategies grows very large.\n2. The paper lacks a deeper theoretical justification for why modular predictors generalize better than joint models and the optimization formulations are straightforward but not novel in theory.\n3. The predictors act as black boxes; it remains unclear how they capture task–strategy interactions or why they perform well in OOD tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)