RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models

Authors: shuhao  chen
weisen  jiang
baijiong  lin
james t kwok
yu  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:26:18

Abstract: Recent works show that assembling multiple off-the-shelf large language models (LLMs) can harness their complementary abilities. To achieve this, routing is a promising method, which learns a router to select the most suitable LLM for each query. However, existing routing models are ineffective when multiple LLMs perform well for a query. To address this problem, in this paper, we propose a method called query-based Router by Dual Contrastive learning (RouterDC). The RouterDC model, which consists of an encoder and LLM embeddings, is trained by two proposed contrastive losses (sample-LLM and sample-sample losses). Experimental results show that RouterDC is effective in assembling LLMs and largely outperforms individual top-performing LLMs as well as existing routing methods on both in-distribution (+2.76\%) and out-of-distribution (+1.90\%) tasks. The source code is available at https://github.com/shuhao02/RouterDC.

Main paper: 
 introduction: 
 Large language models (LLMs) have demonstrated proficient capabilities across various tasks. Many LLMs are publicly available online, such as Mistral [24] , LLaMA-2 [46] , and LLaMA-3 [45] . Those LLMs have been further fine-tuned to be generalists or specialists. For example, MetaMath [54] excels in solving mathematical reasoning problems. Since those LLMs are pre-trained or fine-tuned with various data, they typically exhibit varying strengths and weaknesses across different tasks [ 25, 33] . Therefore, assembling multiple off-the-shelf LLMs can harness their complementary abilities, resulting in better performance than relying on a single LLM.LLM ensembling is a straightforward method to assemble LLMs, which feeds the query to all candidate LLMs and merges all outputs into a final answer by majority voting [31, 51] or pairwise ranking [25] . However, ensembling is computationally prohibitive as it requires generating outputs with all candidate LLMs during inference. To tackle this issue, recent works [43, 33, 44] propose to learn a router to select a suitable LLM for each query. During inference, routing is much more efficient than ensembling as it only needs to perform inference on the selected LLM.The current state-of-the-art routing method is ZOOTER [33] . To train a router, ZOOTER scores the outputs of candidate LLMs as the supervision signal via an off-the-shelf reward model and then learns the router by minimizing the Kullback-Leibler divergence [29] between the selection probability from the router and the softmax normalized score. However, this loss is inappropriate when multiple LLMs perform well for a query. Figure 3 shows the scores of seven LLMs for an example query, where the top three LLMs have significantly higher scores than the bottom three LLMs. After the softmax normalization, the scores are small, leading the router to generate small probabilities on the top LLMs. Moreover, the normalized score tends to be uniform, which is not a strong supervision signal for learning the router. Figure 4 shows that the score difference between the top two LLMs is usually tiny (under the experimental setting in Section 4.1), indicating that the loss used in ZOOTER is inappropriate.In this paper, we propose a query-based Router by Dual Contrastive learning (RouterDC). The RouterDC consists of an encoder, whose architecture is a small language model, and learnable LLM embeddings for candidate LLMs. For each query, we first score the candidate LLMs by comparing their predictions with the gold label. Instead of directly aligning the score distribution, we leverage the score to choose the top-performing and bottom-performing LLMs and then propose a sample-LLM contrastive loss to pull the query embedding (extracted by the encoder) close to the embeddings of top LLMs while pushing far away from the embeddings of bottom LLMs. Based on this loss, our RouterDC could equally select one of top-performing LLMs for a query and hence alleviate the shortcoming of ZOOTER introduced previously. We empirically observe that training the router using the sample-LLM contrastive loss alone is not stable as similar queries can have dissimilar embeddings and be assigned to different LLMs. To improve the training stability, we cluster all the training queries into multiple groups and design a sample-sample contrastive loss to maximize the similarity between queries in the same group while minimizing the similarity between queries from different groups.We conduct experiments on challenging reasoning tasks (language understanding, code generation, and mathematical reasoning) to evaluate the proposed RouterDC in both in-distribution and out-ofdistribution settings. Empirical results show that RouterDC can harness the complementary potentials of LLMs, achieving state-of-the-art performance. Moreover, RouterDC outperforms existing routing methods by a large margin, showing that the proposed two contrastive losses are more beneficial for training the RouterDC.Our contributions are summarized as follows. (i) We propose a novel framework to learn a router to select the suitable LLM for each query by dual contrastive learning, which consists of sample-LLM and sample-sample contrastive losses; (ii) The proposed RouterDC is parameter-efficient (with fewer than 100M parameters) and computation-efficient (without backpropagating the gradients through LLMs) in training. Moreover, RouterDC is also efficient in inference (6× faster than Voting) as it only requires computation cost for the selected LLM and negligible cost for the router;(iii) Experimental results show that RouterDC effectively assembles LLMs and outperforms individual top-performing LLMs as well as existing routing methods on both in-distribution (+2.76%) and out-of-distribution (+1.90%) tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See Abstract and last two paragraphs in Introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Large Language Models (LLMs). LLMs have achieved great success in natural language processing and many foundation models have been released online [24, 46, 45, 27] . Many prior works [54, 47, 58, 9, 8, 41, 20, 34, 52] focus on fine-tuning those foundation models to obtain specialized LLMs for solving versatile tasks, for example, language understanding [58, 20, 26] , code generation [41, 8] , and mathematical reasoning [54, 34] . In this paper, we study the problem of assembling LLMs to harness their strengths by a router.LLM Ensembling. The goal of LLM ensembling is to leverage multiple LLMs to boost performance compared with a single model across various downstream tasks. Voting [31, 51] is a simple but effective ensemble method. Jiang et al. [25] further propose PairRanker and GenFuser to generate an improved output from the outputs of all LLMs, which needs to call LLMs O(T 2 ) times with T as the number of LLMs. LLM cascading [2, 13, 38, 55] query a list of LLMs (whose capacity depends on the model size) sequentially until an LLM's output is satisfied (i.e., having a significantly high confidence score), which is returned as the final output. Fusion of Experts [50] concatenates all LLMs outputs to build the final output and casts it as a supervised learning problem. Unlike the aforementioned ensembling methods which require querying the LLMs at least O(T ) times in inference, our RouterDC is much more efficient as it only needs to call the selected LLM once.LLM Routing. LLM routing aims to select the most suitable model for a query without calling all LLMs. Many works have been proposed to design an effective routing strategy. Shnitzer et al. [43] propose a collection of binary classifiers to evaluate the correctness of each LLM. Lu et al. [33] propose ZOOTER to align a router with the supervision from the reward model. LoraRetriever [56] propose a task-wise router to select the LLM by predicting the task identity of the query. Srivatsa et al. [44] explore the routing ability using both classifier-based and clustering-based approaches. The aforementioned methods neglect the fact that multiple LLMs may be well-suited to answer a single query. Ding et al. [10] design a cost-effective router for two LLMs (a small LLM and a large one). In contrast, the proposed RouterDC can be used for multiple LLMs simultaneously.Contrastive Learning. Contrastive learning learns effective representations by distinguishing between similar and dissimilar pairs of data points. It has been widely used in various tasks, such as visual representation learning [4, 14] , sentence representation leaning [12, 53, 42] , and visionlanguage alignment [39, 59] . In this paper, we propose two contrastive losses to learn the RouterDC for assembling LLMs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See the Limitations section in Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we propose RouterDC, a framework for learning a query-based router to assemble LLMs. An overview is illustrated in Figure 1 . We introduce the problem of router learning in Section 3.1 and design a scoring method to measure the performance of LLMs on each training query (Section 3.2). Next, we propose two contrastive losses to train the router, including a sample-LLM contrastive loss for learning the routing strategy (Section 3.3) and a sample-sample contrastive loss for improving training stability (Section 3.4). The training and inference procedures are provided in Algorithm 1.Consider a set of LLMs {M t : t = 1, . . . , T } and a training set D train = {(x i , y i ) : i = 1, . . . , n}, where x i is a query (i.e., question) and y i is its answer (i.e., ground truth). Usually, no single LLM is universally suitable for all queries in D train . Moreover, LLMs are diverse and have different architectures (e.g., Mistral-based [24] , LLaMA-based [45] ), making it infeasible to merge all LLMs into a single model [36, 22, 28] . In this paper, we study the problem of assembling LLMs by learning a router to select the suitable LLM for each query. The router takes x as input and produces the probability distribution of T LLMs being selected. As training and testing queries may come from different data distributions, the learned router is expected to generalize well on both in-distribution and out-of-distribution scenarios.To learn the router, we need to design a scoring method to assess the performance of LLMs on queries. For an open-ended generation query x i (requiring a long answer, e.g., GSM8K [7] , with an example shown in Example 1), one can directly compare the ground truth y i with the output of the LLM ŷ(t) i = M t (x i ) generated by greedy decoding. Though greedy decoding is simple and efficient, its inherent shortsightedness often prevents it from discovering the optimal solution. Conversely, sampling, like beam sampling [49] , is an advanced approach that is widely used in practice as it explores multiple alternatives in the search space, potentially leading to better results. We repeatedly feed the query x i to the LLM M t M times to obtain outputs {ŷ(t)i,m : m = 1, . . . , M }. Then, we define the score of LLM M t on the query x i as:EQUATIONwhere evaluate(ŷ, y) gives 1 if the prediction ŷ is correct otherwise 0.For a multiple-choice question x i with an option set A i (e.g., MMLU [17] , as an example shown in Example 1), sampling is unnecessary as we can simply define the score based on the probability of options, i.e., sEQUATIONi |xi)EQUATIONwhere P Mt (a|x i ) is the probability of option a predicted by the LLM M t . According to Eq. ( 2), when the LLM M t outputs a correct option (i.e., ŷ(t) i = y i ), we normalize the probability to make it comparable across different LLMs, which will be used in Section 3.3; When the LLM M t generates a wrong option, si is set to 0 to punish M t for x i . Based on the scores {s (t) i : t = 1, . . . , T }, we introduce a sample-LLM contrastive loss in the next section.An open-ended question from GSM8K [7] : Question: Tim has 30 less apples than Martha, and Harry has half as many apples as Tim. If Martha has 68 apples, how many apples does Harry have? Answer: Tim has 68-30 = 68-30=38 apples. Harry has 38/2 = 38/2=19 apples. #### 19 A multiple-choice question from MMLU [17] : Question: An object is placed 100cm from a plane mirror. How far is the image from the object? Options: A. 50cm B. 100cm C. 200cm D. 300cm Answer: CAs illustrated in Figure 1 , the proposed RouterDC consists of an encoder E(x; w) parameterized by w (where in our experiments E(x; w) uses a small language model mDeBERTaV3-base [15] ) to map x into an embedding in R p , and T learnable LLM embeddings {k t ∈ R p : t = 1, . . . , T } for the T LLMs. For a query x i , the RouterDC generates a selection probability distribution over T LLMs asEQUATIONwhere θ ≡ {w, k 1 , k 2 , . . . , k T } denotes the set of the parameters in RouterDC, sim(•, •) denotes the cosine similarity, and softmax(•) denotes the softmax normalization.One can train the router by minimizing the distance between the output of the router and a score distribution over {s ŷ′ = M t ′ (x ′ ).where KL(•, •) is the Kullback-Leibler divergence [29] . This KL loss is recently used in [33] for LLM routing, but we argue that it may not be a good proxy for training the router since the goal of the router is to assign queries to top-performing LLMs instead of aligning the scores with R(x i ; θ), particularly for the bottom-performing LLMs.We draw inspiration from contrastive learning [37, 23] and propose a sample-LLM contrastive loss to learn the router. For a query x i , we construct its positive LLMs index set I + i and its negative LLMs index set I - i based on the scores {s (t)i : t = 1, . . . , T } as: I + i consists of the indices of LLMs corresponding to the top-K + scores, while I - i consists of the indices of LLMs corresponding to the bottom-K -scores with s (t) i < 0.5. Note that K + can be larger than 1 (K + = 3 in our experiments) as there can be multiple LLMs that are suitable for a query in practice. We expect the router to pull the query embedding E(x i ; w) closer to the positive LLMs' embeddings {k t+ : t + ∈ I + i } while pushing apart from the negative LLMs' embeddings {k t-: t -∈ I - i }. To this end, we propose the sample-LLM contrastive loss asEQUATIONWe empirically find that training the router by minimizing the sample-LLM contrastive loss alone is not stable (refer to Figure 12 in Section 4.4). The reason is that some similar queries can have dissimilar embeddings and may be routed to different LLMs. To improve the robustness of the router, we introduce a sample-sample contrastive loss to encourage the encoder to generate similar embeddings for similar queries.First, we cluster queries into multiple groups by unsupervised clustering. Specifically, we extract the embeddings of all training queries using a pre-trained encoder (i.e., mDeBERTaV3-base [15] ) and transform them into low-dimensional vectors by the t-SNE algorithm [48] . Then the k-means clustering algorithm [35] is used to cluster these low-dimensional vectors into N groups {K 1 , . . . , K N }.Next, we construct a sample-sample contrastive loss to encourage samples in the same group to have similar embeddings. Specifically, for a query x i ∈ K j , we randomly select an in-group query x + i ∈ K j and an out-group set X - i ⊂ {∪ j ′ ̸ =j K j ′ } of H queries from the training mini-batch at each iteration. Similar to the sample-LLM contrastive loss, we propose a sample-sample contrastive loss to pull the embedding of x i closer to the embedding of x + i while pushing it away from the embedding of queries in X - i . Formally, the sample-sample contrastive loss is formulated asL sample-sample (x i ; θ) = -log e sim(E(xi;w),E(x + i ;w)) e sim(E(xi;w),E(x + i ;w)) + x - i ∈X - i e sim(E(xi;w),E(x - i ;w)).(5)Training. We learn a router R(x; θ) by minimizing the final objective consisting of sample-LLM and sample-sample contrastive losses, i.e.,EQUATIONwhere λ > 0 is a hyper-parameter. In our experiments, λ is set to 1.RouterDC contains fewer than 100M parameters (that is, the encoder model E(x; w) is small and the number of parameters in the LLM embeddings {k 1 , . . . , k T } is negligible), thus it is parameterefficient. Moreover, training the router is computationally efficient as it does not require backpropagating the gradients through the LLMs.Inference. During inference, for each testing query x ′ , we compute R(x ′ ; θ) and select the LLM with the largest probability, i.e., t ′ = arg max t∈{1,...,T } sim(E(x ′ ; w), k t ). Then we generate the prediction as ŷ′ = M t ′ (x ′ ).Compared with existing LLM assembling methods like voting [31] and cascade [2] , which may call LLMs multiple times for a query, RouterDC is much more efficient as it only needs to call the selected LLM once.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: See abstract.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup: 
 Candidate LLMs. We choose seven open-source LLMs from HuggingFace1 : (i) Mistral-7B [24] is a general LLM released by the Mistral-AI team; (ii) MetaMath-Mistral-7B [54] is fine-tuned on the MetaMathQA dataset [54] ; (iii) zephyr-7b-beta [47] is an aligned version of Mistral-7B using direct preference optimization [40] on a mix of publicly available, synthetic datasets; (iv) Chinese-Mistral-7B [58] expands the vocabulary and incrementally pre-trains Mistral-7B on Chinese corpus; (v) dolphin-2.6-mistral-7b Datasets. We evaluate in various tasks: (i) MMLU [17] is a general benchmark that covers 57 subjects; (ii) GSM8K [7] is a mathematical benchmark with diverse grade school questions;(iii) CMMLU [30] is a comprehensive Chinese benchmark that covers 67 subjects ranging from basic to advanced professional levels; (iv) ARC-C [6] is a reasoning benchmark containing different grade-school level questions; and (v) HumanEval [3] is a code completion benchmark consisting of programming problems assessing language comprehension, algorithms, and simple mathematics. For GSM8K, we use its default training and testing split. As the rest tasks do not have a default split, we randomly split the dataset into training (70%) and testing (30%) sets. All the training sets are unioned together to form the total training set D train for learning the router. The learned router is then evaluated on the testing set of in-distribution tasks.We also evaluate the trained router on three out-of-distribution (OOD) tasks: (i) PreAlgebra [18] , which consists of basic university-level algebra problems; (ii) MBPP [1] , which is a code benchmark that consists of 1,000 crowd-sourced Python programming problems; and (iii) C-EVAL [21] , which is a comprehensive Chinese evaluation benchmark spanning 52 diverse disciplines and four difficulty levels.Baselines. We compare RouterDC with the following baselines: (i) CosineClassifier, which treats the routing problem as a multi-class classification (the top-1 LLM is the label) and trains a cosine classifier on outputs of the encoder. CosineClassifier is equivalent to RouterDC with K + = 1, K -= T -1, and λ = 0; (ii) Voting [31] , which feeds the query to all LLMs and chooses the final prediction by majority voting; (iii) ZOOTER [33] , which trains a router by supervised learning using rewards obtained by the scoring method in Section 3.2; (iv) LoraRetriever [56] designs a routing strategy based on task identities, which are unavailable in practice and we replace them with the cluster indices obtained by the clustering method in Section 3.4.Implementation Details. Following [5] , we use the Language Model Evaluation Harness package [11] for evaluation. For open-ended generation questions, we query LLMs M = 10 times by employing beam sampling with a temperature of 0.2 to calculate the score. For the router, we adopt mDeBERTaV3-base [16] as the encoder E(x; w), which is a small language model containing only 86M parameters. The dimension of each LLM embedding is set to 768. The hyper-parameters K + , K -, H, and λ are set to 3, 3, 3, and 1, respectively. The number of clusters N is set to 5. The router is trained for 1000 steps using the AdamW [32] optimizer with a learning rate of 5 × 10 -5 , a weight decay of 0.01, and a mini-batch size of 64. All experiments are run on NVIDIA A100 80GB GPUs.In-Distribution Results. Table 1 shows the testing accuracies on five in-distribution tasks. As can be seen, RouterDC achieves the highest average accuracy, surpassing the best individual LLM (i.e., dolphin-2.9-llama3-8b) by a large margin of 3.98%. RouterDC achieves accuracy improvements over the top-performing individual model on three tasks, with an increase of +0.51% for GSM8K, +0.57% for ARC-C, and +1.63% for HumanEval. Compared with ZOOTER and CosineClassifier, RouterDC consistently performs better on all the tasks, demonstrating that the proposed dual contrastive losses can train a more effective router. Furthermore, RouterDC achieves an average accuracy improvement of +2.77% over LoraRetriever, validating the usefulness of the sample-LLM contrastive loss. Additionally, RouterDC, with only 28.3 minutes for training, outperforms Voting on four of five tasks and is about 6× faster in inference.Out-of-Distribution Results. Effects of λ. We conduct an experiment to study the effect of λ in Eq. ( 6) w.r.t. the testing accuracy. From Figure 5 (the detailed results are in Table 5 of Appendix A), we can see that using two contrastive losses together (i.e., λ = 1) achieves better overall performance than using the sample-LLM contrastive loss alone (i.e., λ = 0). Moreover, the overall performance of RouterDC is insensitive to a wide range of λ ∈ [0.5, 5], making it easy to choose the value of λ in practice.Effects of number of clusters N . We conduct an experiment to study the effect of the number of clusters (i.e., N ) used in the sample-sample contrastive loss w.r.t. the testing accuracy. From Figure 6 , we can find that RouterDC is insensitive to a wide range of N ∈ [4, 9] . Moreover, increasing N leads to higher average accuracy when N is small (≤ 4), but the accuracy saturates quickly.Effects of number of out-group queries H. Figure 7 shows the testing accuracy with H. When = 0, L sample-sample is constant, which means using L sample-LLM alone is not the best configuration. Moreover, the values of H ≥ 1 play a negligible influence on the average performance of RouterDC.Effects of K + and K -. To investigate the sensitivity of K + and K -, we conduct an experiment using the setting in Section 4.1. Figure 8 shows the average testing accuracy w.r.t. K + and K -with the in-distribution setting. As can be seen, for all the configurations, RouterDC outperforms the best individual LLM (i.e., 54.56% for dolphin-2.9-llama3-8b in Table 1 ). Note that among all the configurations, RouterDC (with K + = 1 and K -= 6) performs worse, showing that selecting only the top-1 LLM as positive and the other LLMs as negative is inappropriate for learning the router. do lp hi n-2. 9-lla m a3 -8 b 0.37 0.00 0.00 0.00 0.06 0.01 0.05 0.00 0.02 0.96 0.00 0.06 0.00 0.63 0.00 0.00 0.02 0.00 0.00 0.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.01 0.05 0.00 0.14 0.00 1.00 0.00 0.04 0.08 0.07 1.00 0.41 0.04 0.00 0.01 0.90 0.28 0.83 0.00 9 shows the distribution, which has a clear structure on both in-distribution and out-distribution tasks. For example, most GSM8K and PreAlgebra queries are assigned to MetaMath-Mistral-7B and dolphin-2.9-llama3-8b, which have strong mathematical ability (Tables 1 and 2 ). To further investigate the routing rule of RouterDC, we compute the average cosine similarity between LLMs and the query embeddings for each task. As shown in Figure 10 , the similarity matrix is roughly aligned with the assignment matrix in Figure 9 .M M L U G S M 8 K C M M L U A R C _ C H u m a n E vFor example, embeddings of GSM8K and PreAlgrebra queries are more similar to MetaMath-Mistral-7B and dolphin-2.9-llama3-8b than to other LLMs.Visualization of Training Queries. Figure 11 shows the t-SNE visualization [48] of the embeddings of training queries using a pre-trained encoder mDeBERTaV3-base [15] . As shown, except for HumanEval, all tasks have a clear clustering structure, confirming that using unsupervised clustering in Section 3.4 is reasonable.Effectiveness of L sample-sample . We conduct experiments to study the effectiveness of L sample-sample (Eq. ( 5)). Figure 12 shows the training and testing accuracy curves of RouterDC (with or without L sample-sample ) on GSM8K. As can be seen, the training curve of RouterDC (w/o L sample-sample ) exhibits considerable oscillation, whereas that of RouterDC is much more stable. ) with the similarity between these two example queries increases to 0.9982. Furthermore, RouterDC achieves higher testing accuracy than its counterpart, verifying the effectiveness of L sample-sample .Routing to Different Numbers of LLMs. We evaluate the performance of RouterDC when the number of LLMs increases. Figure 13 shows the testing accuracies on five in-distribution tasks. As can be seen, adding LLMs consistently enhances the average accuracy. Table 8 in Appendix A shows the detailed results and configurations.Robustness to LLM Losses during Inference. In a production environment, the loss of model servers is sometimes unavoidable due to various reasons such as network problems, thus placing crucial requirements on the robustness of the router. We conduct an experiment to validate the robustness of RouterDC by removing an LLM during inference. Table 3 shows the testing accuracies on five in-distribution tasks. We can see that RouterDC reliably withstands the loss of any single LLM. The robustness is attributed to the fact that multiple LLMs (with top scores) are chosen as positive labels in the sample-LLM contrastive loss, and they can be regarded as each other's backup.Cost-Effectiveness. As cost is an important metric to evaluate LLMs, following [19] , we conduct experiments on two tasks (i.e., GSM8K and MBPP) to consider the LLM costs. We modify the score s(t) i to s (t) i + c (t) i , where c (t)i is the cost of query x i using the tth LLM. As can be seen from Figure 14 , RouterDC is more cost-effective than CosineClassifier and ZOOTER on both tasks. Discussions on Availability of Task Identity. In Section 3.4, we cluster samples into N groups and apply the sample-sample contrastive training to encourage similar queries with similar embeddings. However, when the task identity is available in the training dataset, the samples can be naturally grouped into different tasks. To explore the performance of RouterDC with additional task identity, we replace the {K 1 , . . . , K N } with the groups of different tasks and conduct experiments on five in-distribution tasks. Table 4 shows the testing accuracy comparison between RouterDC and its variant. As can be seen, RouterDC is comparable to RouterDC (w/ task identity), showing the effectiveness of the unsupervised clustering. To further investigate the sensitivity of the number of training samples used RouterDC, we conduct an experiment to study the performance of RouterDC with different numbers of training samples per task. As can be seen from Figure 16 , the testing accuracy saturates quickly, indicating that a small number of samples is sufficient for learning the router (e.g., 100 samples per task). Moreover, with only 30 samples per task, RouterDC already outperforms the previous SOTA overall (57.21 vs 55.77), demonstrating that our RouterDC does not require a large amount of labeled data for training. To validate that RouterDC is a queried-based router rather than a task-wise router, we conduct an experiment in a single task setting, i.e., we train the router on the training set of HumanEval and evaluate it on the testing set. The single task setting is an edge case where all queries may contain the same task information. Hence, the router needs to learn how to route queries appropriately based on the query itself instead of some possible task information contained in the query. Table 9 reports the testing accuracy. As can be seen, RouterDC largely outperforms the best candidate LLM (i.e., dolphin-2.9-llama3-8b) and existing routing methods, demonstrating that the router can select appropriate LLMs for queries based on query characteristics.To further explore the generalization ability of RouterDC, we evaluate the learned router on one more OOD task: JavaScript [57] , which aims to generate JavaScript code to solve problems. Different from HumanEval, which generates Python code to solve problems, JavaScript can be viewed as a distant OOD task. Table 10 reports the testing accuracy. As can be seen, RouterDC outperforms existing routing methods by a large margin, demonstrating that our RouterDC is more effective in routing queries of the distant OOD task. We conduct experiments to study whether the proposed sample-sample contrastive loss is useful for ZOOTER. Table 11 and Table 12 shows the testing accuracy for the ID and OOD scenarios. As can be seen, integrating L sample-sample into ZOOTER leads to improvements of +1.52% and +0.81% for ID and OOD, respectively, demonstrating that the proposed sample-sample contrastive loss is beneficial for ZOOTER. As mentioned in Section 3.2, we set s (t) i = 0 when the LLM M t generates a wrong option for the multiple-choice question x i . We perform an experiment to verify the effectiveness of such a design. Table 13 shows the testing accuracy on five in-distribution tasks. As can be seen, punishing s (t) i performs better on average. Due to the limited computational resources, we only evaluate RouterDC with candidate LLMs that have relatively small numbers of parameters (i.e., 8B for LLaMA-based LLMs and 7B for Mistralbased LLMs). However, there are many LLMs with more parameters and stronger capabilities available for public use (e.g., LLaMA-2-70B [46] and Mistral-8x7B [24] ), making it reasonable to apply the RouterDC to these more capable but expensive models.Moreover, though RouterDC is designed as a query-based router, the framework can be extended to the chat context, e.g., selecting LLMs based on the recent conversation.We leave the investigation of such scenarios to future work.NeurIPS Paper Checklist 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The significant performance improvement is sufficient to justify our contributions.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Justification: There is no societal impact of this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Justification: There are no such risks in this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See the cited reference.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.https://huggingface.co/ 
 
 Appendix:  


Decision: Paper proposes RouterDC, a novel query-based routing model for assembling multiple LLMs using dual contrastive learning. While the paper has merits (see below), certain limitations (e.g., the added value of the proposed loss, the "generalizatibility" of the OOD generalization claims, discussion on cascade based methods) make it more suitable as a poster presentation.

The problem is quite relevant, addressing the challenege of effectively leveraging the diverse strengths of multiple LLMs which is becoming a prevalent approach both by the practitioners (with applications that are facing users) and the researchers; THe approach is novel, sample-LLM contrastive loss; results are strong with detailed discussions during rebuttals by authors providing comprehensive and convincing responses.

Also, further emphasizing the data efficiency -- given the concerns about the scalability and cost of training multiple LLMs -- could be considered as an improvement.

Reviews: review: paper_summary: paper_summary: This paper aims to improve the ability of LLMs by assembling them. The proposed method use a contrastive learning strategy. It uses a sample-LLM contrastive loss which pull the query embedding closer to the top-performed LLM embedding. It also employs a sample-sample contrastive loss, which learns the distribution of input with the help of clustering.
strength_weakness: strength_weakness: The contribution of this work lies in two part, first, it uses sample-llm contrastive loss, instead of directly learning the routing as a classification; second, it use sample-sample contrastive loss to make the training more stable. However, the relation between the two is missing. I am wondering how the two parts contribute to the final performance. Is it possible to use ZOOTER with the sample-sample loss? What do you expect as the result?

I think the response partially addressed my concern. So I raised my score to 5. I do understand there are hyper-parameters that one could control to affect the results, but are the proposed two contrastive losses the most important ones to be consider in this scenario? I would love to see more discussions in the paper.
questions: questions: See the weakness part.
limitations: limitations: N/A
paper_summary: paper_summary: This paper studies the problem of assembling off-the-shelf LLMs to harness their complementary strengths. The authors propose a novel query-based router by Dual Contrastive learning (RouterDC), i.e., a sample-LLM contrastive loss and a sample-sample contrastive loss. The former contrastive loss aims at training the router such that it can assign suitable LLMs for queries, while the latter is for training stability. Experiments on various challenging tasks demonstrate that RouterDC performs better than existing routing methods and individual top-performing LLMs in both in-distribution and out-distribution settings.
strength_weakness: strength_weakness: 1. In Line 157, the authors claim that “The reason is that some similar queries can have dissimilar embeddings and may be routed to different LLMs.” Evidence should be provided to support this claim.
2. For the OOD setting (Table 2), RouterDC fails to beat the best individual LLMs on all tasks (e.g., 38.81 vs 39.72 on PreAlgebra).
3. RouterDC may require a large amount of labeled data to train the router.
questions: questions: 1. How about the performance of RouterDC without the sample-sample contrastive loss?
2. In Fig. 9, it seems that no samples are routed to the Chinese-Mistral-7B model, why?
3. Can you visualize the embeddings of training samples extracted by the encoder $\mathcal{E}(x;w)$ of RouterDC?
limitations: limitations: The authors are encouraged to discuss limitations in the conclusion section.
paper_summary: paper_summary: The authors propose a routing between different LLMs, based on classification of embeddings from a fine-tuned transformer model (mDeBERTaV3-base), on a per-query basis.  To sharpen the classification, they chose positive and negative samples among the training tasks, and to stabilise training they add a loss term that encourages cohesive clusters.  The results suggest that the system is capable of routing appropriately.
strength_weakness: strength_weakness: * Since the LLMs in the pool appear to be pretty decisively good at some of the tasks (Figure 9), the meat of the router task is to see whether it's possible to classify these different tasks via a small LM and embeddings.  On its face, it doesn't seem too remarkable that this works.
* Sample-Sample Contrastive Loss : Smells like a post-experiment fix, rather than a principled choice
  + Particularly since a pretrained (frozen?) mDeBERTaV3-base is used to determine which samples 'belong' to which clusters

Minor point(s)

* L30: "Figure 3 shows the scores of seven LLMs for an example query, where the top three LLMs have significantly higher scores."
  + Seems unclear whether top-3 is so different from top-4 : Maybe a better example could be used
questions: questions: * For the "OOD" results, how disjoint (really) are (i) CMMLU and C-EVAL; (ii) HumanEval and MBPP?

* Table 3: Robustness of RouterDC to LLM losses during inference - isn't this proving that (for instance) Meta-Llama3-8B is being consistently chosen for CMMLU?

* L46: "To improve the training stability, we cluster the training queries" - intra-group vs inter-group (within batch?)
  + L225: The number of clusters N is set to 5 : Is it a coincidence that this matches the number of training tasks?
    - Appendix B : Is it troubling that task-identity is roughly the same as pretrained cluster labels?  Perhaps the task prompts are a give-away...
    - L257: "Moreover, increasing N leads to higher average accuracy when N is small (≤ 4), but the accuracy saturates quickly." (ditto)

* How could this approach work in a chat context?  Is it just for single queries?
limitations: limitations: * The router being learned here is a task-wise classifier.  There is likely commercial value in having this kind of routing, but implementing/testing this doesn't seem particularly novel (apologies for the vagueness).
paper_summary: paper_summary: The paper introduces RouterDC, a novel LLM query router that employs contrastive learning losses to train an encoder and LLM embeddings for routing queries efficiently. The motivation behind this approach is the variability in LLM performance across tasks and domains, and the computational efficiency of routing over ensemble methods. The proposed methodology involves using a small LLM encoder, `mDeBERTaV3-base`, and learnable LLM embeddings for candidate LLMs. RouterDC employs a supervised learning approach with ground truth annotations, using contrastive losses to optimize the routing mechanism. Experimental results demonstrate that RouterDC outperforms existing routing methods on both in-distribution and out-of-distribution tasks.
strength_weakness: strength_weakness: - The parameter efficiency of RouterDC compared to existing routing methods is not addressed, which is important for understanding the scalability of the approach. For example, the training cost is likely significant when the number of LLMs scales up (each training query needs to be evaluated on each LLM), and retraining is required when any new LLMs are incorporated. This could impact the practicality of the system.
- The paper does not incorporate LLM costs into the loss function. When multiple LLMs perform well (which is the motivation of the work), it would be natural to choose the cheapest one to enhance cost efficiency.
- Table 3 shows that removing certain LLMs, such as `Chinese-Mistral-7B` or `dolphin-2.6-mistral-7b`, does not affect overall performance, and removing `Mistral-7B` even improves it. This indicates that including incapable LLMs can lead to unnecessary computational overhead and performance degradation. There is a lack of analysis for this issue, which can be crucial for an effective routing system in the real world.
questions: questions: 1. Table 1 lacks a comparison between different routing methods’ training time or training computational cost. How does RouterDC compare in terms of parameter efficiency with existing methods? This information is crucial for understanding the scalability of the approach. 
2. The performance improvements after incorporating sample-sample contrastive loss seem marginal. Can this be further analyzed or explained?
3. On OOD tasks, RouterDC performs worse than the best-performing LLM on certain individual tasks. Is there any analysis of the reasons?
4. Evaluation results on specialized routing benchmarks like [1] could provide a more comprehensive assessment of RouterDC's performance.
5. A normalized average score, instead of the simple average among the scores of different benchmarks, can provide a better comparison in Tables 1 and 2, considering the varying complexity and score scales across benchmarks.
6. There is no need to include RandomSelect as a weak baseline. 
7. In the "Routing to Different Numbers of LLMs" evaluation, why were LLMs added in the chosen order? Based on Table 1, adding them in performance-descending order might yield smaller accuracy enhancements.
8. Based on Table 3, the incorporation of certain incapable LLMs can lead to unnecessary computational overhead and performance degradation. Would an LLM-dropping mechanism improve overall performance?
9. The visualization of query embeddings from different benchmarks is confusing. Did you cluster queries within one benchmark or across several benchmarks in the sample-sample contrastive learning loss function?

---
[1] Hu, Qitian Jason, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. "ROUTERBENCH: A Benchmark for Multi-LLM Routing System." arXiv preprint arXiv:2403.12031 (2024).
limitations: limitations: The broader implications of this routing approach, including its scalability and cost efficiency in practical applications, should be more adequately addressed.

Comments: title: Reply to Reviewer 4ok1
Reply to Reviewer GGL7 (2)
A Gentle Reminder for Reviewer ytgM
Reply to Reviewer GGL7
comment: Thanks again for your positive rating.
We will certainly add the above discussions and experiments to the revision.

---

> Q12. However, after re-reading the paper, I noticed that it currently lacks a comparison or at least a discussion of the existing cascade-based approaches [1-4] for assembling LLMs. 

**A12.** 
We have discussed a cascade-based method (i.e., FrugalGPT) in the related work of the paper (Lines 75-77, Lines 182-184).
Thanks for bringing the four other cascade-based methods to our attention.
We agree that cascade-based methods are one direction to achieve cost-effectiveness when choosing LLMs, but they are different from RouterDC in terms of **settings, inference cost, and tasks**.
We discuss the differences between our RouterDC and the mentioned cascade-based methods below.


(i) 
RouterDC considers **a different setting** compared with the cascade-based methods [1-4]. The cascade-based methods usually **assume that the capacity of LLMs depends on the model size**.
Their intuitive idea is to query LLMs from weak (small) to strong  (large) until a satisfactory answer is obtained instead of calling the strong LLMs for all queries. 
Our RouterDC does not require this assumption and can select a suitable LLM from multiple small or large candidate LLMs.
Hence, routing-based methods are more general.
Furthermore, even if LLMs are of the same size, they may have different specialized capabilities.

(ii)
Cascade-based methods [1-4] may call LLMs **multiple times** for a query (in the worst case, all candidate LLMs need to be called), but our RouterDC only needs to call the selected LLM **once** in inference/testing.

(iii) 
RouterDC is general and can be used for **generation tasks**, but Model Cascading [1] and Online Cascade Learning [4] are limited to **classification task** (e.g., SST2, MRPC, IMDB).
Generation tasks are usually more useful and challenging than classification tasks in NLP.

We will include the above discussion and related works in the revision.

---

#### References

[1] Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems, EMNLP 2022

[2] Language Model Cascades: Token-level Uncertainty and Beyond, ICLR 2024

[3] Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning, ICLR 2024

[4] Online Cascade Learning for Efficient Inference over Streams, ICML 2024
We are glad that our additional experiments have resolved your concerns, and we will add the experiments to the revision.

---

> Q14. Actually, there was no typo. Your claim is that one dataset is OOD compared to the other : Surely then, it should be surprising that they have textually identical pairs in them?

**A14.**
We apologize for our misunderstanding of your comments.
As discussed previously, most of the CMMLU and C-EVAL samples (about 99%) are different. Thus, the tiny overlap may not significantly affect the evaluation.
A12. """Thanks for your comments! We guess there was a typo in the comment, it should be "there are NOT any strict overlaps between the datasets : Interesting!" """

Actually, there was no typo.  Your claim is that one dataset is OOD compared to the other : Surely then, it should be surprising that they have textually identical pairs in them?

A11(+A13) : This is good additional work, that goes far further in proving your case.  Thank you.

And for that part, I'll inch up one (last) time:

New Rating: 6  New Confidence : 4
Thank you for the detailed responses. I am happy to vote for acceptance. Please ensure that these discussions and results are included in your revision.

However, after re-reading the paper, I noticed that it currently lacks a comparison or at least a discussion of the existing cascade-based approaches [1-4] for assembling LLMs. Unlike ensemble-based methods, cascade approaches do not necessarily invoke all models and can also achieve good cost-performance trade-offs as routing approaches (indeed, they may invoke multiple LLM calls for one query, but this does not necessarily mean that their cost-efficiency is worse). Including such a discussion or comparison would provide a more comprehensive understanding of your work within the context of existing research.

[1] Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems, EMNLP 2022

[2] Language Model Cascades: Token-level Uncertainty and Beyond, ICLR 2024

[3] Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning, ICLR 2024

[4] Online Cascade Learning for Efficient Inference over Streams, ICML 2024
Dear Reviewer ytgM,

We sincerely thank you again for your effort to improve our work.
We have provided a detailed response to resolve your concerns.
We would like to kindly remind the reviewer that the close date of the reviewer-author discussion is approaching.
Please let us know if you have any additional questions or comments.

Best,

The Authors
Thanks for your further comments and **raising the score**.
For the remaining concerns, we address them as follows.

---

> Q11. One of my concerns was that for the data you trained/tested over, the two things were so closely matched that the different datasets would leak the task just due to the wording, etc.

**A11.** 
We understand the reviewer’s concern that certain words in the query may leak the task identity, making it easy for RouterDC to perform like a task classifier.

To resolve this concern, 
we conducted an additional experiment in **a single task setting**, i.e., we train the router on the training set of HumanEval and evaluate it on the testing set.
The
single task setting is an edge case where **all queries may contain the same task information**.
Hence,
the router needs to learn how to route queries appropriately based on the query itself instead of some possible task information contained in the query.
Table below reports the testing accuracy.
As can be seen, 
**RouterDC largely outperforms the best candidate LLM (i.e., dolphin-2.9-llama3-8b) and existing routing methods**,
demonstrating that the router can select appropriate LLMs for queries based on query characteristics.
We will add the experiment and discussion to the revision, which definitely improve our work.

\begin{array}{lc}
\hline
\text{Method} &   \text{HumanEval}   \newline
\hline
\text{Mistral-7B} & 28.98 \newline
\text{MetaMath-Mistral-7B} & 29.80 \newline
\text{zephyr-7b-beta} & 22.04 \newline
\text{Chinese-Mistral-7B} & 21.43 \newline
\text{dolphin-2.6-mistral-7b} & 45.10 \newline
\text{Meta-Llama-3-8B} & 26.73 \newline
\text{dolphin-2.9-llama3-8b} & 49.39 \newline
\hline
\text{ZOOTER} & 39.38 \newline
\text{CosineClassifier} & 52.45 \newline
\text{RouterDC} & \mathbf{56.32} \newline
\hline
\end{array}

---

> Q12. I'm surprised that there are any strict overlaps between the datasets : Interesting! 

**A12.**
Thanks for your comments!
We guess there was a typo in the comment, it should be "there are **NOT** any strict overlaps between the datasets : Interesting!"


> Q13. But that doesn't change the point that the the claim that the alternate datasets are 'OOD' seems like an overreach.

**A13.**
Thanks for your insightful comments!
We appreciate the reviewer raising the concern about the definition of OOD.
In the paper, C-EVAL and MBPP are treated as OOD tasks as they are different task distributions or question-answer instruction.
We briefly summarize their differences below.

(i) CMMLU and C-EVAL have **different task distributions**. 
CMMLU contains more culture- and region-related tasks, while C-EVAL have more STEM tasks.
Moreover, CMMLU and C-EVAL use **different prompts** to ask multiple-choice question. 
C-EVAL uses continuous underscores to indicate the answer's location, whereas CMMLU employs no special notation for referencing the answer, except for brackets when the answer is within a sentence.

(ii) 
HumanEval and MBPP assess the code generation proficiency of LLM from **two distinct perspectives**. 
A HumanEval query **gives the header** of a python function and some comments, requiring the LLM to **implement the rest** of the function. On the other hand, a MBPP query **gives an intent** and  asks the LLM to **generate the function from scratch**.

To further resolve this concerns, 
we evaluate the learned router on **one more OOD task: JavaScript** [R1], 
which aims to generate JavaScript code to solve problems.
Different from HumanEval, which generates Python code to solve problems, 
JavaScript can be viewed as **a distant OOD task**.
Table below reports the testing accuracy.
As can be seen, RouterDC outperforms existing routing methods by a large margin, demonstrating 
that our RouterDC is more effective in routing queries of the distant OOD task.

We will include the additional experiments and discussions in the revision.


\begin{array}{lc}
\hline
 &   \text{JavaScript}   \newline
\hline
\text{Mistral-7B} & 29.88 \newline
\text{MetaMath-Mistral-7B} & 31.83 \newline
\text{zephyr-7b-beta} & 11.71 \newline
\text{Chinese-Mistral-7B} & 17.68 \newline
\text{dolphin-2.6-mistral-7b} & 45.00 \newline
\text{Meta-Llama-3-8B} & 37.07 \newline
\text{dolphin-2.9-llama3-8b} & 53.84 \newline
\hline
\text{ZOOTER} & 41.64 \newline
\text{CosineClassifier} & 37.32 \newline
\text{RouterDC} & \mathbf{48.66} \newline
\hline
\end{array}

---

### references

[R1] CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. KDD 2023.
* A1+A8 - It was already clear that you are doing query-based rather than task-based classification.  One of my concerns was that for the data you trained/tested over, the two things were so closely matched that the different datasets would leak the task just due to the wording, etc.  Your Table R2 helps.

* A4 : I'm surprised that there are *any* strict overlaps between the datasets : Interesting!  But that doesn't change the point that the the claim that the alternate datasets are 'OOD' seems like an overreach.

I'm happy to update a little:

New Rating: 5
New Confidence : 4