EmbedLLM: Learning Compact Representations of Large Language Models

## Abstract

Abstract With hundreds of thousands of language models available on Huggingface today, efficiently evaluating and utilizing these models across various downstream tasks has become increasingly critical. Many existing methods repeatedly learn task-specific representations of Large Language Models (LLMs), which leads to inefficiencies in both time and computational resources. To address this, we propose EmbedLLM, a framework designed to learn compact vector representations of LLMs that facilitate downstream applications involving many models, such as model routing.
We introduce an encoder-decoder approach for learning such embeddings, along with a systematic framework to evaluate their effectiveness.
Empirical results show that EmbedLLM outperforms prior methods in model routing both in accuracy and latency. Additionally, we demonstrate that our method can forecast a model’s performance on multiple benchmarks, without incurring additional inference cost. Extensive probing experiments validate that the learned embeddings capture key model characteristics, e.g. whether the model is specialized for coding tasks, even without being explicitly trained on them.
We open source our dataset, code and embedder to facilitate further research and application: https://github.com/richardzhuang0412/EmbedLLM .

## 1 Introduction

Recent breakthroughs in Large Language Models (LLMs) (Vaswani et al., ) have led to the creation of a vast array of models, each tailored for different use cases. These models, ranging from small, specialized models to large, general-purpose systems (Hao et al., ), differ significantly in their architecture, size, training data, and performance characteristics. For example, while some models excel as conversational agents, others may be more suitable for code generation or logical reasoning tasks. However, with this explosion of diverse LLMs comes a major challenge:

How to efficiently manage, compare, and utilize the growing number of LLMs?

Traditionally, benchmarking has served as the primary method for comparing LLMs, where each model is evaluated on a fixed set of test cases, and a score is generated to represent its performance. Meanwhile, model routing systems are developed to efficiently select models given queries of different types. An example workflow of these tasks can be seen in Figure 2 and Figure 3. While these approaches are often robust indicators of a model’s strengths and weaknesses, the construction of their workflows induces repeatedly learning representations of various LLMs to suit each individual downstream tasks and is therefore time-consuming and compute-demanding.

In response to these challenges, we introduce EmbedLLM, a compute-friendly framework designed to learn compact vector representations of large language models that facilitates different tasks. EmbedLLM map models into a latent vector space that captures important model characteristics. More importantly, EmbedLLM produces a unified representation that can be simultaneously applied to various downstream tasks such as correctness forecasting (Section 5.1), model routing (Section 5.2), and benchmark accuracy evaluation (Section 5.3). The core idea is to enforce this representation learning through a reconstruction-based system that tries to predict the model’s answer (correctness) from the learned embeddings, ensuring that each model’s embedding retains the most salient features relevant to enhance performance across multiple scenarios.

Figure: Figure 1: An illustration of the EmbedLLM Pipeline. An embedder network is pretrained to convert models into vector embeddings. Downstream applications like model routing are adapted by training an additional linear layer on top of these embeddings.
Refer to caption: extracted/5932887/figs/llm2vec_diagram.jpg

In summary, our contributions are as follows:

- 1.
We propose a novel framework based on a reconstruction system for learning compact, low-dimensional representations of LLMs. To the best of our knowledge, this is the first work that explicitly introduces the concept of turning LLMs into embeddings.
- 2.
We introduce an evaluation setup that employs the model embeddings to *simultaneously* predict model answer correctness on unseen questions, perform model routing, and predict benchmark accuracy with the addition of only a linear classifier.
Our system demonstrates the potential for significantly reducing the need for task-specific re-training.
- 3.
We perform probing experiments to validate that the learned embeddings capture meaningful information. We discover that models with similar characteristics remain close in the embedding space, and that the effect of incorporating each benchmark is reflected through the change in model embeddings.

EmbedLLM offers a scalable and unified approach to model evaluation and selection. By producing embeddings that encapsulate important features across tasks, our framework provides a versatile method to navigate the increasingly complex landscape of large language models.

## 2 Related Work

Representation Learning There have been numerous attempts to learn representations of various types of information. For natural languages, Mikolov et al. () and Pennington et al. () revolutionized the way models capture word semantics. In the field of computer vision, self-supervised techniques (Noroozi & Favaro, ) (Vondrick et al., ) are designed to learn low-dimensional representations that bolster downstream classification or segmentation performances. Inspired by these work and realizing an increasing demand of various LLMs being trained, we propose a creative framework to learn embeddings of LLMs.

LLM Benchmarking Benchmarking has been a standard way to compare LLMs, where a collection of questions/prompts is input to the LLMs and the quality of the corresponding responses is evaluated. There are datasets curated for general knowledge and language understanding (Wang et al., ), as well as domain-specific ones evaluating reasoning abilities (Frohberg & Binder, ), narrative control (Hartvigsen et al., ), function-calling (Yan et al., ; Srinivasan et al., ) and instruction-following (Li et al., ; Dubois et al., ). However, assessing quality of benchmarks remains an under studied question, and systematically evaluating all of them incurs an enormous amount of inference cost.

LLM Routing Our work focuses on predictive routing, which is a technique aimed at proposing the most suitable model given a task, without actually passing the query through each one of them. As summarized in Hu et al. (), most routers adopt either a supervised technique (Ong et al., ; Shnitzer et al., ) or a reward-based method (Lu et al., ). Trained by seeing responses from different models to the same prompt, these systems are intrinsically building an understanding of key model characteristics. Our work establishes yet another new interesting and efficient research direction as we find model
embeddings strengthen routing performances.

## 3 Formulation

### 3.1 Problem Setup

Let $\mathcal{M}=\{M_{1},M_{2}\cdots M_{n}\}$ be a set of different LLMs, $\mathcal{P}$ denote the set of all possible prompts, and $\mathcal{A}$ denote the corresponding set of possible answers. We can simply identify any LLM $M$ with an inference function mapping from a prompt space to an answer space $f_{M}:\mathcal{P}\rightarrow\mathcal{A}$, which outputs an answer $a\in\mathcal{A}$ given a prompt $p\in\mathcal{P}$.

Among downstream tasks, representations of different LLMs needed to be constructed in various ways. A naive example is benchmarking: Where the crucial part is to select a test prompt set $\mathbb{P}_{Bench}=\{p_{1},p_{2}\cdots p_{m}\}$ as well as an scoring function $g_{eval}:\mathcal{P}\times\mathcal{A}\rightarrow\mathbb{R}$, mapping model responses to a scalar score. Take MMLU as an example, each model is queried by a set of multiple-choice questions with four choices. The model’s response is recorded by comparing the output probabilities of the answer choices “A”, “B”, “C”, and “D”. Then the responses are simply scored by matching with the ground truth answer key. Within this process, every model $M$’s behavior is essentially summarized by their output on a set of test questions$\{(p_{1},a_{(M,1)}),(p_{2},a_{(M,2)})\cdots(p_{m},a_{(M,m)})\}$.

Figure: Figure 2: An illustration of the traditional workflow of LLM benchmarking.
Refer to caption: extracted/5932887/figs/benchmark_diagram.jpg

Another example is model routing: given a pool of $n$ LLMs, a router function is defined in Ong et al. () as a $n$-way classifier that assigns models to different queries to maximize response quality while minimize inference cost. Similarly, training such router also often involves in transforming each model into a relatively low-dimensional representation by utilizing sample question-answer pairs.

Figure: Figure 3: An illustration of the traditional workflow of model routing, using exemplar routing methodologies from Ong et al. ()
Refer to caption: extracted/5932887/figs/router_diagram.jpg

Identifying a common need for model characterization, we raise the question: what if we could accomplish the above tasks by directly working with a unified, compact representation of LLMs? In this work, we provide a framework to learn such a representation. We define an embedding function $\phi:\mathcal{M}\rightarrow\mathbb{R}^{d}$, parametrized by a neural network or otherwise, that maps each model $M$ to a compact vector representation in a latent embedding space. We aim to learn model embeddings that contain important features of LLMs that are useful to both quantifying differences between various models and aiding across downstream tasks.

### 3.2 Evaluation Metrics

Evaluating the quality of model embeddings is crucial to ensure they effectively capture the underlying structure and semantics of the data which we care about for potential downstream tasks. The core idea is to use the embeddings to predict model behavior on unseen tasks by training an inference function $\psi:\mathbb{R}^{d}\times\mathcal{P}\rightarrow\mathbb{Q}$, that leverages a model embedding and new test prompts to predict a model’s performances as quantified by a desired evaluation metric in the space $\mathbb{Q}$. For instance, if our task is to predict whether a model can correctly answer an unseen question, the inference function would be a network that takes in model embeddings and the unseen question (usually present in the form of its embedding as well) and output a binary label as the prediction of model correctness. Note that model responses can be evaluated through many means such as determining correctness (Hendrycks et al., ), judging by human (Chiang et al., ) or stronger LLMs (Zheng et al., ), or measuring any other task-specific metric. Hence, each task constitute to its own unit of measure and correspondingly determines an evaluation metric. In this work, we focus on evaluating on the following downstream tasks:

- •
Correctness Forecasting: We query the models on benchmarks equipped with ground truth answers, and produce binary correctness label for every model-question pair. For this task, a natural metric to use would be test prediction accuracy.
- •
Model Routing: With the help of a embedding vector of each model, we’re able to develop a simple linear router that directly determine the model to be routed to using the probability of the answer correctness along with some learnable threshold. We measure the routing performance by reporting the average response quality(^1^11Note that this metric can be insufficient under certain settings (Ong et al., ), but it fits well to our goal of assessing whether our model embeddings effectively captures the strengths and weaknesses under the context of correctness forecasting.), namely the proportion of times a router successfully route to an LLM that correctly answers the given query.
- •
Benchmark Accuracy Prediction: We treat the embeddings as primary features to train a linear model in predicting accuracies on unseen benchmarks (measured from 0% to 100% as a percentage). The metric used for this task would be the classical mean-squared-error (MSE) for linear regression.

In addition, we could directly compare between embeddings in their raw vector form following Mikolov et al. (). For instance, models with similar traits should have embeddings that are closer in L2 distance.

## 4 EmbedLLM

### 4.1 Methodology

In order to learn such embeddings, we draw inspirations from image reconstruction algorithms (He et al., ; Ronneberger et al., ): we want to learn a “reconstruction” system, where the choice of the reconstruction target is arbitrary and can be task-dependent. More concretely, let $\mathbb{Q}$ be a space of model performance metric, and $\mathcal{M}$ be a set of possible LLMs, $m$ be the number of models, $n$ be the number of questions/prompts. We want to learn a reconstruction network $\mathcal{R}:\mathbb{Q}^{m\times n}\rightarrow\mathbb{Q}^{m\times n}$ that optimizes to reconstruct a matrix $X\in\mathbb{Q}^{m\times n}$, where $X_{ij}\in\mathbb{Q}$ denotes the model performance metric for model i on question j. The encoder-decoder architecture ensures the imposure of such constraint and enforce the model embeddings to efficiently capture the key characteristics of each model.

In this work, we decide to use the task of predicting model answer correctness as our auxiliary target, i.e., $\mathbb{Q}=\{0,1\}$. Notice that our training objective is only a decoy - any downstream task that requires understanding of model characteristics would qualify and our ultimate goal is to enforce the reconstruction network to learn a compact yet information-rich representation of the models in this process.

### 4.2 Dataset

We now describe the data collection process of correctness results of various LLMs’ responses to questions from mainstream benchmarks.

We selected 112 open-sourced models(^2^22See appendix for full list.) of various sizes, with both general purpose LLMs (Zhu et al., ) and specialized LLMs included to ensure comprehensive coverage. Then we aggregated responses of every model to 36,054 questions from the test sets of MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., ) , SocialQA (Sap et al., ), PIQA(Bisk et al., ), MedMCQA(Pal et al., ), MathQA(Amini et al., ), LogiQA(Liu et al., ), GSM8K(Cobbe et al., ), GPQA(Rein et al., ), and ASDiv(Miao et al., ). The responses to these questions were acquired and evaluated through using the “lm-evaluation-harness” package (Gao et al., ) to give a binary correctness label for each model-question pair. We performed a random 80%-10%-10% train-validation-test split on the questions and used the sentence transformer “all-mpnet-base-v2” (Reimers & Gurevych, ) to convert the questions into an initial embedding state of dimension $dim_{q}=768$. Consequently, our question embedding tensor $X$ has the shape (36054, 768) where the $i$-th row $X_{i}\in\mathbb{R}^{dim_{q}}$ is the embedding for model $i$ , and our label tensor is essentially a correctness matrix $Y$ with shape (112, 36054), where the $ij$-th entry $Y_{ij}$ represents the binary correctness of model $i$ answering question $j$.

### 4.3 Algorithm

We adopt the Encoder Decoder architecture:

Encoder: The encoder consists of a model embedding network and a question embedding network. Let $dim_{embed}$ be the desired dimension of the model embedding. The model embedding network $\phi_{m}:\mathcal{M}$(^3^33In the actual implementation, we give each model an index as an identification in $\mathcal{M}$.) $\rightarrow\mathbb{R}^{dim_{embed}}$ maps each model into a latent representation. Similarly, the question embedding network $\phi_{q}:\mathcal{P}\rightarrow\mathbb{R}^{dim_{embed}}$ maps each questions into a latent representation. In our setting, the question embedding network is a two-step transformation $\phi_{q}=g_{st}\circ h_{proj}$ where $g_{st}:\mathbb{P}\rightarrow\mathbb{R}^{dim_{q}}$ denotes the pre-processing (performed in Section 4.2) that turns each question from text to an initial embedding space by using sentence transformer, and $h_{proj}:\mathbb{R}^{dim_{q}}\rightarrow\mathbb{R}^{dim_{embed}}$ is a projection layer from this original embedding space to the same space as the model embeddings.

Decoder: The decoder is a classifier network $\psi:\mathbb{R}^{dim_{embed}}\times\mathbb{R}^{dim_{embed}}\rightarrow\{0,1\}$ that takes in both the encoded embeddings of the model and the question, and output a binary label which is the prediction of whether the model answer the question correctly. For this work, our decoder is represented as $\psi(v_{m},v_{q}^{{}^{\prime}})=\varphi(v_{m}\odot v_{q}^{{}^{\prime}})$ where $\varphi:\mathbb{R}^{dim_{embed}}\rightarrow\mathbb{R}^{2}$ is a linear classifier and $u\odot v$ represents the Hadamard (element-wise) product between two vectors. For each model-question pair, this decoder network outputs two logits $p_{{(m,q)}_{0}}$ and $p_{{(m,q)}_{1}}$, and the “correctness score” $s_{m,q}=\sigma(p_{{(m,q)}_{1}}-p_{{(m,q)}_{0}})$ represent the predicted probability of the model $m$ correctly answering question $q$, where $\sigma(x)$ is the sigmoid function.

Suppose $y$ is the correctness label of model $m$ answering question $q$, we calculate the following BCE loss function during training,

$$ $L(m,q,y)=-\left(y\cdot\log(s_{m,q})+(1-y)\cdot\log(1-s_{m,q})\right)$ (1) $$

In essence, this algorithm analogizes to a Matrix Factorization algorithm, where we learn a $n\times m$ model embedding matrix and a $m\times p$ question embedding matrix such that their product reconstructs the original $n\times p$ correctness matrix. Hence, we refer to this reconstruction system as “Matrix Factorization”, or “MF” in short, in the following sections.

## 5 Experiment Results

As described in Section 3.2, we conducted experiment in correctness forecasting, model routing, and benchmark accuracy prediction to evaluate the quality of the learned embeddings.

### 5.1 Correctness Forecasting

For correction prediction, we compare the effect of our matrix factorization model to a KNN-classifier (Fix, ). In the context of our formulation, although without an explicit embeddings for models, KNN-classifier can be seen as using the integration of all question-correctness tuples from a model as its “embedding”, and making inference from this aggregation. For brevity, we refer to this approach as KNN in the subsequent text. As mentioned in Section 3.2, we use correctness forecasting accuracy on the test set as the evaluation metric.

We evaluate the performance of KNN and Matrix Factorization across various sizes of training set. We produce the smaller training sets from randomly subsetting from the full training set. For each training set, we conduct hyperparameter tuning (number of neighbors for KNN, model embedding dimension for MF) on a fixed validation set and evaluate prediction accuracy using a fixed test set. The result in Table 1 indicates a better scalability of our method.

**Table 1: Performance of Predicting Model Correctness on Unseen Questions where the columns indicate the number of questions in the training set. MF constantly outperforms KNN on training sets of all scale. We see a steady increase in the performance as the dataset size grows, indicating further scalability of our method.**
| Algorithm | Dataset Size |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1K | 5K | 10K | 15K | 20K | 25K | Full (29K) |  |
| KNN | 0.6372 | 0.7078 | 0.7107 | 0.7128 | 0.7143 | 0.7146 | 0.7152 |
| Matrix Factorization | 0.6443 | 0.7331 | 0.7362 | 0.7378 | 0.7390 | 0.7394 | $\mathbf{0.7409}$ |

### 5.2 Model Routing

Using the same correctness data, we can evaluate the quality of a matrix-factorization-based router. For this task, we evaluate the router’s accuracy by measuring the proportion of times it successfully route to a model that could correctly answer the given query. For a test question $q_{k}$, we pass it through the router network along with all $n$ possible model embeddings, producing $n$ correctness score $s_{M_{1},q_{k}},s_{M_{2},q_{k}},\cdots,s_{M_{n},q_{k}}$, and check if the model with the highest correctness score correctly answers the question. Aggregating through all questions, we report the router accuracy as $acc_{router}=\frac{1}{m}\sum_{k=1}^{m}\mathbbm{1}\{X_{i^{*},k}=1\}$ where $i^{*}={\operatorname*{arg\,max}}\{s_{M_{i},q_{k}}|i=1\cdots n\}$ is the routed model and $X$ is the correctness matrix described in Section 4.2.

We compare the performance of the Matrix Factorization router with two baselines.
The first one is the single-best model router which always selects the model with the highest accuracy and thus gives a constant accuracy. The second one is a random router that select each model the same number of times as the Matrix Factorization router, but instead randomly assign models to questions. For this router, we can calculate its expected accuracy given the proportions of times each model is selected by the Matrix Factorization router. For instance, if our router selects $M_{1}$ 70% of the time, $M_{2}$ 20% of the time, and $M_{3}$ 10% of the time, the expected accuracy of the random router will be calculated as a weighted accuracy $acc_{weighted}=0.7*acc_{M_{1}}+0.2*acc_{M_{2}}+0.1*acc_{M_{3}}$. Note that this weighted accuracy will always be smaller than the single-best model accuracy - we propose this metric as an evaluation of how well our router direct models to the questions they are good at given a fixed “budget” of model calls. We report both the overall accuracy across the whole test set and accuracy per source benchmark.

As seen in Figure 4, MF router performs better than both the single-best router and the random router overall. As the best performing model is re-determined on every benchmark, the single-best model router performs better for each benchmark than in overall case. Here, MF router achieves near single-best model router accuracy while still managing to utilize the respective strengths of different models which is shown by the significant difference between accuracies of MF router and weighted router.

Another advantage of MF router we find is its high routing speed. On one NVIDIA A100 80GB GPU, it takes in average 3.80 seconds for MF router to route 3,000 questions on 50 repeated trials which is basically free compare to the downstream model inference time. Specifically, compared to the causal LLM router in Ong et al. () that processes less than 50 requests per second, our router delivers more than 750 model selections, which is 15x faster while selecting from a much larger model pool (112 models against 2 models for causal LLM router). In addition, it only takes less than 1GB of GPU memory when training a MF router using our dataset, which is 60x cheaper than fine-tuning Llama-3-8B as a router in terms of memory usage. This illustrates the superiority of MF router in both performance, latency, and training cost.

Figure: Figure 4: Performance accuracy of MF router compared to baselines. MF router performs better across the whole test set and achieves accuracies close to the single-best model on every benchmark.
Refer to caption: extracted/5932887/figs/routing_accuracy.png

### 5.3 Benchmark Accuracy Prediction

To predict model’s average accuracy on a benchmark $B$, we trained MF using leave-one-out correctness data, which includes correctness results of all models on all questions except the ones in $B$. Then we take the model embeddings directly as features to train a linear regression of the form:

$$ $a\mathbf{E}=\mathbf{y}$ $$

where $E$ is a model embedding matrix with the $i$-th row representing the model embedding for the $i$-th model, and the $j$-th entry in the vector $\mathbf{y}$ corresponds to the $j$-th model’s average correctness accuracy on the test benchmark, which is a number from 0 to 1.

For each test benchmark, we conducted 100 random train-test splits on the 112 models contained in our dataset, trained a linear regression on the training set, and evaluated the correlation between model embeddings and test benchmark performances on the test set through applying Kendall’s Tau test(^4^44The Kendall’s Tau test is a measure of correspondence between two rankings. We use this test to see if model ability can be correctly ordered simply using a linear system with the embeddings as the only feature.). From Figure 5, statistical significance is found in 7 out of the 10 benchmarks, indicating that model embedding contains information to distinguish between model performances on most benchmarks.

Figure: Figure 5: Left: Sorted Kendall’s Tau test result of accuracy prediction on the benchmarks. The “Significance” column represents the number of times with significant correlation detected (at a 5% significance level) out of 100 random model splits. Right: An example comparing actual model accuracies on MathQA against model accuracies on MathQA predicted from the embeddings trained without MathQA data.
Refer to caption: extracted/5932887/figs/regression_example_mathqa.png

Notice that this prediction systems works even for large benchmarks like MMLU. Using correctness result from the rest of the 18,000 questions, our method predicts MMLU accuracy to a statistically significant extent. This saves an inference cost of 14,000 questions, equivalent to around 2 million input tokens per model, let alone cost for model output tokens and subsequent response evaluation. As number of models, model sizes and number of benchmarks are still rising rapidly, enabling benchmark accuracy prediction through model embeddings is vital to save both time and compute from repeatedly inferencing models on every new benchmark.

## 6 What information is in the Model embeddings

In this section we describe the probing experiments designed to understand what information is captured in the embedding.

### 6.1 Sanity Check Using Similarity

We expect the model embeddings to satisfy some basic properties: If two models $M,M^{\prime}$ generate the same answers for every prompt, then their embeddings are the same. Similarly, models with similar characteristics, trained using similar data, or adopted similar training pipelines should have similar embeddings, and vice versa. For instance, the model embedding of DeepSeekMath-7B (Shao et al., ) should be more similar to the embedding of other math models like MetaMath-Llemma-7B (Yu et al., ) than to the embedding of Medicine-LLM-13B (Cheng et al., ) which is adapted for biomedical applications. This property is easily fulfilled by Matrix Factorization algorithm as any two identical/similar models of such would produce identical/similar correctness result against most questions.

As a further sanity check, we assign binary labels to the 112 models we have evaluated according to the following 6 keywords: [7B, 13B, 70B, Coding, Bio/Med, Physics], forming 6 characteristic communities. For each community, we compare between the average intra-community and inter-community L2 distance of the embeddings. As shown in Figure 6, for all above 6 communities, the averaged intra-community L2 distances are smaller than the inter-community ones. This provides a preliminary guarantee that our embeddings are “meaningful” with respect to distance metrics.

Figure: Figure 6: The averaged intra-community L2 distance of the model embeddings is closer for all 6 communities selected, suggesting that basic model traits are captured in the latent embedding space.
Refer to caption: extracted/5932887/figs/community_distance.png

### 6.2 embeddings Capture Intrinsic Characteristics of Benchmarks

Next, as indicated from Figure 5, as a set of model embeddings is produced from a fixed training set, the embeddings seem to capture information of some benchmarks better and overlook information in some benchmarks. Hence, we design a set of ablation experiments to further understand the contribution of each benchmarks in the training data. Specifically, extending the experiment setup from Section 5.3,we have a question embedding tensor $X$ of shape $(\operatorname{num_questions},\operatorname{embedding_dimension})$ and a label tensor $Y$ where $Y_{ij}$ is the binary label of whether model $i$ correctly answers question $j$, with questions from a set of benchmarks $S=\{B_{1},B_{2}\cdots B_{n}\}$, to measure the effect of incorporating/removing an “contributor” benchmark $B_{i}$ on predicting correctness of a “testee” benchmark $B_{j}$, we:

- 1.
Construct two sets of benchmarks $S_{added}=S\setminus B_{j}$ and $S_{removed}=S\setminus(B_{i}\cup B_{j})$ and produce two new sets of question embedding and label tensor, $X_{added}$, $X_{removed}$, $Y_{added}$, and $Y_{removed}$, so that only questions containing in $S_{added}$ and $S_{removed}$ are kept respectively.
- 2.
Train a Matrix Factorization embedder separately on $(X_{added},Y_{added})$ and $(X_{removed},Y_{removed})$ to get two sets of model embeddings $E_{added}$ and $E_{removed}$, and respectively perform zero-shot benchmark prediction on $B_{j}$ with 100 random splits of models as in Section 5.3. Aggregate the total test mean squared error (MSE) $e_{removed}$ and $e_{added}$.
- 3.
Take the difference between the two error to compute a contribution score $C_{ij}=e_{removed}-e_{added}$ which quantifies the improvement on predicting model accuracy on $B_{j}$ when $B_{i}$ is added in training.

Essentially, we hypothesize that the addition/removal of every training benchmark would be reflected through the change in model embeddings which induces a performance difference in benchmark accuracy prediction. With this setup, we produce a $n\times n$ ($n$ is the total number of benchmarks) contribution matrix $C$ where the where the $ij$-th entry can be extracted exactly as $C_{ij}$ from the steps above(^5^55The diagonal entries of this matrix are set default to 0 as it is meaningless to train and test on the same benchmark.). To aggregate the overall effect of one specific benchmark $B_{i}$ as the contributor benchmark, we compute a row sum of the contribution matrix
$\sum_{j}C_{ij}$. This can be interpreted as the total improvements of adding benchmark $B_{i}$ on predicting model correctness of the rest of the benchmarks. Correspondingly, the column sum $\sum_{i}C_{ij}$ represents the total amount of improvement of predicting correctness of benchmark $B_{j}$ when each of the rest of the benchmarks is added.

Some noticeable phenomenon emerged from this set of experiments(^6^66We omit results of SocialQA in probing experiments as most of our models perform similarly poorly, resulting in all model accuracies crowding in the low accuracy region and becoming indistinguishable.):

- 1.
We find that incorporating MMLU into training the embeddings significantly help predicting accuracies on other benchmarks. This result matches the comprehensive nature of MMLU as it contains questions covering various topics.
Figure 7: Left: Effect of removing benchmarks on testing all other benchmarks. Higher value suggests that the addition of that benchmark into the training of model embedding enhances predicting model correctness on the rest of the benchmarks. Right: Effect of removing each of the rest of the all other benchmarks on the benchmark being tested. Higher value suggests that predicting models’ correctness in answering questions from the tested benchmark becomes easier with each of the rest of the benchmarks added overall.
- 2.
We find that incorporating other benchmarks into training set would harm the embedding’s predictive power on GPQA. This suggests that additional information that the embeddings capture from incorporating new benchmarks into training set is unrelated or negatively related to model performance on GPQA. In fact, GPQA is an extremely difficult benchmark for current LLMs, so this finding aligns with our expectation as model’s ability in answering simpler questions clearly do not transfer to answering harder ones.
- 3.
Additionally, we identify subsets of benchmarks that mutually improve each other’s accuracies when incorporated into training. For instance, there is a total MSE improvement of 0.271 when GSM8k is incorporated to predicting MathQA, 0.190 when MathQA is incorporated to predict ASDiv, and 0.103 when ASDiv is incorporated to predict MathQA. As all three benchmarks are math-related, we can deduce that in this specific subset of experiments, the level of math knowledge of our selected models are indeed reflected through the change in model embeddings.

## 7 Limitation

Despite the promising results, our work has several limitations. First, our dataset, though effective in demonstrating the potential of our embeddings with a limited number of samples, is relatively small. With data from only 112 models, the embeddings we extract are moderately sparse which limits deeper exploration of relationships between them. Second, while our embeddings effectively support various downstream tasks, they rely on a fixed pool of models. Introducing new models requires retraining the Matrix Factorization algorithm. Lastly, our study is restricted to correctness-based datasets, leaving other potentially valuable data types, such as text embeddings of model outputs, unexplored. To address these limitations, we have open-sourced our datasets and codebase for further research and experimentation.

## 8 Conclusion

We showcase the possibility of learning an unified, compact representation of LLMs via Matrix Factorization. Through extensive empirical evaluation, our method displays solid performance on correctness forecasting, model routing, and benchmark accuracy prediction, while significantly reducing the need of retraining and avoiding repetitive evaluations. Furthermore, we conduct various probing experiment to understand the information contained in the model embeddings. The results show that our embeddings capture not only key characteristics of the models, but also properties of the data used to train the embedder.

## Appendix A Appendix

### A.1 Model List

Here is an exhaustive list of models that we extract our dataset from:

**Table 2: The comprehensive list of the 112 models used to create the correctness dataset. This dataset is created on May 2025 so models released after that time are not available.**
| meta-llama/LlamaGuard-7b | meta-llama/Llama-2-13b-chat-hf |
| --- | --- |
| 01-ai/Yi-34B-Chat | meta-llama/Llama-2-70b-chat-hf |
| WizardLM/WizardLM-70B-V1.0 | allenai/tulu-2-dpo-70b |
| lmsys/vicuna-13b-v1.5 | lmsys/vicuna-33b-v1.3 |
| Qwen/Qwen-14B-Chat | upstage/SOLAR-10.7B-Instruct-v1.0 |
| openchat/openchat-3.5-0106 | openchat/openchat-3.5 |
| berkeley-nest/Starling-LM-7B-alpha | HuggingFaceH4/zephyr-7b-beta |
| TheBloke/tulu-30B-fp16 | mistralai/Mistral-7B-Instruct-v0.1 |
| tiiuae/falcon-40b-instruct | lmsys/vicuna-13b-v1.5-16k |
| codellama/CodeLlama-34b-Instruct-hf | TheBloke/WizardLM-13B-V1.2-GGUF |
| lmsys/vicuna-7b-v1.5 | NousResearch/Nous-Hermes-13b |
| project-baize/baize-v2-13b | lmsys/vicuna-7b-v1.5-16k |
| mosaicml/mpt-30b-instruct | meta-llama/Llama-2-7b-chat-hf |
| TheBloke/koala-13B-HF | nomic-ai/gpt4all-13b-snoozy |
| h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-13b | mosaicml/mpt-7b-chat |
| databricks/dolly-v2-12b | stabilityai/stablelm-tuned-alpha-7b |
| OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5 | deepseek-ai/deepseek-llm-67b-chat |
| NousResearch/Nous-Hermes-2-Yi-34B | CausalLM/34b-beta |
| SUSTech/SUS-Chat-34B | SUSTech/SUS-Chat-72B |
| Qwen/Qwen-72B | Intel/neural-chat-7b-v3-3 |
| ibivibiv/alpaca-dragon-72b-v1 | JaeyeonKang/CCK-Asura-v1 |
| ConvexAI/Luminex-34B-v0.2 | ConvexAI/Luminex-34B-v0.1 |
| CorticalStack/pastiche-crown-clown-7b-dare-dpo | eren23/ogno-monarch-jaskier-merge-7b-OH-PREF-DPO |
| bardsai/jaskier-7b-dpo-v5.6 | FelixChao/Scorpio-7B |
| dfurman/HermesBagel-34B-v0.1 | kevin009/llamaRAGdrama |
| sail/Sailor-7B | AiMavenAi/Prometheus-1.3 |
| Q-bert/Optimus-7B | cognitivecomputations/yayi2-30b-llama |
| zhengr/MixTAO-7Bx2-MoE-v8.1 | fblgit/UNA-SimpleSmaug-34b-v1beta |
| mistralai/Mixtral-8x7B-Instruct-v0.1 | microsoft/Orca-2-13b |
| EleutherAI/pythia-12b | cloudyu/Mixtral-11Bx2-MoE-19B |
| rishiraj/CatPPT-base | Deci/DeciLM-7B |
| microsoft/phi-2 | scb10x/typhoon-7b |
| 01-ai/Yi-6B-200K | 01-ai/Yi-6B |
| TigerResearch/tigerbot-13b-base | augmxnt/shisa-base-7b-v1 |
| microsoft/phi-1.5 | golaxy/gowizardlm |
| bigscience/bloom-7b1 | mlabonne/AlphaMonarch-7B |
| CultriX/NeuralTrix-bf16 | shadowml/MBeagleX-7B |
| yam-peleg/Experiment26-7B | deepseek-ai/deepseek-math-7b-instruct |
| meta-math/MetaMath-Mistral-7B | kyujinpy/Sakura-SOLRCA-Math-Instruct-DPO-v1 |
| FelixChao/llama2-13b-math1.2 | Plaban81/Moe-4x7b-math-reason-code |
| MaziyarPanahi/WizardLM-Math-70B-v0.1 | abhishek/zephyr-beta-math |
| meta-math/MetaMath-Llemma-7B | EleutherAI/llemma-34b |
| EleutherAI/llemma-7b | FelixChao/vicuna-7B-physics |
| Harshvir/Llama-2-7B-physics | FelixChao/vicuna-7B-chemical |
| BioMistral/BioMistral-7B | BioMistral/BioMistral-7B-DARE |
| PharMolix/BioMedGPT-LM-7B | Biomimicry-AI/ANIMA-Nectar-v2 |
| codellama/CodeLlama-7b-hf | codellama/CodeLlama-13b-Instruct-hf |
| deepseek-ai/deepseek-coder-1.3b-base | deepseek-ai/deepseek-coder-6.7b-instruct |
| OpenBuddy/openbuddy-codellama2-34b-v11.1-bf16 | TheBloke/CodeLlama-70B-Instruct-AWQ |
| AdaptLLM/medicine-chat | AdaptLLM/medicine-LLM |
| AdaptLLM/medicine-LLM-13B | Writer/palmyra-med-20b |
| SciPhi/SciPhi-Self-RAG-Mistral-7B-32k | Neko-Institute-of-Science/metharme-7b |
| Neko-Institute-of-Science/pygmalion-7b | SciPhi/SciPhi-Mistral-7B-32k |
| shleeeee/mistral-ko-tech-science-v1 | codefuse-ai/CodeFuse-DeepSeek-33B |
| WizardLM/WizardCoder-Python-34B-V1.0 | bigcode/octocoder |
| meta-llama/Meta-Llama-3-8B | meta-llama/Meta-Llama-3-8B-Instruct |
| meta-llama/Meta-Llama-3-70B | meta-llama/Meta-Llama-3-70B-Instruct |
| meta-llama/Meta-Llama-Guard-2-8B | Qwen/Qwen1.5-32B-Chat |
| Qwen/Qwen1.5-4B-Chat | Qwen/Qwen1.5-0.5B-Chat |
| Qwen/Qwen1.5-7B-Chat | Nexusflow/Starling-LM-7B-beta |
| google/gemma-7b-it | google/gemma-2b-it |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Please address the weakness I mentioned above.
soundness: 2
strengths: Embedding LLMs to handle downstream tasks is indeed a fascinating approach! This method allows you to create compact representations of each model that capture its unique strengths and weaknesses, enabling efficient task-specific decisions without running each model on every input. This approach streamlines the workflow significantly, as it allows for general-purpose embeddings that can adapt to a variety of downstream tasks without retraining the models themselves. It's especially beneficial in settings where computational resources are a concern or when the model pool is large.
summary: This paper proposes a novel framework for generating compact vector representations of LLMs to enhance model routing, task efficiency, and performance forecasting. The EmbedLLM framework creates embeddings that capture important characteristics of different LLMs, such as suitability for specific tasks like coding or conversational response generation. Experiments results indicate that the embeddings effectively capture key characteristics of LLMs, enabling efficient and accurate task allocation and performance prediction across a variety of benchmarks.
weaknesses: The term "decoder" in this paper is a bit misleading. In typical encoder-decoder architectures, the "decoder" reconstructs or generates the output in its full or intended form, such as reconstructing text in sequence-to-sequence tasks. Here, however, the so-called "decoder" is merely a binary classifier that outputs a label indicating whether the LLM correctly answered a question.
We have to re-train the embedder if we want to represent new models, this makes the whole framework non-scalable. I'd like to see details cost metrics for re-train embedder for new model vs traditional benchmarking approaches.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 2
questions: See the weaknesses
soundness: 3
strengths: - The paper innovatively proposes the embedding of LLMs to facilitate managing and comparing them.    
- The experiments in the paper are comprehensive, tested on 112 large models
summary: The paper presents EmbedLLM, a framework for creating compact vector embeddings of large language models (LLMs) to improve efficiency in tasks like model routing and benchmark performance prediction. EmbedLLM uses an encoder-decoder architecture to map LLMs into a unified embedding space that captures important model characteristics. This representation allows accurate model selection and performance forecasting across multiple tasks without repetitive re-evaluation, reducing both time and computational costs. Experiments show that EmbedLLM enhances accuracy and latency in model routing and can predict benchmark scores reliably. The embeddings also reflect intrinsic model attributes, useful for identifying task-specific strengths, even in models not explicitly trained for certain tasks.
weaknesses: - The paper proposes a method for encoding LLMs. However, in the implementation, this encoding is merely based on model IDs, treating each model entirely as a black box. With only 30,000 data for training, can the resulting encoding truly capture all the characteristics of the models? Large models differ significantly in their strengths across various domains and capabilities. Can such an approach, based solely on one round question-answer pairs, truly distinguish the models’ abilities when facing complex reasoning problems? Another issue is whether the scale of the proposed embedding network is sufficient to represent the characteristics of numerous models effectively.

- A little confused about line 256-257. Why are two values (p(m,q)_0 and p(m,q)_1) output here? Is it simply to add a nonlinear operation?

- The experiments do not appear to categorize models by scale. Intuitively, the larger the model, the smaller the performance differences between models. What is the authors' view on this issue?

- The paper does not seem to provide a clear and intuitive illustration of the overall architecture, including training, input, and output processes. Figures 1-3 are quite similar and take up too much space, and it wasn’t until section 4.1 that I understood how the whole system works. Of course, I do not deny the paper's contributions, but I suggest the authors improve the visual presentation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why does the method assume that answer have to be a binary label? Is this a limitation of this method?
2. The explanation of kNN-classifer baseline is unclear. How is final label chosen? -- based on the majority voting of k-nearest neighbours?
3. Lines 264--267 are unclear. What is mxp and nxp represent?
4. Some qualitative discussion with examples on what happens when the resulting data points are widely different from already seen data points will be useful. 
5. Does the prompt embedding have an impact on the results, i.e., using better recent LLM-based embeddings. 
6. I would also like to see the TSNE plots of the embeddings.

Minor:
1. Vaswani et al. should be 2017. The paper opens up with a wrong citation. This paper is also wrongly attributed to LLMs. You could be more careful with your citations.
2. The method should perhaps be named EmbedLLM rather than Matrix Factorization.

I am willing to increase the score based on better presentation and qualitative analysis. The paper is quite weak in this aspect. 

====

I increased my score after author rebuttal. I am worried about the quality of these embeddings given that no patterns are observed in TSNE plots, but a score of 7 is appropriate for other clarifications. Since there is no 7, I had to chose 8.
soundness: 3
strengths: 1. A simple and intuitive method.
2. Once the embedding of an LLM is built, the performance of the LLM can be accessed without access to the model.
summary: The goal of this paper is to represent LLMs using an embedding that can predict performance on new inputs and benchmarks. Such an embedding can be learned from matrix factorization methods applied to a matrix that contain the behavior of the LLM on several data points as well as behavior of other LLMs on these datapoints, i.e., a matrix with rows as LLMs and columns as data points, and each cell indicate an LLM's performance on a specific data point. The factorization aims to reconstruct this matrix by learning an embedding for each LLM and an embedding for each data point (built from its static embedding which is further projected into the same embedding space of LLM). 

The resulting embeddings are evaluated to predict performance of new data points and benchmarks and to improve model routing.
weaknesses: 1. The paper can be motivated better and clarify why the current formulation is intuitive, i.e., a model's behavior on new data points can be predicted based on its behavior on already existing data points.
2. Details are hard to follow or underspecified, e.g., kNN classifier, random routing.
3. There are no references to highly similar work that predicts performance on a new task based on the performance of existing tasks. For example, Xia et al. Predicting Performance for Natural Language Processing Tasks, 2020
4. Calling the method as encoder-decoder in Sec 4.3 is confusing. The decoder is nothing but a classifier, and calling it straightforwardly that will make it easier to follow instead of calling it a decoder.
5. There is no analysis/visualations of the embeddings. Current histogram plot in Figure 6 is not that informative. Could you provide TSNE plots of the resulting embeddings.

Meta-review: This paper presents an approach for learning a representation of the language model which helps it produce an embedding for a more compact representation of the language model. This allows the embedding to be used in three different tasks of interest: model routing, accuracy prediction and binary correctness prediction that bypasses running inference over a large set of benchmark tasks. Embeddings are learned through matrix factorization, where the matrix represents the performance of language models on test data points. The paper presents experiments for accuracy and latency in model routing and benchmark performance prediction. The authors claim that embeddings could reflect intrinsic model attributes, useful for identifying task-specific strengths, even in models not explicitly trained for certain tasks.


**Strengths:** The paper presents an innovative approach to streamline the evaluation of LLMs by proposing compact representations. Experiments over a large number of models are presented.

**Weaknesses:** It is natural to be concerned about the strength of the learned representations, where a single embedding is supposed to represent an entire language model. This raises questions about how well the models may generalize to new settings (other tasks), models and evaluation points.

**Reason for acceptance (poster)**: See strengths above. Despite the obvious weakness about representational capacity, the work shows improvements on the benchmarks of choice. Compact representations can be useful for a variety of use cases beyond the ones addressed in this work.

Decision: Accept (Spotlight)