ICLR 2025

Published as a conference paper at ICLR 2025
EMBEDLLM:
LEARNING
COMPACT
REPRESENTA-
TIONS OF LARGE LANGUAGE MODELS
Richard Zhuang∗Tianhao Wu∗
Zhaojin Wen
Andrew Li
Jiantao Jiao
Kannan Ramchandran
University of California, Berkeley
ABSTRACT
With hundreds of thousands of language models available on Huggingface to-
day, efficiently evaluating and utilizing these models across various downstream
tasks has become increasingly critical. Many existing methods repeatedly learn
task-specific representations of Large Language Models (LLMs), which leads to
inefficiencies in both time and computational resources. To address this, we pro-
pose EmbedLLM, a framework designed to learn compact vector representations
of LLMs that facilitate downstream applications involving many models, such as
model routing. We introduce an encoder-decoder approach for learning such em-
beddings, along with a systematic framework to evaluate their effectiveness. Em-
pirical results show that EmbedLLM outperforms prior methods in model routing
both in accuracy and latency. Additionally, we demonstrate that our method can
forecast a model’s performance on multiple benchmarks, without incurring ad-
ditional inference cost. Extensive probing experiments validate that the learned
embeddings capture key model characteristics, e.g. whether the model is special-
ized for coding tasks, even without being explicitly trained on them. We open
source our dataset, code and embedder to facilitate further research and applica-
tion: https://github.com/richardzhuang0412/EmbedLLM.
1
INTRODUCTION
Recent breakthroughs in Large Language Models (LLMs) (Vaswani et al., 2023) have led to the
creation of a vast array of models, each tailored for different use cases. These models, ranging from
small, specialized models to large, general-purpose systems (Hao et al., 2022), differ significantly
in their architecture, size, training data, and performance characteristics. For example, while some
models excel as conversational ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000029. Conference bucket: ICLR.