Mapping 1,000+ Language Models via the Log-Likelihood Vector
Authors: Momose Oyama, Hiroaki Yamagiwa, Yusuke Takase, Hidetoshi Shimodaira
Accepted at: acl25

Mapping 1,000+ Language Models via the Log-Likelihood Vector

Momose Oyama$^{1,2}$  Hiroaki Yamagiwa$^1$  Yusuke Takase$^1$  Hidetoshi Shimodaira$^{1,2}$
$^1$Kyoto University  $^2$RIKEN
oyama.momose@sys.i.kyoto-u.ac.jp, h.yamagiwa@i.kyoto-u.ac.jp,
y.takase@sys.i.kyoto-u.ac.jp, shimo@i.kyoto-u.ac.jp

Abstract

To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities. Our method is highly scalable, with computational cost growing linearly in both the number of models and text samples, and is easy to implement as the required features are derived from cross-entropy loss. Applying this method to over 1,000 language models, we constructed a “model map,” providing a new perspective on large-scale model analysis.

1 Introduction

Language models have been evolving rapidly, and their community has expanded significantly. To understand this landscape and its future directions, it is essential to systematically analyze model similarity and positioning based on language modeling principles. On the Hugging Face Hub, models are categorized by name and attributes, while other studies assess similarity based on outputs (Yax et al., 2025) or activations (Zhou et al., 2025). Leaderboards (Beeching et al., 2023; Fourrier et al., 2024; Chiang et al., 2024) are commonly used to assess model standings.

Since language models are probability models, we propose representing each model using coordinates that capture the geometric structure of the space of probability distributions. Concretely, we define a language model’s coordinates as its log-likelihood vector across a large collection of texts. Figure 1 shows a model map obtained through dimensionality reduction applied to the coordinates of 1,018 language models. This visualization reveals that models of the same type tend to cluster together, while models in close proximity often share the same primary text category, forming a continuous distribution across the map.

![Map of 1,018 language models. Their log-likelihood vectors are visualized using t-SNE. (Top) Colors indicate model types. (Bottom) Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 text categories. See Section 4 for details.](page_682_370_682_682.png)

Figure 1: Map of 1,018 language models. Their log-likelihood vectors are visualized using t-SNE. (Top) Colors indicate model types. (Bottom) Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 text categories. See Section 4 for details.
<table>
  <tr>
    <th>meta-llama/Meta-Llama-3-8B</th>
    <th>KL [bpb]</th>
    <th>google/codegemma-2b</th>
    <th>KL [bpb]</th>
    <th>deepeek-a/deepeek-llm-7b-base</th>
    <th>KL [bpb]</th>
  </tr>
  <tr>
    <td>Und95/Meta-Llama-3-8B-hf</td>
    <td>4.56e-06</td>
    <td>deepeek-a/deepeek-coder-1.3b-instruct</td>
    <td>2.46</td>
    <td>deepeek-a/deepeek-moe-16b-base</td>
    <td>0.194</td>
  </tr>
  <tr>
    <td>dftuman/Llama-3-8B-Orpo-v0.1</td>
    <td>0.0104</td>
    <td>bigcode/starcodebase-1b</td>
    <td>2.69</td>
    <td>deepeek-a/deepeek-llm-7b-chat</td>
    <td>0.364</td>
  </tr>
  <tr>
    <td>migittserra/Tess-2.0-Llama-3-8B</td>
    <td>0.0428</td>
    <td>deepeek-a/deepeek-coder-1.3b-base</td>
    <td>3.14</td>
    <td>deepeek-a/deepeek-moe-16b-chat</td>
    <td>0.368</td>
  </tr>
  <tr>
    <td>freewheelin/rlama3-dps-v0.2</td>
    <td>0.10</td>
    <td>bigcode/gpt-bigcode-santacoder</td>
    <td>3.92</td>
    <td>deepeek-a/DeepSeek-V2-Lite</td>
    <td>0.455</td>
  </tr>
  <tr>
    <td>jondurbin/bagel-8b-v1.0</td>
    <td>0.164</td>
    <td>deepeek-a/deepeek-coder-6.7b-instruct</td>
    <td>3.97</td>
    <td>deepeek-a/ESFT-vanilla-lite</td>
    <td>0.456</td>
  </tr>
  <tr>
    <td>migittserra/Llama-3-8B-Syntha-v3.5</td>
    <td>0.190</td>
    <td>Qwen/CodeQwen-1.5-TB-Chat</td>
    <td>4.09</td>
    <td>deepeek-a/DeepSeek-V2-Lite-Chat</td>
    <td>0.868</td>
  </tr>
  <tr>
    <td>nvidia/Llama-3-ChatQA-1.5-8B</td>
    <td>0.191</td>
    <td>NTQAI/Nxcode-CQ-TB-orpo</td>
    <td>4.11</td>
    <td>mistralai/Mistral-7B-Instruct-v0.1</td>
    <td>1.356</td>
  </tr>
  <tr>
    <td>nvidia/Llama-3-ChatQA-3-8B</td>
    <td>0.201</td>
    <td>bigcode/gpt-bigcode-santacoder</td>
    <td>4.24</td>
    <td>mistralai/Mistral-7B-Instruct-v0.1-full-orpo</td>
    <td>1.616</td>
  </tr>
  <tr>
    <td>FairMind/Llama-3-8B-4bit-UltraChat-Ita</td>
    <td>0.296</td>
    <td>bigcode/starcodebase-7b</td>
    <td>4.44</td>
    <td>Severian/ANIMA-Phi-Negpine-Mistral-7B</td>
    <td>1.692</td>
  </tr>
  <tr>
    <td>NousResearch/Hermes-2-Theta-Llama-3-8B</td>
    <td>0.330</td>
    <td>deepeek-a/deepeek-coder-6.7b-base</td>
    <td>4.87</td>
    <td>sethuiyer/Medichat-Llama3-8B</td>
    <td>1.718</td>
  </tr>
</table>

Table 1: Top 10 nearest neighbors among the 1,018 language models for each model listed in the first row. The values indicate the KL divergence measured in bits per byte (bpb), as defined in Section 3.5. These are computed using formula (3) in Section 2, by multiplying the original KL divergence by 0.001484. Tables for the nearest neighbors of the models labeled in the top panel of Fig. 1 are provided in Appendix I.

We find that the distances in our defined coordinate system accurately capture relationships among language models. On this map, each point represents a single model, with those having similar text-generation probability distributions appearing closer together and those with more distinct distributions positioned farther apart. In Section 2, we show that the squared Euclidean distance in this coordinate system approximates the Kullback-Leibler (KL) divergence among models. Table 1 lists the nearest neighbors for each language model. For example, many of the closest neighbors of meta-llama/Meta-Llama-3-8B (AI@Meta, 2024) also contain Llama-3 in their names.

Several studies have explored methods for comparing language models (see Appendix A). In particular, prior work on comparing generated text includes approaches that construct phylogenetic trees based on model-generated text (Yax et al., 2025) and approaches that measure differences in text-generation probabilities conditioned on given prompts using KL divergence (Melamed et al., 2024). However, these methods require generating text with each model, thus incurring the cost of pairwise distance computations, which becomes prohibitively expensive at large scale. By contrast, our method does not involve actual text generation; instead, we compute generation probabilities on a predefined text corpus. This enables us to derive model coordinates without pairwise comparisons, allowing efficient large-scale comparison of many models.

To gain insights from visualizing model attributes on the model map, in Section 4, we analyze various attributes1 and their relationships. Additionally, by comparing log-likelihood and benchmark performance, we demonstrate the ability to detect data leakage. Then, in Section 5, we treat the log-likelihood vector as a feature and show how it can predict benchmark performance. Finally, in Section 6, we validate the theoretical relationship between model coordinates and KL divergence through experiments.

2 Mapping Language Models into the Space of Text Probability Distributions

In this section, we present our proposed method. Sections 2.2 and 2.3 introduce model feature vectors derived from text-generation probabilities. Section 2.4 demonstrates that the squared Euclidean distance in the coordinate system built using these features approximates the KL divergence between models. Section 2.5 offers an interpretation of the resulting model coordinates. An extension of this method, which defines model coordinates using the sequence of conditional probabilities for generating a given token sequence, is presented in Appendix E.

2.1 Autoregressive language models

Let \( \mathcal{X} \) be the set of all possible texts, and let \( \mathcal{V} \) be the token vocabulary. A text \( x \in \mathcal{X} \) is represented as a sequence of tokens:

\[
x = (y_1, \ldots, y_n), \quad y_i \in \mathcal{V}.
\]

Denoting the maximum text length by \( n_{\max} \), we have \( \mathcal{X} = \bigcup_{n=0}^{n_{\max}} \mathcal{V}^n \). We consider a set of \( K \) language models \( \{p_i\}_{i=1}^K \). With \( y_0 \) denoting the beginning-of-sequence (BOS) token, each language model \( p_i \) predicts the next token \( y_t \) given the preceding token sequence \( y^{t-1} = (y_0, \ldots, y_{t-1}) \). Thus, the conditional probability defined by \( p_i \) is given by

\[
y_t \sim p_i(y_t \mid y^{t-1}), \quad t = 1, \ldots, n.
\]

1Attributes include model type, primary text category, model performance, model size, and creation date.
Accordingly, the probability of a text \( x \) under model \( p_i \), denoted \( x \sim p_i \), is

\[
p_i(x) = \prod_{t=1}^n p_i(y_t \mid y^{t-1}).
\]

In addition to the \( K \) language models \( p_1, \ldots, p_K \), we introduce a language model \( p_0 \) that represents an underlying distribution for theoretical purposes. We assume we have a dataset (corpus)

\[
D = (x_1, x_2, \ldots, x_N) \in \mathcal{X}^N,
\]

consisting of \( N \) texts, where each text is independently drawn from \( p_0 \).

2.2 Log-likelihood vector
For a model \( p_i \), the probability of generating a text \( x \) is denoted \( p_i(x) \). Following the convention in statistical model selection, we refer to \( p_i(x) \) as the likelihood of model \( p_i \) given the text \( x \). The log-likelihood \( \ell_i(x) = \log p_i(x) \) is then computed as

\[
\ell_i(x) = \sum_{t=1}^n \log p_i(y_t \mid y^{t-1}).
\]

In language model implementations, \( -\ell_i(x) \) corresponds to the cross-entropy loss for the text \( x \), and \( \exp(-\ell_i(x)/n) \) is known as the perplexity.

Our approach is straightforward. Given that the dataset \( D \) consists of \( N \) texts, we use the log-likelihood vector

\[
\boldsymbol{\ell}_i = (\ell_i(x_1), \ldots, \ell_i(x_N))^\top \in \mathbb{R}^N
\]

as the feature vector for model \( p_i \). The first step in our model analysis is to construct the log-likelihood matrix

\[
L = (\boldsymbol{\ell}_1, \ldots, \boldsymbol{\ell}_K)^\top \in \mathbb{R}^{K \times N}
\]

by stacking the vectors \( \boldsymbol{\ell}_i \) for the \( K \) models.

2.3 Double centering
As a preprocessing step for model analysis, we apply a technique called double centering (Borg and Groenen, 2005) to \( L \). First, we perform row-wise centering. The mean of each row, referred to as the mean log-likelihood, is given by

\[
\overline{\ell}_i = \frac{1}{N} \sum_{s=1}^N \ell_i(x_s)/N.
\]

Subtracting this value from each component of \( \boldsymbol{\ell}_i \), we define the centered log-likelihood vector \( \boldsymbol{\xi}_i = (\xi_{i1}, \ldots, \xi_{iN})^\top \in \mathbb{R}^N \), where

\[
\xi_{is} := \ell_i(x_s) - \overline{\ell}_i, \quad s = 1, \ldots, N.
\]

Next, we apply column-wise centering to the matrix of centered feature vectors \( (\boldsymbol{\xi}_1, \ldots, \boldsymbol{\xi}_K)^\top \). The mean vector is \( \bar{\boldsymbol{\xi}} = \frac{1}{K} \sum_{i=1}^K \boldsymbol{\xi}_i \), and by subtracting this vector from each \( \boldsymbol{\xi}_i \), we define the double-centered log-likelihood vector

\[
q_i = \boldsymbol{\xi}_i - \bar{\boldsymbol{\xi}}.
\]

For further details, see Appendices B and C.

2.4 Kullback-Leibler divergence
The Kullback-Leibler (KL) divergence is often used to measure how far apart two models \( p_i \) and \( p_j \) are in the space of probability distributions\footnote{\( \mathbb{E}(\cdot) \) denotes expectation and \( \mathrm{Var}(\cdot) \) denotes variance.}. It is defined as

\[
\begin{align*}
\mathrm{KL}(p_i, p_j) &= \sum_{x \in \mathcal{X}} p_i(x) \log \frac{p_i(x)}{p_j(x)} \\
&= \mathbb{E}_{x \sim p_i} (\ell_i(x) - \ell_j(x)). \tag{1}
\end{align*}
\]

We assume the dataset \( D \) is generated from an unknown underlying model \( p_0 \) and that the models \( p_i \) and \( p_j \) provide good approximations of \( p_0 \). Under this assumption, the KL divergence can be approximated as follows:

\[
2 \mathrm{KL}(p_i, p_j) \approx \mathrm{Var}_{x \sim p_0} (\ell_i(x) - \ell_j(x)). \tag{2}
\]

While the definition of KL divergence in (1) involves the expectation of \( \ell_i(x) - \ell_j(x) \), the approximation in (2) takes the form of a variance. This result is somewhat surprising yet quite insightful. Notably, although KL divergence is not symmetric in the two models, the approximation in (2) is symmetric. We estimate (2) from the dataset \( D \) as

\[
2 \mathrm{KL}(p_i, p_j) \approx \| q_i - q_j \|_2^2 / N. \tag{3}
\]

Thus, if we regard the model coordinates of \( p_i \) as \( q_i/\sqrt{N} \) by scaling with \( N \), then the squared Euclidean distance between two points approximates \( 2 \mathrm{KL}(p_i, p_j) \).

The main results, namely (2) and (3), are proved in Appendix D using the theory of exponential family of distributions (Barndorff-Nielsen, 2014; Efron, 1978, 2022; Amari, 1982), similar to the
discussion on the relationship between the norm of embeddings and KL divergence (Oyama et al., 2023). Although the concepts of model map and model coordinates have been discussed in statistics (Shimodaira, 1993; Shimodaira and Cao, 1998; Shimodaira, 2001), and there have been a few applications of model maps (Shimodaira and Hasegawa, 2005; Shimodaira and Terada, 2019), they have received little attention or use in practice.

2.5 Model coordinates

We primarily use \( q_i \) as the feature vector of model \( p_i \) and refer to it as the model coordinates\footnote{We refer to the Euclidean space where the vectors \( \ell_i, \xi_i \), or \( q_i \), possibly scaled, reside as the *log-likelihood space*.}. As shown in (3), the squared Euclidean distance in the \( q \)-coordinate system approximates the KL divergence between language models\footnote{That is, \( \|q_i - q_j\|^2 \approx 2N \mathrm{KL}(p_i, p_j) \). For simplicity, we omit the constant scaling factor in expressions of this type.}, indicating that \( q_i \) represents the position of \( p_i \) in the space of probability distributions. Since \( \xi_i \) differs from \( q_i \) only by an offset from the origin, \( \xi_i \) also serves as a model coordinate, and \( \|q_i - q_j\|^2 = \|\xi_i - \xi_j\|^2 \). However, we prefer \( q_i \) for its more interpretable components and thus adopt it throughout this paper.

For visualization purposes, we mainly use \( \ell_i \) as the coordinates of the model map, as \( \ell_i \) can be intuitively interpreted as encoding \( \sqrt{N} \ell_i \) in the “height” dimension and \( q_i \) in the “horizontal” dimensions. As shown in Appendix D.6,

\[
\| \ell_i - \ell_j \|^2 = \| q_i - q_j \|^2 + N (\bar{\ell}_i - \bar{\ell}_j)^2,
\]

which means the squared Euclidean distance in the \( \ell \)-coordinate system can be decomposed into the sum of \( 2N \mathrm{KL}(p_i, p_j) \) and \( N (\bar{\ell}_i - \bar{\ell}_j)^2 \).

3 Experimental Setup

We describe the key components of our experiment. In particular, Section 3.1 explains the procedure for selecting the 10,000 texts used to compute the model coordinates, and Section 3.2 discusses how we selected the 1,018 language models. Further details are given in Appendix G.

3.1 Selection of text data

The texts used for computing the language models’ coordinates were extracted from the Pile (Gao et al., 2020), with five categories of copyrighted material removed\footnote{https://huggingface.co/datasets/monology/pile-uncopyrighted}. This yielded a dataset \( D \) consisting of 10,000 texts, each tagged with a category label from the Pile. Figure 2 visualizes these texts.

To build the dataset, we began by dividing the first 1M texts from the Pile Uncopyrighted corpus into 1,024-byte chunks (UTF-8 encoded). In cases where decoding errors occurred, we truncated by one byte at a time. Chunks smaller than 256 bytes were discarded, resulting in about 5.7M valid chunks. From these, we randomly sampled 10,000 texts to create the final dataset used for computing model coordinates. The average length of these 10,000 texts was 972.3188 bytes.

![t-SNE Visualization of Text Embeddings](page_1012_120_388_312.png)

Figure 2: Text embeddings for 10,000 texts in dataset \( D \), computed via simcse-roberta-large (Gao et al., 2021b) and visualized with t-SNE. Colors indicate 17 text categories.

3.2 Selection of language models

We used \( K = 1,018 \) language models in total. Of these, 1,000 were selected from models listed on Open LLM Leaderboard v1. Specifically, we considered CausalLM models ranging from 1B to 13B parameters and ranked them by their number of downloads over the 30 days preceding February 1, 2025\footnote{Hugging Face’s API provides the total number of downloads in the past 30 days.}. We initially selected the top 1,100 models by download count and attempted log-likelihood calculations. Among these, 1,011 successfully produced valid log-likelihood values, and we chose the 1,000 most frequently downloaded from that set. In addition, we included 18 models from the DeepSeek language model series. We obtained information on model parameter sizes and architectures from the Leaderboard. Appendix G provides basic information on the selected models and details on how model types were defined. A complete list of models used in this study is given in Appendix L.
Figure 3: The double-centered log-likelihood matrix \( Q \), with rows and columns reordered by hierarchical clustering. Each row corresponds to one of the 1,018 models, color-coded by model type. Each column represents one of the 10,000 texts, color-coded by text category.

Figure 4: Hierarchical clustering of the top 100 most-downloaded models, based on their feature vectors \( q_i \). Model names are color-coded by model type. KL divergence is reported in units of bits per byte (bpb).

3.3 Computation of the log-likelihood

The log-likelihood matrix \( L \) was computed in float16 precision, with the bottom 2% of values clipped. This clipping mitigates the large impact of extremely low likelihoods on (3). After computing \( L \), we applied row-wise and column-wise centering to obtain the double-centered log-likelihood matrix \( Q \). Figure 3 visualizes \( Q \). Each value in the matrix can be interpreted in two ways: as the relative probability of a text for each model or as the relative likelihood of a model for each text. Both models and texts exhibit clustering patterns. Figure 4 shows a dendrogram of the top 100 models. We examined the effective dimension from the perspective of feature vector dimensionality reduction and found that the cumulative contribution ratio, based on the sum of squared singular values of \( Q \), reached 90% at 42 dimensions and 95% at 82 dimensions.

3.4 Obtaining the leaderboard scores

We obtained benchmark scores for the language models used in our experiments from Open LLM Leaderboard v1\(^7\). Between April 2023 and June 2024, this leaderboard evaluated language models on six tasks: AI2 Reasoning Challenge (ARC) (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2021a), TruthfulQA (Lin et al., 2022a), Winogrande (Sakaguchi et al., 2019), and GSM8K (Cobbe et al., 2021). Along with individual task scores, we also use the average score across all six tasks, referred to as 6-TaskMean.

3.5 Byte-normalized KL divergence for cross-experiment comparison

The KL divergence of text generation, \( \mathrm{KL}(p_i, p_j) \), generally increases with the number of tokens in the text. As a result, it cannot be directly compared with the KL divergence from other experiments using different text data. For models that use the same tokenizer, one can normalize the KL divergence by the average number of tokens in the text to obtain the KL divergence per token. However, when comparing KL divergence between models with different tokenizers, as in this study, it is more appropriate to normalize by the average text length in bytes and use the KL divergence per byte. For instance, if \( \mathrm{KL}(p_i, p_j) = 1,000 \), dividing by the average text length of 972.3188 bytes yields a KL divergence per byte of \( 1,000/972.3 = 1.028 \) nats = 1.484 bits\(^8\).

\footnotetext{7https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard}
\footnotetext{8To convert the code length unit from nats to bits, multiply by \( 1/\log 2 = 1.4427 \).}
KL divergence can be understood from the perspective of coding theory as a measure of how much longer a message becomes when encoded using an incorrect probability distribution. KL\((p_i, p_j)\) represents the extra code length required when encoding text generated from probability distribution \(p_i\) using a different distribution \(p_j\). Dividing this value by the average text length in bytes gives the additional code length per byte. In the example above, this means that, due to differences between the models, an extra 1.484 bits are needed to encode each byte of text.

4 Map of Language Models

We applied t-SNE (van der Maaten and Hinton, 2008) to the log-likelihood matrix \(L\) for dimensionality reduction9. Using this visualization, we analyze the insights gained from the model map in this section. While this paper presents model maps using \(L\), alternative maps using the double-centered log-likelihood matrix \(Q\), as well as a model map with labels for all language models, are available in Appendix H.

4.1 Visualizing attributes on the model map

Model type. The top panel of Fig. 1 visualizes the distribution of model types, with each model color-coded according to its type. We observe that models belonging to the same type tend to cluster together, forming distinct regions on the map (e.g., llama-2, mistral, and gemma). In particular, models optimized for coding tasks10 appear in a relatively compact region, suggesting that these models share notable similarities in their probability distributions.

Text category. The bottom panel of Fig. 1 shows the model map with each model color-coded according to the text category in which it achieves the highest standardized log-likelihood score (Appendix G.4). From this figure, we see that models exhibiting high likelihoods for the same text category are grouped together. Notably, the cluster containing coding-specialized models in the top panel aligns with the GitHub/StackExchange region in the bottom panel, suggesting that these models have relatively high likelihoods for text originating from GitHub and StackExchange.

Model performance. Figure 5 visualizes two evaluation metrics: mean log-likelihood and benchmark task performance. From the left and central panels, we see that both metrics exhibit similar trends on the map, where models that lie close together tend to show similar metric values. Additionally, in the right panel, the GSM8K/MMLU region corresponds to the ArXiv/PubMed Central region in the bottom panel of Fig. 1, suggesting that models with high likelihoods on academic and scientific texts also tend to perform well on mathematical reasoning and academic knowledge-intensive tasks.

Model size and creation date. Figure 6 shows the distribution of models by size and creation date. Compared to Fig. 5, newer models generally perform better, but model size does not always correlate with performance, as some smaller models perform comparably to larger ones.

4.2 Detection of data leakage

Since the text data we used was extracted from the Pile corpus, models that were pre-trained on the Pile are likely to exhibit higher log-likelihood values than their actual capabilities measured by benchmarks. We analyze this effect using the model map in Fig. 7. The left panel highlights models that used the Pile for pre-training. The right panel shows models with high mean log-likelihood relative to their 6-TaskMean score. The alignment between these two distributions suggests that models pre-trained on the Pile tend to achieve higher likelihoods on our text data, while their benchmark performance remains comparatively lower.

5 Predicting Model Performance from Model Coordinates

As shown in the central panel of Fig. 5, the positioning of models on the map suggests that a model’s benchmark performance may be inferred from its coordinates. In this section, we conduct a regression analysis using the q-coordinates to predict benchmark scores and evaluate predictive performance.

5.1 Benchmark scores and models

We use six benchmark scores from Open LLM Leaderboard v1, as described in Section 3.4. Our
Figure 5: Model maps illustrating model performance. From left to right, the panels show each model’s mean log-likelihood, 6-TaskMean score, and the “primary task,” meaning the task for which each model achieves its highest standardized score among the six tasks (Appendix G.4). The color bar is clipped at the 10th percentile for mean log-likelihood and 6-TaskMean, with darker colors indicating better performance. In the primary task panel, models with standardized scores below zero on all six tasks are labeled “All Under 0.”

Figure 6: Model maps color-coded by (Left) number of parameters and (Right) model creation date.

Figure 7: (Left) Models tagged with “the Pile.” (Right) Difference between the standardized mean log-likelihood and the standardized 6-TaskMean score.

experiments are conducted on 996 models for which these benchmark scores are available11.

5.2 Setting for regression analysis

For each benchmark task, the dataset is given as \( \{ (q_1, v_1), \ldots, (q_K, v_K) \} \), where \( q_i \in \mathbb{R}^N \) is the double-centered log-likelihood vector of the language model \( p_i \), and \( v_i \in [0, 100] \) is its corresponding benchmark score. We use ridge regression to predict each benchmark score. Let \( Q \in \mathbb{R}^{K \times N} \) be the matrix of explanatory variables, and let \( v = (v_1, \ldots, v_K)^T \in \mathbb{R}^K \) be the vector for the target variable. The objective function with parameter \( w \in \mathbb{R}^N \) is given by:

\[
\mathcal{L}(w) = \| v - Qw \|_2^2 + \alpha \| w \|_2^2,
\]

where \( \alpha \in \mathbb{R}_{>0} \) is a hyperparameter that controls the strength of regularization. Since the number of variables \( N \) is much larger than the sample size \( K \) (\( N \gg K \)), making this a high-dimensional regression setting, we carefully set \( \alpha \) using cross-validation to avoid overfitting.

We partition the models into five folds based on model types and perform parameter training and benchmark score prediction12. To mitigate the effect of randomness, we repeat the data splitting with five different seeds and take the average of the predictions as the final predicted score. As evaluation metrics, we compute Pearson’s \( r \) and Spearman’s \( \rho \) to measure the correlation between the predicted and benchmark scores. Additionally, we conduct experiments by replacing the target variable with 6-TaskMean and mean log-likelihood, leading to a total of eight experimental settings. See Appendix K.1 for details.

11From the 1,018 models, we excluded four that lacked TruthfulQA scores and 18 from deepseek-ai, as their benchmark scores were incomplete due to the models being relatively new, leaving 996 for analysis. For simplicity, we denote the number of models as \( K \).

12We used the GroupKFold and RidgeCV implementations provided by scikit-learn (Varoquaux et al., 2015).

5.3 Results and discussion

Regression analysis. Table 2 summarizes the results of the regression analysis using \( q \)-coordinates. Across all benchmark tasks, both Pearson’s \( r \) and
<table>
  <tr>
    <th></th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Winogrande</th>
    <th>GSM8K</th>
    <th>6-TaskMean</th>
    <th>mean log-likelihood</th>
  </tr>
  <tr>
    <td>Pearson’s \( r \)</td>
    <td>0.946</td>
    <td>0.909</td>
    <td>0.932</td>
    <td>0.901</td>
    <td>0.941</td>
    <td>0.884</td>
    <td>0.953</td>
    <td>0.989</td>
  </tr>
  <tr>
    <td>Spearman’s \( \rho \)</td>
    <td>0.948</td>
    <td>0.956</td>
    <td>0.934</td>
    <td>0.884</td>
    <td>0.948</td>
    <td>0.857</td>
    <td>0.960</td>
    <td>0.974</td>
  </tr>
</table>

Table 2: Results of ridge regression for predicting benchmark scores from model coordinates. Predictions for 6-TaskMean and mean log-likelihood are also included. High correlation coefficients are observed across all settings. See Table 7 in Appendix K for the mean and standard deviation of correlation coefficients across the five data splits.

<table>
  <tr>
    <th></th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Winogrande</th>
    <th>GSM8K</th>
    <th>6-TaskMean</th>
  </tr>
  <tr>
    <td>Pearson’s \( r \)</td>
    <td>0.453</td>
    <td>0.598</td>
    <td>0.346</td>
    <td>0.072</td>
    <td>0.508</td>
    <td>0.239</td>
    <td>0.395</td>
  </tr>
  <tr>
    <td>Spearman’s \( \rho \)</td>
    <td>0.432</td>
    <td>0.467</td>
    <td>0.422</td>
    <td>0.048</td>
    <td>0.512</td>
    <td>0.364</td>
    <td>0.400</td>
  </tr>
</table>

Table 3: Correlation between mean log-likelihoods and benchmark scores. The results show that perplexity has only moderate correlations on most tasks, especially compared to model coordinates in Table 2.

![Scatter plot comparing predicted and actual 6-TaskMean scores for test sets (the dashed line indicates the identity line). Pearson’s r is 0.953 and Spearman’s ρ is 0.960, indicating strong predictive performance. Points are color-coded by the mean log-likelihood, with the color scale clipped at the 10th percentile. While higher mean log-likelihood values tend to correspond to higher benchmark scores, a few models with unusually high log-likelihoods due to data leakage (Section 4.2) deviate from this trend. Nevertheless, the predictions remain accurate overall. Scatter plots for individual benchmark tasks are shown in Fig. 18 in Appendix K.2.](page_184_682_495_388.png)

Figure 8: Scatter plot comparing predicted and actual 6-TaskMean scores for test sets (the dashed line indicates the identity line). Pearson’s \( r \) is 0.953 and Spearman’s \( \rho \) is 0.960, indicating strong predictive performance. Points are color-coded by the mean log-likelihood, with the color scale clipped at the 10th percentile. While higher mean log-likelihood values tend to correspond to higher benchmark scores, a few models with unusually high log-likelihoods due to data leakage (Section 4.2) deviate from this trend. Nevertheless, the predictions remain accurate overall. Scatter plots for individual benchmark tasks are shown in Fig. 18 in Appendix K.2.

Spearman’s \( \rho \) are consistently high, indicating that the ridge regression model based on model coordinates achieves strong predictive performance. This trend holds not only for individual tasks but also for the 6-TaskMean, where the correlation coefficients remain high, as illustrated in the scatter plot in Fig. 8. Even when the target variable is the mean log-likelihood, defined as the simple average over the 10,000 texts, the regression model still achieves high correlation. This suggests that the model coordinates, despite being double-centered and excluding direct information about the mean, retain a sufficiently rich structure to accurately predict the average log-likelihood.

Performance prediction based on perplexity. The mean log-likelihood equals \( -\log(\text{perplexity}) \) and is often used as a performance indicator. We evaluated its correlation with benchmark scores, and Table 3 shows moderate correlations across all tasks, consistent with prior findings (Liu et al., 2023a; Fang et al., 2025; Thrush et al., 2025). Unlike mean log-likelihood, which uniformly averages over texts, regression using the log-likelihood matrix \( L \) can assign task-dependent weights. As shown in Table 9 in Appendix K, its prediction accuracy is nearly the same as that of \( Q \). This indicates that combining \( Q \) and the mean in \( L \) offers little improvement. Accurate prediction with \( Q \) alone suggests that model positions already encode a structure aligned with benchmark performance.

6 Empirical Validation of Theory

In Section 2, we discussed model coordinates for text probability models. Appendix E extends this framework to token-sequence probability models. These theoretical results state that the squared Euclidean distance in the log-likelihood space approximates the KL divergence between models. To validate this, we first evaluate token-level conditional probability models (Section 6.1), since token-level experiments are generally easier to conduct than text-level experiments. We then extend our analysis to text probability models (Section 6.2). Additionally, in Section 6.3, we explore the relationship between model weight parameters and model coordinates.

6.1 Validation of (32) for token-level models

Settings. Two models with a shared tokenizer Llama-2-7b-hf and Llama-2-7b-chat-hf (Touvron et al., 2023b), denoted as \( p_1 \) and \( p_2 \), were used. Following the method described in Appendix E.1, we computed the model coordinates \( \zeta_1(x) \) and \( \zeta_2(x) \) for each text \( x = (y_1, \cdots, y_n) \in D \), where
these coordinates are centered vectors with elements \( \log p_i(y_t|y^{t-1}) \) as defined in (28). We then calculated the squared Euclidean distance between these coordinates, \( \| \zeta_1(x) - \zeta_2(x) \|^2 \). To obtain the exact KL divergence between models, we used the outputs of the softmax function in the language models. We computed the sum of per-token KL divergences: \( \sum_{t=1}^n \mathrm{KL}(p_1(y_t|y^{t-1}), p_2(y_t|y^{t-1})) \), which is also used in Lv et al. (2023).

Results and discussion. The left panel of Fig. 9 shows a scatter plot of the squared Euclidean distance and KL divergence for \( x \in D \), with a Pearson’s correlation coefficient of \( r = 0.893 \). This result indicates that (32) provides a good approximation in actual language models.

6.2 Validation of (3) for text-level models

Settings. Among the 292 language models sharing the tokenizer with Llama-2-7b-hf (Touvron et al., 2023b), we excluded the five models with the largest values of \( \sum_{x \in D} \| \zeta_i(x) \|^2 \), leaving 287 models for our experiment. Using the approach described in Section 2, we computed the model coordinates for each model and calculated the squared Euclidean distance \( \| q_i - q_j \|^2 \) between every pair of models. Because it is extremely difficult to directly compute \( \mathrm{KL}(p_i, p_j) \), we instead used \( \frac{1}{N} \sum_{x \in D} \| \zeta_i(x) - \zeta_j(x) \|^2 \) as a proxy. For a theoretical justification that this quantity approximates \( \mathrm{KL}(p_i, p_j) \), see (33) in Appendix E.

Results and discussion. As shown in the right panel of Fig. 9, the scatter plot of squared Euclidean distance versus KL divergence exhibits a Pearson’s correlation coefficient of \( r = 0.904 \). This finding confirms that the relationship in (3) holds approximately in practical language models.

6.3 Relationship between model weights and model coordinates

For language models with the same architecture, comparison can also be conducted via weight parameters. We investigated how the structure of the log-likelihood space aligns with the structure of the weight-parameter space. We generated 36 new language models by linearly interpolating the weights of Llama-2-7b-hf, Llama-2-7b-chat-hf (Touvron et al., 2023b), and vicuna-7b-v1.5 (Zheng et al., 2023), using a \( 6 \times 6 \) grid of coefficients. For these 36 models, we computed text-generation log-likelihoods and visualized the resulting model coordinates in two dimensions using Principal Component Analysis (PCA). Figure 10 shows these 36 models in both weight space and log-likelihood space, where the two-dimensional grid structure is preserved. Interestingly, the mean log-likelihoods of the interpolated models closely follow the linear interpolation of the three base models’ mean log-likelihoods. This suggests that such interpolation can be a practical tool for exploring high-performing models. See Appendix J for details.

Figure 9: Relationship between the squared Euclidean distance of model coordinates and KL divergence. (Left) In the token-level experiment (Section 6.1), each point represents a text. (Right) In the text-level experiment (Section 6.2), each point represents a pair of models.

![Scatter plots showing the relationship between squared Euclidean distance and KL divergence for token-level and text-level experiments](page_1012_120_388_246.png)

Figure 10: Visualization of 36 language models obtained by linearly interpolating pretrained model weights based on Llama-2-7b-hf. Each point is color-coded according to its mean log-likelihood. (Left) Models in the weight parameter space. (Right) Models in the log-likelihood space, represented by the \( q \)-coordinate system.

![PCA visualization of 36 language models in weight and log-likelihood spaces](page_1012_420_388_246.png)

7 Conclusion

We proposed a method to compare autoregressive language models using log-likelihoods on a predefined text set. By interpreting these as model coordinates, we showed that the squared Euclidean distance approximates KL divergence. Experiments on over 1,000 models confirmed the method’s effectiveness for analyzing model relationships, predicting benchmark performance, and validating the theory.
Limitations

• Changing the text data will alter the analysis results of the model map. This is both a limitation and an advantage of the proposed method, because it allows us to choose text data according to the analysis objective. For example, if we want to investigate code-focused language models in more detail, we can increase the proportion of code data from GitHub, thereby increasing the resolution of code-focused models on the model map.

• The proof (Section 2 and Appendix D) that the squared Euclidean distance in the model coordinate system approximates the KL divergence assumes that the language model’s text-generation probabilities closely match the distribution of the text data. When this assumption does not hold, the approximation accuracy decreases. However, even under such circumstances, the model coordinates should still function sufficiently as model features.

• If the text data used for the model coordinates is contained in a language model’s pre-training corpus, that model’s mean log-likelihood may be overestimated. This data leakage, or data contamination, is generally non-negligible, as shown in Fig. 7 in Section 4.2, which illustrates the effect of using the Pile corpus. However, comparing it against benchmark scores makes it possible to detect such data leakage, and one can remove models that are affected. Furthermore, the model map based on the \( q \)-coordinate system is robust to contamination in the data, as the estimation of KL divergence using the squared Euclidean distance in the \( q \)-coordinate system remains valid even if the true generative model \( p_0 \) that produces the dataset \( D \) varies, as long as all compared models remain sufficiently close to \( p_0 \). For instance, even if \( D \) is generated from one of the \( K \) models, such as \( p_i \), the estimation formula (3) remains correct (Appendix D.7).

• Beyond the data leakage mentioned above, other systematic errors introduced into the model coordinates can also affect the model map. As noted in Appendix C, \( \ell_i \) and \( \bar{\ell}_i \) are susceptible to systematic biases. However, thanks to double centering, \( q_i \) is less influenced by bias terms.

• Although the calculation of model coordinates is linear time \( O(KN) \) in the number of models \( K \) and the number of texts \( N \), it still requires a non-negligible amount of computation. In our experiments, it took about 10 minutes on a single GPU (RTX 6000 Ada) to compute coordinates (\( N = 10^4 \), float16) for a single 7B model.

• Computing the model map visualization from the model coordinates is generally not linear in \( K \). For example, t-SNE requires a distance matrix, incurring \( O(K^2N) \) computational cost. However, in modern computing environments, as long as \( K \) is not extremely large (e.g., in the millions), the cost of visualization is negligible compared to the cost of calculating the model coordinates.

• A sufficiently large number of text samples, \( N \), is desirable for the model coordinates. We used \( N = 10^4 \). Since the error in the KL divergence estimate due to randomness decreases proportionally to \( N^{-1/2} \), \( N \) must be increased according to the desired resolution of the model map.

• When using model coordinates as feature vectors, \( N = 10^4 \) can be unwieldy. According to the experiment in Section 3.3, applying PCA to reduce the dimensionality of \( q \)-coordinates to 82 dimensions still retains 95% of the information. However, the predictive performance of such dimension-reduced features has not yet been tested.

• Since the language models used in our experiments were obtained from the Open LLM Leaderboard v1 (which ran from April 2023 to June 2024), our discussion of models released after June 2024 is limited.

• The results of the task-performance prediction in Section 5 should be interpreted conservatively. We employed a proper cross-validation, splitting the models based on their model types so that the training, validation, and test sets were disjoint, thereby limiting data leakage. However, if nearly identical models appear in both the training and test partitions (for example, models labeled with different types but built on the same base model and
modified only slightly by fine-tuning), prediction becomes artificially easier. As shown in Tables 7 and 8 in Appendix K.3, random splits that permit such leakage yield higher predictive performance than splits that strictly follow model-type groupings. Finally, note that models not listed on the leaderboard were excluded from our evaluation.

• The theoretical validation experiment in Section 6 is limited. Currently, it is difficult to directly compute exact KL divergence values among text-generation probability models, so conducting more precise validation experiments remains a future challenge.

• In the method of computing model coordinates from token-sequence conditional probabilities (Appendix E and Appendix F), the proof that the squared Euclidean distance in the model coordinate system approximates the KL divergence requires additional assumptions (Appendix F.3). In practice, Assumption 2 does not hold, and due to the variations in (39), \( \| \zeta_i - \zeta_j \|^2 \) in (32) tends to overestimate the KL divergence. Nonetheless, even in such situations, token-level model coordinates will still likely function sufficiently as features.

Acknowledgments

This study was partially supported by JSPS KAKENHI 22H05106, 23H03355, JST CREST JP-MJCR21N3, JST BOOST JPMJBS2407.

References

01. AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, and 13 others. 2025. Yi: Open foundation models by 01.ai. Preprint, arXiv:2403.04652.

42dot Inc. 2023. 42dot IIm: A series of large language model by 42dot.

Rishiraj Acharya. 2023. Catppt.

AI@Meta. 2024. Llama 3 model card.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. Preprint, arXiv:2305.13245.

AI@Waktaverse. 2024. Waktaverse llama 3 model card.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cococaru, Merouane Debba, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance.

Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks. Preprint, arXiv:2402.17733.

Shun-Ichi Amari. 1982. Differential Geometry of Curved Exponential Families-Curvatures and Information Loss. The Annals of Statistics, 10(2):357 – 385.

Shun-Ichi Amari. 1998. Natural gradient works efficiently in learning. Neural computation, 10:251–276.

Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. 2023. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo.

Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. 2021. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. Preprint, arXiv:2310.11511.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. Preprint, arXiv:2108.07732.

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. Llemma: An open language model for mathematics. Preprint, arXiv:2310.10631.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. Preprint, arXiv:1607.06450.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, and 29 others. 2023. Qwen technical report. Preprint, arXiv:2309.16609.
Abhinand Balachandran. 2023. Tamil-llama: A new tamil language model based on llama 2. Preprint, arXiv:2311.05845.

Ole Barndorff-Nielsen. 2014. Information and exponential families: in statistical theory. John Wiley & Sons.

Pierpaolo Basile, Elio Musacchio, Marco Polignano, Lucia Siciliani, Giuseppe Fiameni, and Giovanni Semeraro. 2023. Llamantino: Llama 2 models for effective text generation in italian language. Preprint, arXiv:2312.09993.

Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle. Preprint, arXiv:2207.14255.

Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sansevierio, Lewis Tunstall, and Thomas Wolf. 2023. Open Ilm leaderboard.

Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. Preprint, arXiv:2004.05150.

Leonard Bereska and Stratis Gavves. 2024. Mechanistic interpretability for AI safety - a review. Transactions on Machine Learning Research. Survey Certification. Expert Certification.

Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. Preprint, arXiv:2201.07311.

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Afzal Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Preprint, arXiv:2304.01373.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. Preprint, arXiv:1911.11641.

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregressive language model. Preprint, arXiv:2204.06745.

Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata.

Ingwer Borg and Patrick J. F. Groenen. 2005. Modern Multidimensional Scaling: Theory and Applications, 2 edition. Springer Series in Statistics. Springer, New York, NY.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. Preprint, arXiv:2005.14165.

CeADAR. 2023. Financeconnect-13b (revision 5f7841d).

Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation.

Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024a. Alpagasus: Training a better alpaca with fewer data. Preprint, arXiv:2307.08701.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. 2023a. Symbolic discovery of optimization algorithms. Preprint, arXiv:2302.06675.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024b. Longlora: Efficient fine-tuning of long-context large language models. Preprint, arXiv:2309.12307.

Yukang Chen, Shaozuo Yu, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. Long alpaca: Long-context instruction-following models.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolaos Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. Preprint, arXiv:2403.04132.
chilu. 2023. Mamba-gpt-3b-v4.

François Chollet. 2019. On the measure of intelligence. Preprint, arXiv:1911.01547.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. Preprint, arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.

CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Choo, Jingyue Shen, Joe Kelley, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, and 8 others. 2024. Codegemma: Open code models based on gemma. Preprint, arXiv:2406.11409.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm.

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yanakai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Ultrafeedback: Boosting language models with scaled ai feedback. Preprint, arXiv:2310.01377.

Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. Preprint, arXiv:2401.06066.

Alpin Dale, Wing Lian, Bley Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and “Teknium”. 2023. Llongcora13b: Llama2-13b model instruct-tuned for long context on filtered openorca1 v gpt-4 dataset.

Tri Dao. 2023. FlashAttention-2: Faster attention with better parallelism and work partitioning.

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Preprint, arXiv:2205.14135.

Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. 2024. Griffin: Mixing gated linear recurrences with local attention for efficient language models. Preprint, arXiv:2402.19427.

DeciAI Research Team. 2023. Decilm-7b-instruct.

DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kange Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, and 68 others. 2024a. Deepseek llm: Scaling open-source language models with longtermism. Preprint, arXiv:2401.02954.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, and 138 others. 2024b. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. Preprint, arXiv:2405.04434.

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. Preprint, arXiv:2110.02861.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qloara: Efficient finetuning of quantized llms. Preprint, arXiv:2305.14314.

Peter Devine. 2024. Tagengo: A multilingual chat dataset. Preprint, arXiv:2405.12612.

Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. Preprint, arXiv:2101.11718.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. Preprint, arXiv:2305.14233.

Duy Quang Do, Hoang Le, and Duc Thang Nguyen. 2023. Torolama: The vietnamese instruction-following and chat model.
Bradley Efron. 1978. The geometry of exponential families. The Annals of Statistics, 6:362–376.

Bradley Efron. 2022. Exponential Families in Theory and Practice. Cambridge University Press.

Kawin Ethayarajah, Winnie Xu, Dan Jurafsky, and Douwe Kiela. 2023. Human-centered loss functions (halos).

Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, and Yisen Wang. 2025. What is wrong with perplexity for long-context language modeling? In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR).

Clémentine Fourrier, Nathan Habib, Alina Lozovskaya-Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.

Victor Gallego. 2024. Configurable safety tuning of language models with synthetic preference data. Preprint, arXiv:2404.00495.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027.

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muenninghoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021a. A framework for few-shot language model evaluation.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. Preprint, arXiv:2009.11462.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, and 1331 others. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.

Xinyang Geng and Hao Liu. 2023. Openllama: An open reproduction of llama.

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. Tora: A tool-integrated reasoning agent for mathematical problem solving. Preprint, arXiv:2309.17452.

Diego Granzioi, Stefan Zohren, and Stephen Roberts. 2021. Learning rates as a function of batch size: A random matrix theory approach to neural network training. Preprint, arXiv:2006.09092.

Griffin Team, Soham De, Samuel L Smith, Anushan Fernando, Alex Botev, George-Christian Muraru, Ruba Haroun, and Leonard Berrada et al. 2024. Recurrent-gemma.

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. Olmo: Accelerating the science of language models. Preprint, arXiv:2402.00838.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. Preprint, arXiv:2401.14196.

Jan Philipp Harries. 2023. orca_mini_v2_ger_7b: An explain tuned llama-7b model based on orca mini v2 and adapted to german language.

Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. Preprint, arXiv:2203.09509.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.

Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. 2020. Query-key normalization for transformers. Preprint, arXiv:2010.04245.

Jiwoong Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. Preprint, arXiv:2403.07691.

Eliahu Horwitz, Nitzan Kurier, Jonathan Kahana, Liel Amar, and Yedid Hoshen. 2025. Charting and navigating hugging face’s model atlas. Preprint, arXiv:2503.10633.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. Preprint, arXiv:1902.00751.

Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. 2024. Breeze-7b technical report. Preprint, arXiv:2403.02712.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.

IDEA-CCNL. 2021. Fengshenbang-lm.

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations.

"interstellarninja", "Teknium", "theemozilla", "karan4d", and "huemin_art". 2024. Hermes-2-pro-mistral-7b.

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaiov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. 2023. Opt-impl: Scaling language model instruction meta learning through the lens of generalization. Preprint, arXiv:2212.12017.

Neel Jain, Ping yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somapalli, Brian R. Bartoldson, Bhavya Kaikkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Neftune: Noisy embeddings improve instruction finetuning. Preprint, arXiv:2310.05914.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressaud, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhui Chen. 2024. Tigerscore: Towards building explainable metric for all text generation tasks. Preprint, arXiv:2310.00752.

Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. Preprint, arXiv:1705.05551.

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunus Kim, Sanghoon Kim, and Chanjun Park. 2024a. sdpo: Don’t use your data all at once. Preprint, arXiv:2403.19270.

Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunus Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024b. Solar 10.7b: Scaling large language models with simple yet effective depth upscaling. Preprint, arXiv:2312.15166.

Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. 2022. Kobest: Korean balanced evaluation of significant tasks. Preprint, arXiv:2204.04541.

Seungduk Kim, Seuntaek Choi, and Myeongho Jeong. 2024c. Efficient and effective vocabulary expansion towards multilingual large language models. Preprint, arXiv:2402.14714.

Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Jiwing Hyun, Sungho Park, and Kyubyong Park. 2023. A technical report for polyglot-ko: Open-source large-scale korean language models. Preprint, arXiv:2306.02254.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yūsuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916.

Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2023. Sparse upcycling: Training mixture-of-experts from dense checkpoints. Preprint, arXiv:2212.05055.

Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, and Edward Choi. 2024. Publicly shareable clinical large language model built on synthetic clinical notes. Preprint, arXiv:2309.00237.

Andreas Köpf, Yannis Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations – democratizing large language model alignment. Preprint, arXiv:2304.07327.

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. Biomistral: A collection of open-source pretrained large language models for medical domains. Preprint, arXiv:2402.10373.

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying
the carbon emissions of machine learning. Preprint, arXiv:1910.09700.

Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2024. Platypus: Quick, cheap, and powerful refinement of llms. Preprint, arXiv:2308.07317.

Ariel N. Lee, Cole J. Hunter, Nataniel Ruiz, Bleys Goodson, Wing Lian, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorcaplatypus: Llama2-13b model instruct-tuned on filtered openorca1 gpt-4 dataset and merged with divergent stem and logic dataset model.

Junbum Lee. 2023. llama-2-ko-7b (revision 4a9993e).

Junbum Lee. 2024a. Llama-3-koen.

Junbum Lee. 2024b. Yi-ko-6b (revision 205083a).

Jungwon Lee and Seungjun Ahn. 2024. Llama-3-instruction-constructionsafety-layerturning.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. Preprint, arXiv:2005.01643.

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zhetelnozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, and 48 others. 2023a. Starcoders: may the source be with you! Preprint, arXiv:2305.06161.

Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. 2023b. Colossal-ai: A unified deep learning system for large-scale parallel training. Preprint, arXiv:2110.14883.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023c. Textbooks are all you need ii: phi-1.5 technical report. Preprint, arXiv:2309.05463.

Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023d. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Preprint, arXiv:2303.14070.

Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023a. Openorca_preview1: A llama-13b model fine-tuned on small portion of openorca1 dataset.

Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023b. Jackalope 7b: Mistral-7b model multi-turn chat tuned on filtered openorca1 gpt-4 dataset.

Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023c. Llongorca7b: Llama2-7b model instruct-tuned for long context on filtered openorca1 gpt-4 dataset.

Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023d. Mistralorca: Mistral-7b model instruct-tuned on filtered openorca1 gpt-4 dataset.

Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023e. Mistralslimorca: Mistral-7b model instruct-tuned on filtered, corrected, openorca1 gpt-4 dataset.

Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023f. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification.

Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, "Teknium", and Nathan Hoos. 2023g. Slimorca dedup: A deduplicated subset of slimorca.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022a. Truthfulqa: Measuring how models mimic human falsehoods. Preprint, arXiv:2109.07958.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettle-moyer, Zornitsa Kozareva, Mona Diab, and 2 others. 2022b. Few-shot learning with multilingual language models. Preprint, arXiv:2112.10668.

Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2025. World model on million-length video and language with blockwise ringattention. Preprint, arXiv:2402.08268.

Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023a. Same pre-training loss, better downstream: Implicit bias matters for language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 22208–22242. PMLR.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b. Lost in the middle: How language models use long contexts. Preprint, arXiv:2307.03172.

Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Chatqa: Surpassing gpt-4 on conversational qa and rag. Preprint, arXiv:2401.10225.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. Preprint, arXiv:2301.13688.

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtiar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, and 47 others. 2024. Starcoder 2 and the stack v2: The next generation. Preprint, arXiv:2402.19173.

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. Preprint, arXiv:2308.09583.

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. Preprint, arXiv:2306.08568.

Xingtai Lv, Ning Ding, Yujia Qin, Zhiyuan Liu, and Maosong Sun. 2023. Parameter-efficient weight ensembling facilitates task-level knowledge transfer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 270–282. Toronto, Canada: Association for Computational Linguistics.

Manuel Romero. 2023. llama-2-coder-7b (revision d30d193).

Pankaj Mathur. 2023a. orca_mini_v2_7b: An explain tuned llama-7b model on uncensored wizardlm, alpaca, & dolly datasets.

Pankaj Mathur. 2023b. orca_mini_v3_13b: An orca style llama2-70b model.

Pankaj Mathur. 2023c. orca_mini_v3_7b: An explain tuned llama2-7b model.

Rimon Melamed, Lucas Hurley McCabe, Tanay Wakhare, Yejin Kim, H. Howie Huang, and Enric Boix-Adserà. 2024. Prompts have evil twins. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 46–74, Miami, Florida, USA. Association for Computational Linguistics.

Sean Meyn and Richard L. Tweedie. 2009. Markov Chains and Stochastic Stability, 2nd edition. Cambridge Mathematical Library. Cambridge University Press.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. Preprint, arXiv:1809.02789.

Xu Ming. 2023. textgen: Implementation of language model finetune.

Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. 2023. Orca 2: Teaching small language models how to reason. Preprint, arXiv:2311.11045.

Koh Mitsuda, Xinqi Chen, Toshiaki Wakatsuki, and Kei Sawada. 2024. rinna/llama-3-youko-8b.

MosaicML NLP Team. 2023a. Introducing mpt-30b: Raising the bar for open-source foundation models. Accessed: 2023-06-22.

MosaicML NLP Team. 2023b. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-03-28.

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2025. Generative representational instruction tuning. Preprint, arXiv:2402.09906.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. Preprint, arXiv:2211.01786.

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. Preprint, arXiv:2306.02707.

Xavier Murias. 2023. Cybertron: Uniform neural alignment.

Nexusflow.ai team. 2023. Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling.

Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. SealLms – large language models for southeast asia. Preprint, arXiv:2312.00738.

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. Preprint, arXiv:2203.13474.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023. Capabilities of gpt-4 on medical challenge problems. Preprint, arXiv:2303.13375.
Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, SpeakLeash Team, and Cyfronet Team. 2024a. Introducing bielik-7b-v0.1: Polish language model. Accessed: 2024-04-01.

Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Sebastian Kondracki, SpeakLeash Team, and Cyfronet Team. 2024b. Introducing bielik-7b-instruct-v0.1: Instruct polish language model. Accessed: 2024-04-01.

Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwoździej, and Remigiusz Kinas. 2024c. Bielik 7b v0.1: A polish language model – development, insights, and evaluation. Preprint, arXiv:2410.18565.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltusc, Haiming Bao, Mohammad Bavarian, Jeff Belgium, and 262 others. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Momose Oyama, Sho Yokoi, and Hideotsu Shimodaira. 2023. Norm of word embedding encodes information gain. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2108–2130, Singapore. Association for Computational Linguistics.

Ankit Pal and Malaikannan Sankarasubbu. 2024a. Gemini goes to med school: Exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations. Preprint, arXiv:2402.07023.

Ankit Pal and Malaikannan Sankarasubbu. 2024b. Openbiolms: Advancing open-source large language models for healthcare and life sciences.

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddhartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimization with dpo-positive. Preprint, arXiv:2402.13228.

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. Preprint, arXiv:2110.08193.

Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. 2024. Llama 3 gradient: A series of long context models.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Alamzouree, and Julien Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. Preprint, arXiv:2306.01116.

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023a. Instruction tuning with gpt-4. Preprint, arXiv:2304.03277.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shipppole. 2023b. Yarn: Efficient context window extension of large language models. Preprint, arXiv:2309.00071.

Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, and Nathan Cooper. 2024. Stable code 3b.

Kunat Pipatanakul, Phattrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarmmongkol, Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Thampipitchai. 2023. Typhoon: Thai large language models. Preprint, arXiv:2312.13951.

Marco Polignano, Pierpaolo Basile, and Giovanni Semeraro. 2024. Advanced natural-based interaction for the italian language: Llamantino-3-anita. Preprint, arXiv:2405.07101.

Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. Preprint, arXiv:2309.15088.

Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. Preprint, arXiv:2108.12409.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Preprint, arXiv:2305.18290.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. Preprint, arXiv:1910.02054.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafori, Wenhan Xiong, Alexandre Défossez, and 7 others. 2024. Code llama: Open foundation models for code. Preprint, arXiv:2308.12950.

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. Preprint, arXiv:1804.09301.

Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2024. Gollie: Annotation guidelines improve zero-shot information-extraction. Preprint, arXiv:2310.03668.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. Preprint, arXiv:1904.09728.

Akira Sasaki, Masato Hirakawa, Shintaro Horie, and Tomoaki Nakamura. 2023a. Elyza-japanese-llama-2-7b.

Akira Sasaki, Masato Hirakawa, Shintaro Horie, Tomoaki Nakamura, Sam Passaglia, and Daisuke Oba. 2023b. Elyza-japanese-llama-2-13b.

Ryoma Sato. 2022. Word tour: One-dimensional word embeddings via the traveling salesman problem. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2166–2172. Association for Computational Linguistics.

Kei Sawada, Tianyu Zhao, Makoto Shing, Kentaro Mitsui, Akio Kaga, Yukiya Hono, Toshiaki Watanabe, and Koh Mitsuda. 2024. Release of pretrained models for the japanese language. Preprint, arXiv:2404.01657.

Zhihong Shao, Peiyi Wang, Qihaq Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.

Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra. 2023. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. Preprint, arXiv:2312.13558.

Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. Preprint, arXiv:1911.02150.

Noam Shazeer. 2020. Glu variants improve transformer. Preprint, arXiv:2002.05202.

Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Gergely Szilvasy, Rich James, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Scott Yih, and Mike Lewis. 2024. In-context pretraining: Language modeling beyond document boundaries. Preprint, arXiv:2310.10638.

Hidetoshi Shimodaira. 1993. A model search technique based on confidence set and map of models [モデルの信頼集合と地図によるモデル探索] (in japanese). Proceedings of the Institute of Statistical Mathematics, 41(2):131–147.

Hidetoshi Shimodaira. 2001. Multiple comparisons of log-likelihoods and combining nonnested models with applications to phylogenetic tree selection. Communications in Statistics-Theory and Methods, 30(8-9):1751–1772.

Hidetoshi Shimodaira and Ying Cao. 1998. A graphical technique for model selection diagnosis. The Institute of Statistical Mathematics Research Memorandum, 680.

Hidetoshi Shimodaira and Masami Hasegawa. 2005. Assessing the Uncertainty in Phylogenetic Inference, pages 463–493. Springer New York, New York, NY.

Hidetoshi Shimodaira and Yoshikazu Terada. 2019. Selective inference for testing trees and edges in phylogenetics. Frontiers in ecology and evolution, 7:174.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-lm: Training multi-billion parameter language models using model parallelism. Preprint, arXiv:1909.08053.

Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, Philip Mansfield, Blaise Aguera y Arcas, Dale Webster, and 11 others. 2022. Large language models encode clinical knowledge. Preprint, arXiv:2212.13138.

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaeckermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, and 12 others. 2023. Towards expert-level medical question answering with large language models. Preprint, arXiv:2305.09617.

Shashank Sonkar, Naiming Liu, Debshila Basu Mallick, and Richard G. Baraniuk. 2023. Class: A design framework for building intelligent tutoring systems based on learning science principles. Preprint, arXiv:2305.13272.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, and 432 others. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Preprint, arXiv:2206.04615.

Stability AI Language Team. 2024. Stable lm 2 1.6b.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023a. Roformer: Enhanced transformer with rotary position embedding. Preprint, arXiv:2104.09864.

Yixuan Su, Tian Lan, and Deng Cai. 2023b. Openllapaca: A fully open-source instruction-following model based on openllama.

Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash Srivastava. 2024. Lab: Large-scale alignment for chatbots. Preprint, arXiv:2403.01081.

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2024. A simple and effective pruning approach for large language models. Preprint, arXiv:2306.11695.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. Preprint, arXiv:1811.00937.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model.

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaxiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023. U12: Unifying language learning paradigms. Preprint, arXiv:2205.05131.

Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q. Tran, David R. So, Siamak Shakeri, Xavier Garcia, Huaxiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc V. Le, and Mostafa Dehghani. 2022. Transcending scaling laws with 0.1% extra compute. Preprint, arXiv:2210.11399.

"Teknium", Charles Goddard, "interstellarninja", "theemozilla", "karan4d", and "huemin_art". 2024a. Hermes-2-theta-llama-3-8b.

"Teknium", "interstellarninja", "theemozilla", "karan4d", and "huemin_art". 2024b. Hermes-2-pro-llama-3-8b.

"Teknium", "theemozilla", "karan4d", and "huemin_art". 2024c. Nous hermes 2 mistral 7b dpo.

Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. 2025. Improving pretraining data using perplexity correlations. In Proceedings of the International Conference on Learning Representations (ICLR).

Migel Tissera. 2023. Synthia-7b-v1.3: Synthetic intelligent agent.

Together Computer. 2023. Redpajama-data: An open source recipe to reproduce llama training dataset.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenjin Fu, and 49 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.

Jonathan Tow. 2023. Stablelm alpha v2 models.

Jonathan Tow, Marco Bellagente, Dakota Mahan, and Carlos Riquelme. 2023. Stablelm 3b 4e1t.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sansevieri, Alexander M. Rush, and Thomas Wolf. 2023. Zephyr: Direct distillation of lm alignment. Preprint, arXiv:2310.16944.

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605.

Bram Vanroy. 2024. Geitje 7b ultra: A conversational model for dutch. Preprint, arXiv:2412.04092.

Gaël Varoquaux, Lars Buitinck, Gilles Louppe, Olivier Grisel, Fabian Pedregosa, and Andreas Mueller. 2015. Scikit-learn: Machine learning without learning the machinery. GetMobile Mob. Comput. Commun., 19(1):29–33.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, and 16 others. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272.

Leandro von Werra, Alex Havrilla, Max reciprocated, Jonathan Tow, Aman cat state, Duy V. Phung, Louis Castricato, Shahbuland Matiana, Alan, Ayush Thakur, Alexey Bukhtiyarov, aaronrmnm, Fabrizio Milo, Daniel, Daniel King, Dong Shin, Ethan Kim, Justin Wei, Manuel Romero, and 10 others. 2023. CarperAI/tlx: v0.6.0: LLaMa (Alpaca), Benchmark Util, T5 ILQL, Tests.

Ben Wang. 2021. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.
Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.

Guan Wang, Sijie Cheng, Qiyng Yu, and Changling Liu. 2023a. OpenChat: Advancing Open-source Language Models with Imperfect Data.

Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024a. Openchat: Advancing open-source language models with mixed-quality data. Preprint, arXiv:2309.11235.

Guan Wang, Bleys Goodson, Wing Lian, Eugene Pentland, Austin Cook, Chanvichet Vong, and “Teknium”. 2023b. Openorcaopenchatpreview2: Llama2-13b model instruct-tuned on filtered openorca1 gpt-4 dataset.

Shenzhi Wang, Yaowei Zheng, Guoyin Wang, Shiji Song, and Gao Huang. 2024b. Llama3-8b-chinese-chat (revision 6622a23).

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khaythi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023c. How far can camels go? exploring the state of instruction tuning on open resources. Preprint, arXiv:2306.04751.

Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y. Wu. 2024c. Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models. Preprint, arXiv:2407.01906.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R-Bowman. 2020. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392.

Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Preprint, arXiv:2203.05482.

Haoyuan Wu, Haisheng Zheng, Zhuolun He, and Bei Yu. 2024. Parameter-efficient sparsity crafting from dense to mixture-of-experts for instruction tuning on general tasks. Preprint, arXiv:2401.02731.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. Preprint, arXiv:2310.06694.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient streaming language models with attention sinks. Preprint, arXiv:2309.17453.

Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. 2023. Lm-cocktail: Resilient tuning of language models via model merging. Preprint, arXiv:2311.13534.

Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. 2024a. Deepseek-prover: Advancing theorem proving in ilms through large-scale synthetic data. Preprint, arXiv:2405.14333.

Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. 2024b. Deepseek-prover-v1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. Preprint, arXiv:2408.08152.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023a. Wizardlm: Empowering large language models to follow complex instructions. Preprint, arXiv:2304.12244.

Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023b. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. Preprint, arXiv:2304.01196.

Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. Preprint, arXiv:2309.11674.

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. Preprint, arXiv:2401.08417.

Xwin-LM Team. 2023. Xwin-lm.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023a. Ties-merging: Resolving interference when merging models. Preprint, arXiv:2306.01708.

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023b. Ties-merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, volume 36, pages 7093–7115. Curran Associates, Inc.

Hiroaki Yamagiwa, Yusuke Takase, and Hidetoshi Shimodaira. 2024. Axis tour: Word tour determines the order of axes in ica-transformed embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 477–506. Association for Computational Linguistics.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 43 others. 2024a. Qwen2 technical report. Preprint, arXiv:2407.10671.

Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024b. Mentallama: Interpretable mental health analysis on social media with large language models. Preprint, arXiv:2309.13567.

Ping Yang, Junjie Wang, Ruyi Gan, Xinyu Zhu, Lin Zhang, Ziwei Wu, Xinyu Gao, Jiaxing Zhang, and Tetsuya Sakai. 2022. Zero-shot learners for natural language understanding via a unified multiple choice perspective. Preprint, arXiv:2210.08590.

Nicolas Yax, Pierre-Yves Oudeyer, and Stefano Palminteri. 2025. PhyloLM: Inferring the phylogeny of large language models and predicting their performances in benchmarks. In The Thirteenth International Conference on Learning Representations.

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024a. Language models are super mario: Absorbing abilities from homologous models as a free lunch. Preprint, arXiv:2311.03099.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhengguo Li, Adrian Weller, and Weiyang Liu. 2024b. Metamath: Bootstrap your own mathematical questions for large language models. Preprint, arXiv:2309.12284.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqiu Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others. 2024a. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. Preprint, arXiv:2311.16502.

Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhui Chen. 2024b. Mammoth2: Scaling instructions from the web. Preprint, arXiv:2405.03548.

YuLan-Team. 2023. Yulan-chat: An open-source bilingual chatbot.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830.

Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. Preprint, arXiv:1910.07467.

Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2024a. Towards the law of capacity gap in distilling language models. Preprint, arXiv:2311.07052.

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, and 3 others. 2024b. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. Preprint, arXiv:2401.11944.

Jiaxing Zhang, Ruyi Gan, Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaojun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, and 6 others. 2022a. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. CoRR, abs/2209.02970.

Kaiyan Zhang, Ning Ding, Biqing Qi, Sihang Zeng, Haoxin Li, Xuekai Zhu, Zhang-Ren Chen, and Bowen Zhou. 2024c. Ultramedical: Building specialized generalists in biomedicine.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi-haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022b. Opt: Open pre-trained transformer language models. Preprint, arXiv:2205.01068.

Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Preprint, arXiv:2306.05179.

Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. Preprint, arXiv:1804.06876.

Tianyu Zhao, Akio Kaga, and Kei Sawada. 2023a. rinna/youri-7b.

Tianyu Zhao and Kei Sawada. 2023. rinna/japanese-gpt-neox-3.6b.

Tianyu Zhao, Toshiaki Wakatsuki, Akio Kaga, Koh Mitsuda, and Kei Sawada. 2023b. rinna/bilingual-gpt-neox-4b.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. Preprint, arXiv:2304.06364.
Xinyu Zhou, Delong Chen, Samuel Cahyawijaya, Xufeng Duan, and Zhenghua Cai. 2025. Linguistic minimal pairs elicit linguistic similarity in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6866–6888, Abu Dhabi, UAE. Association for Computational Linguistics.

Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. 2023a. Starling-7b: Improving llm helpfulness & harmlessness with raif.

Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, Michael I. Jordan, and Jiantao Jiao. 2023b. Fine-tuning language models with advantage-induced policy alignment. Preprint, arXiv:2306.02231.

Sally Zhu, Ahmed M Ahmed, Rohith Kuditipudi, and Percy Liang. 2025. Independence tests for language models.

Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. 2025. EmbedLLM: Learning compact representations of large language models. In The Thirteenth International Conference on Learning Representations.

Daniel M. Ziegler, Nisan Stienon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. Preprint, arXiv:1909.08593.

A Related Work

In recent years, research on comparing large language models (LLMs) has gained attention. This section provides an overview of existing studies from three perspectives: model parameters, activations\footnote{In general, “activations” refer to the intermediate outputs of Transformer models (e.g., the residual stream or neurons) (Bereska and Gavves, 2024).}, and model outputs.

Comparison of model parameters. One approach to comparing LLMs is to analyze their parameters. Zhu et al. (2025) proposed a statistical framework for evaluating parameter similarity between different models and introduced a method for determining whether these models were trained independently. Horwitz et al. (2025) compare weights to complement undocumented model relationships on Hugging Face. Additionally, Yadav et al. (2023b) focused on parameter changes due to task adaptation, specifically analyzing task vectors\footnote{The difference in parameters before and after fine-tuning is referred to as a task vector, and arithmetic operations on these vectors are effective (Ilharco et al., 2023).}. They proposed a method to mitigate interference when integrating task vectors from different models. Specifically, by reducing redundant numerical components and adjusting for conflicting signs, their approach enables effective model merging.

Comparison of activations. Comparisons of LLMs based on activations have also been studied. Zhou et al. (2025) quantified the similarity between LLMs by measuring the cosine similarity of activation differences for linguistic minimal pairs. In particular, they used datasets such as BLiMP (Warstadt et al., 2020) and showed that model similarity is significantly influenced by the pre-training dataset.

Comparison of model outputs. Several approaches compare LLMs based on their outputs. Lv et al. (2023) proposed a method for computing coefficients in parameter ensembling by providing the same input text to two models and comparing the softmax probability distributions at each token. Specifically, they used KL divergence and summed the results to derive appropriate coefficients. Furthermore, Yax et al. (2025) proposed a similarity metric based on the conditional probabilities of LLMs and introduced a method for calculating the phylogenetic distance between different models. Zhuang et al. (2025) propose a framework for vector representations of language models based on their performance on downstream tasks. Additionally, a method has been proposed for measuring differences in conditional probabilities based on prompts using KL divergence (Melamed et al., 2024).

B Double Centering

We confirm the notation and computational operations. The matrices \( L \), \( \Xi \), and \( Q \) are all of size \( K \times N \), and their elements are denoted by \( \ell_{is}, \xi_{is}, q_{is} \), respectively. In particular, we have \( \ell_{is} = \ell_i(x_s) \). First, row-wise centering of \( L \) is performed by subtracting the mean log-likelihood \( \bar{\ell}_i \) of each model from each row (\( \ell_{i1}, \ldots, \ell_{iN} \)), resulting in \( \Xi = (\xi_1, \ldots, \xi_K)^T \). Next, column-wise centering of \( \Xi \) is performed by subtracting the coordinate component \( \xi_s \) of the mean vector \( \xi \) from each column (\( \xi_{1s}, \ldots, \xi_{Ks} \)^T), yielding \( Q = (q_1, \ldots, q_K)^T \). Thus, this process involves double centering, where column-wise centering follows row-wise centering. Notably, even after column-wise centering, the row-wise mean of \( Q \)
remains zero:

\[
\frac{1}{N} \sum_{s=1}^N q_{is} = \frac{1}{N} \sum_{s=1}^N (\xi_{is} - \bar{\xi}_s)
\]
\[
= \frac{1}{N} \sum_{s=1}^N \xi_{is} - \frac{1}{NK} \sum_{i=1}^K \sum_{s=1}^N \xi_{is}
\]
\[
= 0 - 0 = 0.
\] (6)

The column-wise centering can be interpreted as follows. In the \( \xi \)-coordinate system, the mean vector \( \bar{\xi} \) of the \( K \) model coordinates \( \xi_1, \ldots, \xi_K \) can be regarded as representing an “average model.” By redefining this average model as the new origin, we obtain the \( q \)-coordinate system. From the definition \( q_i = \xi_i - \bar{\xi} \), its mean satisfies

\[
\sum_{i=1}^K q_i / K = 0.
\]

The row-wise centering can be interpreted as follows. Let \( \mathbf{1}_N = (1, \ldots, 1)^T \in \mathbb{R}^N \). Since \( \bar{\ell}_i = \mathbf{1}_N^\top \ell_i / N \), we have \( \xi_i = \ell_i - \bar{\ell}_i \mathbf{1}_N \). Thus,

\[
\mathbf{1}_N^\top \xi_i = \mathbf{1}_N^\top \ell_i - \bar{\ell}_i N = 0,
\]

and furthermore, \( \mathbf{1}_N^\top \bar{\xi} = 0 \). From this, equation (6) is actually trivial, as

\[
\mathbf{1}_N^\top q_i = \mathbf{1}^\top (\xi_i - \bar{\xi}) = 0 - 0 = 0.
\]

The row-wise centering implies that \( \xi_1, \ldots, \xi_K \) and \( q_1, \ldots, q_K \) lie in the subspace orthogonal to \( \mathbf{1}_N \).

C Effect of Errors in Model Coordinates

We analyze the impact of additive errors \( \epsilon_{is} \) in the log-likelihood vector components \( \ell_{is} \) for \( i = 1, \ldots, K \) and \( s = 1, \ldots, N \). Denoting the true values with an asterisk as \( \ell_{is}^* \), the observed values can be expressed as

\[
\ell_{is} = \ell_{is}^* + \epsilon_{is}.
\]

We decompose the error as follows:

\[
\epsilon_{is} = a + b_i + c_s + d_{is},
\]

where we assume, without loss of generality, the constraints

\[
\sum_{i=1}^K b_i = \sum_{s=1}^N c_s = \sum_{i=1}^K d_{is} = \sum_{s=1}^N d_{is} = 0.
\]

Here, \( a, b_i, \) and \( c_s \) are bias terms, while \( d_{is} \) represents interaction terms. Using simple calculations, we obtain:

\[
\bar{\ell}_i = \frac{1}{N} \sum_{s=1}^N \ell_{is} = \ell_{i}^* + a + b_i,
\]
\[
\xi_{is} = \ell_{is} - \bar{\ell}_i = \xi_{is}^* + c_s + d_{is},
\]
\[
\bar{\xi}_s = \frac{1}{K} \sum_{i=1}^K \xi_{is} = \xi_{s}^* + c_s,
\]
\[
q_{is} = \xi_{is} - \bar{\xi}_s = q_{is}^* + d_{is}.
\]

Additionally, for the differences between two models, which are crucial for the model map, we obtain:

\[
\ell_{is} - \ell_{js} = \ell_{is}^* - \ell_{js}^* + b_i - b_j + d_{is} - d_{js},
\]
\[
\xi_{is} - \xi_{js} = \xi_{is}^* - \xi_{js}^* + d_{is} - d_{js},
\]
\[
q_{is} - q_{js} = q_{is}^* - q_{js}^* + d_{is} - d_{js}.
\]

Thus, the terms affected by the error are \( \bar{\ell}_i \), which is influenced by \( a + b_i \), and \( \ell_{is} - \ell_{js} \), which is affected by \( b_i + d_{is} \), meaning it is influenced by the bias terms. However, in the centered values \( \xi_{is} - \xi_{js} \) and \( q_{is} - q_{js} \), only the interaction term \( d_{is} \) contributes to the error.

D Theory of Model Coordinates for Text Probability Distributions

In this section, we prove the main results of Section 2, namely (2) and (3). Our discussion applies not only to text probability distributions but also more generally to any setting where i.i.d. observations \( x_1, \ldots, x_N \sim p_i(x) \) are available. Compared to the previous study that proposed model maps (Shimodaira, 1993; Shimodaira and Cao, 1998), we conduct a more precise analysis in this paper. Specifically, while the previous study provides only a brief evaluation of the approximation, we present a more transparent discussion based on the properties of the exponential family of distributions. In the next section, as the starting point of our discussion, we construct a super model that includes the \( K \) models \( p_i, i = 1, \ldots, K \), as sub-models. This model is introduced as a mathematical tool to rigorously prove the main theorem of this paper, and we do not compute it numerically in practice.
D.1 Exponential family of distributions
We first consider a model in the exponential family of distributions parameterized by a \( K \)-dimensional parameter \( \theta \in \mathbb{R}^K \):

\[
p(x; \theta) = p_0(x) \exp(\theta^\top b(x) - \psi(\theta)). \tag{7}
\]

Here, the function \( b(x) = (b_1(x), \ldots, b_K(x))^\top \) will be defined later using the \( K \) models. The normalization constant is given by

\[
Z(\theta) = \sum_{x \in \mathcal{X}} p_0(x) \exp(\theta^\top b(x)),
\]
\[
\psi(\theta) = \log Z(\theta),
\]

which ensures that \( \sum_{x \in \mathcal{X}} p(x; \theta) = 1 \). For \( \theta = 0 \), where \( 0 = (0, \ldots, 0)^\top \), we obtain

\[
p(x; 0) = p_0(x).
\]

To associate the \( K \) models with (7), we define \( b(x) \). For a constant \( \lambda > 0 \), we set

\[
\lambda b_i(x) := \ell_i(x) - \ell_0(x). \tag{8}
\]

The constant \( \lambda \) is an order parameter introduced for theoretical convenience, and in our theoretical framework, we assume that \( b_i(x) \) is of constant order and that \( \lambda \) is sufficiently small\footnote{If numerical computation were to be performed, \( \lambda \) could be set to any arbitrary value (e.g., \( \lambda = 1 \)).}. Thus, we essentially assume that \( |\ell_i(x) - \ell_0(x)| = O_p(\lambda) \) is sufficiently small, implying that each model \( p_i \) provides a good approximation of the true generative model \( p_0 \). In the proof of the main theorem, we consider the asymptotic theory as \( \lambda \to 0 \), retaining terms up to \( O(\lambda^2) \) while ignoring those of \( O(\lambda^3) \).

A one-hot vector is defined as \( e_i = (0, \ldots, 0, 1, 0, \ldots, 0)^\top \in \mathbb{R}^K \) for \( i = 1, \ldots, K \), where only the \( i \)-th element is 1. Then, setting \( \theta = \lambda e_i \) gives

\[
p(x; \lambda e_i) = p_i(x). \tag{9}
\]

Indeed, substituting (8) into (7) yields

\[
\begin{align*}
p(x; \lambda e_i) \\
= & p_0(x) \exp(\lambda e_i^\top b(x) - \psi(\lambda e_i)) \\
= & p_0(x) \exp(\ell_i(x) - \ell_0(x) - \psi(\lambda e_i)) \\
= & p_0(x) (p_i(x)/p_0(x)) \exp(-\psi(\lambda e_i)) \\
= & p_i(x) \exp(-\psi(\lambda e_i)) \\
= & p_i(x),
\end{align*}
\]

where \( \psi(\lambda e_i) = 0 \).

D.2 Properties of the exponential family of distributions

This section outlines some well-known basic properties of the exponential family of distributions, which have been established in the literature (Barndorff-Nielsen, 2014; Efron, 1978, 2022; Amari, 1982). We define the expectation and covariance matrix of \( b(x) \) as follows:

\[
\eta(\theta) := \mathbb{E}_{x \sim p(\theta)} (b(x)) = \sum_{x \in \mathcal{X}} b(x)p(x; \theta),
\]
\[
G(\theta) := \mathbb{E}_{x \sim p(\theta)} \left\{ (b(x) - \eta(\theta))(b(x) - \eta(\theta))^\top \right\}
= \operatorname{Var}_{x \sim p(\theta)} (b(x)).
\]

Here, the elements of \( \eta(\theta) \) are expectations given by \( \mathbb{E}_{x \sim p(\theta)}(b_i(x)) \), and the elements of \( G(\theta) \) are covariances given by \( G_{ij}(\theta) = \operatorname{Cov}_{x \sim p(\theta)}(b_i(x), b_j(x)) \). These quantities can be expressed in terms of \( \psi(\theta) \) as follows:

\[
\eta(\theta) = \frac{\partial \psi(\theta)}{\partial \theta}, \tag{10}
\]
\[
G(\theta) = \frac{\partial^2 \psi(\theta)}{\partial \theta \partial \theta^\top}. \tag{11}
\]

We now derive these two equations. First, from

\[
\frac{\partial Z(\theta)}{\partial \theta} = \sum_{x \in \mathcal{X}} b(x)p_0(x)e^{\theta^\top b(x)},
\]

we obtain

\[
\begin{align*}
\frac{\partial \psi(\theta)}{\partial \theta} &= \frac{\partial \log Z}{\partial \theta} = \frac{1}{Z(\theta)} \frac{\partial Z}{\partial \theta} \\
&= \frac{1}{Z(\theta)} \sum_{x \in \mathcal{X}} b(x)p_0(x)e^{\theta^\top b(x)} \\
&= \sum_{x \in \mathcal{X}} b(x)p(x; \theta) - \eta(\theta).
\end{align*}
\]

Thus, we have established (10). Next, using

\[
\frac{\partial p(x; \theta)}{\partial \theta} = \left( b(x) - \frac{\partial \psi}{\partial \theta} \right) p(x; \theta)
= (b(x) - \eta(\theta))p(x; \theta),
\]
we obtain
\[
\frac{\partial^2 \psi(\theta)}{\partial \theta \partial \theta^\top} = \frac{\partial \eta(\theta)^\top}{\partial \theta}
= \frac{\partial}{\partial \theta} \sum_{x \in \mathcal{X}} b(x)^\top p(x; \theta)
= \sum_{x \in \mathcal{X}} (b(x) - \eta(\theta)) b(x)^\top p(x; \theta)
= \sum_{x \in \mathcal{X}} (b(x) - \eta(\theta))(b(x) - \eta(\theta))^\top p(x; \theta)
=G(\theta).
\]
Thus, we have established (11).

D.3 Approximation of the KL divergence
The Kullback-Leibler (KL) divergence between the models \( p(\theta) \) and \( p(\theta') \) at parameter values \( \theta, \theta' \in \mathbb{R}^K \) is given by
\[
\text{KL}(p(\theta), p(\theta')) \\
= \sum_{x \in \mathcal{X}} p(x; \theta) \log \frac{p(x; \theta)}{p(x; \theta')}
= \sum_{x \in \mathcal{X}} p(x; \theta) \{ (\theta - \theta')^\top b(x) - \psi(\theta) + \psi(\theta') \}
= (\theta - \theta')^\top \eta(\theta) - \psi(\theta) + \psi(\theta').
\] (12)

Here, we assume that the parameter values \( \theta \) and \( \theta' \) are sufficiently close to 0. In particular, we assume \( \| \theta \| = O(\lambda) \) and \( \| \theta' \| = O(\lambda) \). Substituting (10) and (11) into the Taylor expansion of \( \psi(\theta) \) gives
\[
\psi(\theta')
=\psi(\theta) + \frac{\partial \psi}{\partial \theta}(\theta' - \theta)
+ \frac{1}{2} (\theta' - \theta)^\top \frac{\partial^2 \psi(\theta)}{\partial \theta \partial \theta^\top} (\theta' - \theta)
+ O(\| \theta' - \theta \|^3)
=\psi(\theta) + \eta(\theta)^\top (\theta' - \theta)
+ \frac{1}{2} (\theta' - \theta)^\top G(\theta) (\theta' - \theta) + O(\lambda^3). \tag{13}
\]
Substituting (13) into (12) gives
\[
\text{KL}(p(\theta), p(\theta'))
= \frac{1}{2} (\theta' - \theta)^\top G(\theta) (\theta' - \theta) + O(\lambda^3). \tag{14}
\]
This corresponds to eq. (9) of Oyama et al. (2023). Here, the equation holds approximately by ignoring higher-order terms of \( O(\lambda^3) \). For more details, refer to Amari (1982, p. 369) and Efron (2022, p. 35). More generally, \( G(\theta) \) represents the Fisher information metric, and (14) holds for a wide class of probability models (Amari, 1998) with \( \| \theta' - \theta \| = O(\lambda) \). Furthermore, since \( G(\theta) = G(0) + O(\| \theta \|) = G(0) + O(\lambda) \), we obtain
\[
\begin{align*}
\text{KL}(p(\theta), p(\theta')) \\
&= \frac{1}{2} (\theta' - \theta)^\top G(0) (\theta' - \theta) + O(\lambda^3).
\end{align*}
\] (15)

D.4 The variance representation of the KL divergence
Substituting \( p_i = p(\lambda e_i) \) into (15) gives
\[
\begin{align*}
2\text{KL}(p_i, p_j) &= 2\text{KL}(p(\lambda e_i), p(\lambda e_j)) \\
&= \lambda^2 (e_i - e_j)^\top G(0) (e_i - e_j) + O(\lambda^3).
\end{align*}
\] (16)

Here, we have
\[
\begin{align*}
e_i^\top G(0) e_j &= G_{ij}(0) \\
&= \mathbb{E}_{x \sim p_0} \left\{ (b_i(x) - \eta_i(0))(b_j(x) - \eta_j(0)) \right\}.
\end{align*}
\] (17)

Next, we substitute (17) and (8) into the right-hand side of (16) to derive an alternative expression for the KL divergence:
\[
\begin{align*}
&\lambda^2 (e_i - e_j)^\top G(0) (e_i - e_j) \\
=& \lambda^2 \mathbb{E}_{x \sim p_0} \Big[ \Big\{ (b_i(x) - \eta_i(0)) \\
&\qquad - (b_j(x) - \eta_j(0)) \Big\}^2 \Big] \\
=& \mathbb{E}_{x \sim p_0} \Big[ \Big\{ (\ell_i(x) - \ell_0(x)) - (\ell_j(x) - \ell_0(x)) \\
&\qquad - \mathbb{E}_{x' \sim p_0} \Big( (\ell_i(x') - \ell_0(x')) \\
&\qquad - (\ell_j(x') - \ell_0(x')) \Big) \Big\}^2 \Big] \\
=& \mathbb{E}_{x \sim p_0} \Big[ \Big\{ \ell_i(x) - \ell_j(x) \\
&\qquad - \mathbb{E}_{x' \sim p_0} \Big( \ell_i(x') - \ell_j(x') \Big) \Big\}^2 \Big] \\
=& \operatorname{Var}_{x \sim p_0} \Big( \ell_i(x) - \ell_j(x) \Big).
\end{align*}
\]
Finally, substituting this result into (16) yields
\[
2\text{KL}(p_i, p_j) = \operatorname{Var}_{x \sim p_0} \Big( \ell_i(x) - \ell_j(x) \Big) + O(\lambda^3).
\] (18)

This establishes (2). Furthermore, since \( |\ell_i(x) - \ell_j(x)| = O_p(\lambda) \), the magnitude of (18) is \( O(\lambda^2) \).
D.5 Estimation of the KL divergence
If the expected value \( \mathbb{E}_{x \sim p_0}(f(x)) \) of a function \( f(x) \) exists and is bounded, then by the law of large numbers, the sample mean\footnote{\( \mathbb{E}_{x \sim D}(f(x)) = \frac{1}{N} \sum_{x \in D} f(x) = \frac{1}{N} \sum_{s=1}^N f(x_s) \) represents the sample mean, and \( \mathrm{Var}_{x \sim D}(\cdot) \) represents the sample variance.} converges to the expected value as \( N \to \infty \), and we have
\[
\mathbb{E}_{x \sim D}(f(x)) = \mathbb{E}_{x \sim p_0}(f(x)) + O_p(N^{-1/2}).
\]
Applying this to (18) and ignoring the terms of order \( O_p(\lambda^3 + \lambda^2 N^{-1/2}) \), we obtain the following approximation:
\[
2\mathrm{KL}(p_i, p_j) \approx \mathrm{Var}_{x \sim D}\left( \ell_i(x) - \ell_j(x) \right). \tag{19}
\]
We define the coordinates \( \xi_i \in \mathbb{R}^N \) of model \( p_i \) as
\[
\xi_i = (\xi_{i1}, \ldots, \xi_{iN})^\top
\]
with
\[
\xi_{is} := \ell_i(x_s) - \mathbb{E}_{x \sim D}(\ell_i(x))
\]
for \( s = 1, \ldots, N \). From (19), we obtain
\[
2\mathrm{KL}(p_i, p_j) \approx \frac{1}{N} \sum_{s=1}^N (\xi_{is} - \xi_{js})^2 \\
= \frac{1}{N} \| \xi_i - \xi_j \|^2. \tag{20}
\]
Since
\[
\| \xi_i - \xi_j \|^2 = \| (q_i + \bar{\xi}) - (q_j + \bar{\xi}) \|^2 = \| q_i - q_j \|^2,
\]
this establishes (3).

D.6 Relationships among the three types of model coordinates
Let \( 1_N = (1, \ldots, 1)^\top \in \mathbb{R}^N \). From the definitions of the \( \xi \)-coordinate system and the \( q \)-coordinate system, we have
\[
\xi_i = q_i + \bar{\xi}, \\
\ell_i = \xi_i + \ell_i 1_N \\
= q_i + \bar{\ell}_i 1_N + \bar{\xi}. \tag{21}
\]
Additionally, equation (6) in Appendix B can be rewritten as
\[
1_N^\top q_i = 0. \tag{22}
\]
Thus, we obtain
\[
\begin{align*}
\|\ell_i - \ell_j\|^2 &= \| (q_i - q_j) + (\bar{\ell}_i - \bar{\ell}_j) 1_N \|^2 \\
&= \| q_i - q_j \|^2 + N (\bar{\ell}_i - \bar{\ell}_j)^2 \\
&\quad + 2 (\bar{\ell}_i - \bar{\ell}_j) 1_N^\top (q_i - q_j) \\
&= \| q_i - q_j \|^2 + N (\bar{\ell}_i - \bar{\ell}_j)^2,
\end{align*}
\]
where (21) and (22) are used in the first and last equations, respectively. This establishes (4).

Moreover, since \( \bar{\ell}_i = 1_N^\top \ell_i / N \), it is straightforward that the component of the \( \ell \)-coordinate system in the \( 1_N \) direction is given by
\[
(1_N / \sqrt{N})^\top \ell_i = \sqrt{N} \bar{\ell}_i.
\]

D.7 Additional notes on modifying the underlying generative model
We examine the effect of changing the true generative model \( p_0 = p(0) \) that produces the data \( D \). For clarity, we continue to use the same \( p_0 \) as before in constructing \( p(x; \theta) \) as described in Section D.1. We then introduce a new parameter value \( \theta^* \). We assume that each element \( x_s \) of the dataset \( D \) is generated independently from
\[
x_1, \ldots, x_N \sim p(x; \theta^*), \tag{23}
\]
and that \( \| \theta^* \| = O(\lambda) \). In other words, the true generative model remains sufficiently close to the \( K \) models, satisfying \( \| \theta_i - \theta^* \| = O(\lambda) \) for \( i = 1, \ldots, K \).

First, consider replacing \( G(\theta) \) in (14) of Section D.3 with \( G(\theta^*) \). Because \( G(\theta^*) = G(\theta) + O(\lambda) \), we obtain
\[
\begin{align*}
\mathrm{KL}(p(\theta), p(\theta')) \\
= \frac{1}{2} (\theta' - \theta)^\top G(\theta^*)(\theta' - \theta) + O(\lambda^3). \tag{24}
\end{align*}
\]
We are generalizing the discussion in the previous sections, and indeed, if we set \( \theta^* = 0 \) in (24), then (15) is recovered.

Now substitute \( p_i = p(\lambda e_i) \) into (24), yielding
\[
\begin{align*}
2\mathrm{KL}(p_i, p_j) &= 2\mathrm{KL}(p(\lambda e_i), p(\lambda e_j)) \\
&= \lambda^2 (e_i - e_j)^\top G(\theta^*)(e_i - e_j) + O(\lambda^3), \tag{25}
\end{align*}
\]
which is a generalization of (16) in Section D.4. Using the definition of \( G(\theta) \), we have
\[
e_i^\top G(\theta^*) e_j = G_{ij}(\theta^*) = \underset{x \sim p(\theta^*)}{\mathrm{Cov}}(b_i(x), b_j(x)). \tag{26}
\]
Substituting (26) and (8) into the right-hand side of (25) yields

\[
\begin{align*}
&\lambda^2 (e_i - e_j)^T G(\theta^*) (e_i - e_j) \\
&= \lambda^2 \Big\{ \underset{x \sim p(\theta^*)}{\mathrm{Var}} (b_i(x)) + \underset{x \sim p(\theta^*)}{\mathrm{Var}} (b_j(x)) \\
&\qquad - 2 \underset{x \sim p(\theta^*)}{\mathrm{Cov}} (b_i(x), b_j(x)) \Big\} \\
&= \lambda^2 \underset{x \sim p(\theta^*)}{\mathrm{Var}} (b_i(x) - b_j(x)) \\
&= \underset{x \sim p(\theta^*)}{\mathrm{Var}} (\ell_i(x) - \ell_j(x)).
\end{align*}
\]

Finally, substituting this back into (25) gives

\[
2 \mathrm{KL}(p_i, p_j) = \underset{x \sim p(\theta^*)}{\mathrm{Var}} (\ell_i(x) - \ell_j(x)) + O(\lambda^3),
\]

which is a generalization of (2).

Hence, the estimators for KL divergence in Section D.5, specifically (19) and (20), and also (3) in Section 2, remain valid even when the texts are generated by (23). Since (27) holds for any \( \theta^* \) with \( \| \theta^* \| = O(\lambda) \), this result also applies when the true data-generating model is \( p(\theta^*) = p_0 \), or, for instance, one of the models \( p(\theta^*) = p_i \), or a mixture of the \( K \) models, \( \theta^* = \sum_{i=1}^K \alpha_i \lambda e_i \) with \( \alpha_i = O(1) \). Therefore, this method is robust to contamination in the dataset \( D \) (e.g., when the text corpus used for pre-training a model is included in \( D \)), as the estimation of KL divergence via the squared Euclidean distance in the \( \xi \)-coordinate system or the \( q \)-coordinate system remains relatively unaffected.

E  Mapping Language Models into the Space of Token Probability Distributions

In Section 2, we discussed model maps based on the probability distributions \( p_i(x) \) of texts generated by language models. This approach requires computing probabilities for a large number of texts in the dataset \( D = (x_1, \ldots, x_N) \), leading to high computational costs. To mitigate this issue, we focus on the fact that a text \( x = (y_1, \ldots, y_n) \) is a sequence of tokens. Instead of using text probabilities, we discuss model maps based on the conditional probability distributions of token generation, \( p_i(y_t|y^{t-1}) \). In this approach, model coordinates are computed using only a single text \( x \). A limitation of this approach is that it can only be used for comparing models that share the same tokenizer.

Furthermore, the current estimation method ignores the variance of the expected log-likelihood ratio of conditional probabilities, resulting in a rough approximation. Thus, the estimated values should be regarded only as reference values rather than precise measurements.

E.1 Model coordinates

For a text \( x = (y_1, \ldots, y_n) \), the coordinates of model \( p_i \)

\[
\zeta_i = (\zeta_{i1}, \ldots, \zeta_{in})^\top \in \mathbb{R}^n
\]

are defined as

\[
\zeta_{it} := \log p_i(y_t|y^{t-1}) - \ell_i(x)/n
\]

for \( t = 1, \ldots, n \). This is centered for each \( i \) and for each text, satisfying \( \sum_{t=1}^n \zeta_{it} = 0 \).

E.2 Kullback-Leibler divergence

The KL divergence for next-token generation in language models, where \( y_t \sim p_i(y_t|y^{t-1}) \), is given by

\[
\mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) =
\sum_{y_t \in \mathcal{V}} p_i(y_t|y^{t-1}) \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})}.
\]

We apply the results for text probability distributions from Section 2 and Appendix D to the conditional probability distributions of token generation. The equation corresponding to (2) is

\[
2 \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) \approx \underset{y_t \sim p_0(y_t|y^{t-1})}{\mathrm{Var}} \left\{ \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})} \right\}.
\]

The squared Euclidean distance in the \( \zeta \)-coordinate system provides an estimate of the sum of (31) over all tokens in the text \( x \):

\[
\begin{align*}
&\| \zeta_i - \zeta_j \|^2 \\
\approx & 2 \sum_{t=1}^n \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) \\
\approx & 2 \mathrm{KL}(p_i, p_j).
\end{align*}
\]

The proof is provided in Appendix F. To justify the estimation in (32), we assume the following:

\[
\underset{y_t \sim p_0(y_t|y^{t-1})}{\mathbb{E}} \left\{ \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})} \right\}
\]
takes a constant value independent of \( t \). In reality, this assumption is not entirely correct, and the degree of variation affects the accuracy of the approximation in (32)\footnote{Since this assumption does not affect (3), there is no concern regarding the use of model maps based on text probabilities.}. On the other hand, the approximation in (33) holds more generally and is demonstrated in Appendix F.5.

F Theory of Model Coordinates for Token Probability Distributions

In this section, we provide a more detailed explanation of the content discussed in Appendix E. We extend the discussion of text probability distributions in Appendix D to the case of conditional probability distributions for token generation.

F.1 Exponential family of distributions

We apply the same setting as for \( p_i(x) \) in Section D to the conditional probability distributions of tokens:

\[
y_t \sim p_i(y_t|y^{t-1}), \quad t = 1, \ldots, n.
\]

The exponential family of distributions incorporating \( K \) models, corresponding to (7), is given here as

\[
p(y_t|y^{t-1}; \theta) := p_0(y_t|y^{t-1}) \\
\exp(\theta^\top b(y_t|y^{t-1}) - \psi(\theta|y^{t-1})). \tag{35}
\]

The setting (8), which associates the \( K \) models with (35), is given here as

\[
\lambda b_i(y_t|y^{t-1}) := \\
\log p_i(y_t|y^{t-1}) - \log p_0(y_t|y^{t-1}). \tag{36}
\]

Thus, we have

\[
p_i(y_t|y^{t-1}) = p(y_t|y^{t-1}; \lambda e_i)
\]

for \( i = 1, \ldots, K \).

F.2 The variance representation of the KL divergence

The KL divergence is given by (30). Applying the result for the model \( p(x; \theta) \) in (18) to the token-level conditional distribution model \( p(y_t|y^{t-1}; \theta) \), we obtain

\[
2\mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) =
\frac{\mathrm{Var}}{y_t \sim p_0(y_t|y^{t-1})}
\left\{ \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})} \right\} + O(\lambda^3). \tag{37}
\]

F.3 Two additional assumptions

To estimate the KL divergence from a single text \( x \), two additional assumptions are required, as described below. Such assumptions were not necessary when estimating the KL divergence from the dataset \( D \) in Appendix D. In reality, these two assumptions are not strictly satisfied, and the discrepancy between these assumptions and reality affects the accuracy of the KL divergence approximation.

Assumption 1: We assume that the probability distribution of \( y_t \) depends only on the past \( k \) tokens, denoted as \( y_{t-k} = (y_{t-k}, y_{t-k+1}, \ldots, y_{t-1}) \). That is,

\[
p_i(y_t|y^{t-1}) = p_i(y_t|y_{t-k}^{t-1}),
\]

which allows us to regard \( y_{t-k}^{t} \) as the state of a Markov chain. More generally, we use the notation \( y^k \) to represent a state variable. We consider a function \( f \) of the state variable \( y^k \). Furthermore, we assume that this Markov chain is positive Harris recurrent, has a stationary distribution \( \pi \), and that \( f \) is absolutely integrable, i.e.,

\[
\mathbb{E}_{y^k \sim \pi}(|f(y^k)|) < \infty.
\]

Then, by the strong law of large numbers for Markov chains (Meyn and Tweedie, 2009, Theorem 17.0.1 (i)), in the limit \( n \to \infty \),

\[
\frac{1}{n} \sum_{t=1}^n f(y_{t-k}^t) \to \mathbb{E}_{y^k \sim \pi}(f(y^k)) \quad \text{a.s.}
\]

For simplicity in notation and discussion, we assume that \( y_{-k+1}, \ldots, y_0 \) are appropriately defined. Since the Markov chain converges to \( \pi \), we also have

\[
\frac{1}{n} \sum_{t=1}^n f(y_{t-k}^t) \to \frac{1}{n} \sum_{t=1}^n \mathbb{E}_{y^k \sim p_0}(f(y_{t-k}^t)) \tag{38}
\]

almost surely as \( n \to \infty \).

Assumption 2:

\[
\mathbb{E}_{y_t \sim p_0(y_t|y^{t-1})} \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})} = c \tag{39}
\]

for some \( c \in \mathbb{R} \) that can depend on the indices \( i \) and \( j \) but not on \( t \). In other words, (39) takes a constant value independent of \( t \).
F.4 Estimation of the KL divergence

Define
\[
h(y^t) := \log \frac{p_i(y_t|y^{t-1})}{p_j(y_t|y^{t-1})}.
\]
From Assumption 1, \( h(y^t) \) can be written in the form \( h(y^t) = f_1(y_{t-k}^t) \) for some \( f_1 \), so applying (38), for sufficiently large \( n \), we obtain
\[
\frac{1}{n} \sum_{t=1}^n h(y^t) \approx \frac{1}{n} \sum_{t=1}^n \mathbb{E}_{y^{t-1}\sim p_0} (h(y^t)).
\]
Applying (39) to the right-hand side gives
\[
\frac{1}{n} \sum_{t=1}^n h(y^t) \approx c.
\]
Next, since \( (h(y^t) - c)^2 \) can be written in the form of \( f_2(y_{t-k}^t) \) for some function \( f_2 \), applying (38) again yields
\[
\frac{1}{n} \sum_{t=1}^n (h(y^t) - c)^2 \\
\approx \frac{1}{n} \sum_{t=1}^n \mathbb{E}_{y^{t-1}\sim p_0} \left\{ \mathrm{Var}_{y_t\sim p_0(y_t|y^{t-1})}(h(y^t)) \right\} \\
\approx \frac{1}{n} \sum_{t=1}^n \mathrm{Var}_{y_t\sim p_0(y_t|y^{t-1})}(h(y^t)).
\]
In the final equation, we applied (38) using the fact that \( \mathrm{Var}_{y_t\sim p_0(y_t|y^{t-1})}(h(y^t)) = f_3(y_{t-k}^t) \) for some \( f_3 \). Using (37), we obtain
\[
\sum_{t=1}^n (h(y^t) - c)^2 \approx \\
2 \sum_{t=1}^n \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})). \tag{40}
\]
Meanwhile, the components of the model coordinate \( \zeta_i \) are given by
\[
\zeta_{it} = \log p_i(y_t|y^{t-1}) - c_i
\]
where
\[
c_i = \frac{1}{n} \sum_{t=1}^n \log p_i(y_t|y^{t-1}).
\]
Since
\[
\zeta_{it} - \zeta_{jt} = h(y^t) - (c_i - c_j)
\]
with \( c_i - c_j \approx c \), equation (40) can be rewritten as
\[
\|\zeta_i - \zeta_j\|^2 \approx \\
2 \sum_{t=1}^n \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})). \tag{41}
\]
Thus, (32) is established.

F.5 Connecting the KL divergence of token and text probability distributions

Here, we fix the sequence length of the text \( x = (y_1, \ldots, y_n) \) as \( n \), i.e., we set \( \mathcal{X} = \mathcal{V}^n \). For notational simplicity, we define
\[
g_i(y^t) = \log p_i(y_t|y^{t-1}).
\]
Noting that
\[
p_i(x) = \prod_{t=1}^n p_i(y_t|y^{t-1}) = \prod_{t=1}^n e^{g_i(y^t)},
\]
we obtain
\[
\begin{align*}
\mathrm{KL}(p_i, p_j) &= \sum_{x \in \mathcal{X}} \prod_{t'=1}^n e^{g_i(y^{t'})} \sum_{t=1}^n (g_i(y^t) - g_j(y^t)) \\
&= \sum_{t=1}^n \sum_{y^{t'} \in \mathcal{V}^{t'-1}} \prod_{t'=1}^t e^{g_i(y^{t'})} (g_i(y^t) - g_j(y^t)) \\
&= \sum_{t=1}^n \sum_{y^{t'-1} \in \mathcal{V}^{t'-1}} \prod_{t'=1}^{t-1} e^{g_i(y^{t'})} \\
&\quad \sum_{y_t \in \mathcal{V}} e^{g_i(y^t)} (g_i(y^t) - g_j(y^t)) \\
&= \sum_{t=1}^n \sum_{y^{t'-1} \in \mathcal{V}^{t'-1}} p_i(y^{t'-1}) \\
&\quad \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) \\
&= \sum_{t=1}^n \mathbb{E}_{y^{t'-1}\sim p_i} \left\{ \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) \right\} \\
&= \mathbb{E}_{x\sim p_i} \left\{ \sum_{t=1}^n \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})) \right\}.
\end{align*}
\]
Thus, for sufficiently large \( n \), by the strong law of large numbers for Markov chains, we obtain
\[
\mathrm{KL}(p_i, p_j) \approx \sum_{t=1}^n \mathrm{KL}(p_i(y_t|y^{t-1}), p_j(y_t|y^{t-1})). \tag{41}
\]
This corresponds to (33). Assumption 1 from Appendix F.3 is used in (41), but Assumption 2 is not needed in the discussion of this subsection.

G Details of Experiments

G.1 Information obtained via the Hugging Face Hub API

We used the Hugging Face Hub API to retrieve information about each language model’s tags, the
<table>
  <tr>
    <th>Model type</th>
    <th>Models</th>
  </tr>
  <tr><td>llama-1</td><td>69</td></tr>
  <tr><td>llama-2</td><td>223</td></tr>
  <tr><td>llama-3</td><td>62</td></tr>
  <tr><td>llama</td><td>217</td></tr>
  <tr><td>mistral</td><td>232</td></tr>
  <tr><td>gpt_neox</td><td>54</td></tr>
  <tr><td>deepseek</td><td>26</td></tr>
  <tr><td>gptj</td><td>19</td></tr>
  <tr><td>gemma</td><td>18</td></tr>
  <tr><td>opt</td><td>15</td></tr>
  <tr><td>bloom</td><td>12</td></tr>
  <tr><td>falcon</td><td>11</td></tr>
  <tr><td>qwen2</td><td>10</td></tr>
  <tr><td>mixtral</td><td>9</td></tr>
  <tr><td>mpt</td><td>6</td></tr>
  <tr><td>stablelm</td><td>6</td></tr>
  <tr><td>gpt_neo</td><td>3</td></tr>
  <tr><td>phi</td><td>3</td></tr>
  <tr><td>gpt_bigcode</td><td>3</td></tr>
  <tr><td>phi3</td><td>3</td></tr>
  <tr><td>xglm</td><td>3</td></tr>
  <tr><td>rwkv</td><td>3</td></tr>
  <tr><td>starcoder2</td><td>2</td></tr>
  <tr><td>olmo</td><td>2</td></tr>
  <tr><td>camelidae</td><td>2</td></tr>
  <tr><td>codegen</td><td>2</td></tr>
  <tr><td>deci</td><td>1</td></tr>
  <tr><td>recurrent_gemma</td><td>1</td></tr>
  <tr><td>stablelm_alpha</td><td>1</td></tr>
  <tr><th>Total</th><th>1,018</th></tr>
</table>

Table 4: Number of models by model type.

date the model was created, the number of downloads over the past 30 days, and the model’s configuration details. All of this information is current as of February 1, 2025.

Among the model tags, we specifically used llama2, llama-2, license:llama2, llama3, llama-3, and license:llama3 to determine the model type (llama-1, llama-2, or llama-3). Furthermore, to identify language models that were pre-trained on the Pile in Section 4, we employed tags such as dataset:eleutherai/pile, dataset:eleutherai/the_pile, dataset:eleutherai/the_pile_deduplicated, and arxiv:2101.00027.

G.2 How the model type was determined

In principle, we used the value of model_type in the config retrieved from the Hugging Face Hub API as the model type. However, out of the 1,018 language models we examined, there were 587 whose config model_type was listed as llama. For these, we used the following procedure to determine whether they were the original Llama (llama-1), Llama-2, or Llama-3; if we were able to identify which version they were, we reclassified them as llama-1, llama-2, or llama-3 accordingly.

1. We checked the tags assigned to each model. Of these, 136 models that included any of llama2, llama-2, or license:llama2 were classified as llama-2. Similarly, 39 models that included any of llama3, llama-3, or license:llama3 were classified as llama-3.

2. For the remaining 412 models whose classification was not determined by tags alone, we used the creation date and the model name (converted to lowercase) to make a decision. First, 69 models that were created prior to July 18, 2023 (the Llama-2 release date) were classified as llama-1. Next, 88 models whose lowercase model name contained either llama2 or llama-2 were classified as llama-2. Among those whose lowercase model name contained llama3 or llama-3, 22 models whose creation date was after April 18, 2024 (the Llama-3 release date) were classified as llama-3.

3. After following the steps above, the 217 models that could not be classified were left as llama.

Furthermore, any model whose name prior to the slash (/) was deepseek-ai was defined as deepseek. In addition, even though abacusai/Llama-3-Smaug-8B was tagged with license:llama2, we manually reclassified it as llama-3.

Table 4 shows the number of models classified into each model type.

G.3 Basic information on the dataset

The dataset used in our experiments consists of a total of 10,000 texts, which are divided into 17 text categories. Table 5 shows the number of texts in each category.
<table>
  <tr>
    <th>Text category</th>
    <th>Texts</th>
  </tr>
  <tr>
    <td>Pile-CC</td>
    <td>2,353</td>
  </tr>
  <tr>
    <td>PubMed Central</td>
    <td>1,763</td>
  </tr>
  <tr>
    <td>ArXiv</td>
    <td>1,172</td>
  </tr>
  <tr>
    <td>Github</td>
    <td>925</td>
  </tr>
  <tr>
    <td>FreeLaw</td>
    <td>837</td>
  </tr>
  <tr>
    <td>StackExchange</td>
    <td>712</td>
  </tr>
  <tr>
    <td>Wikipedia (en)</td>
    <td>567</td>
  </tr>
  <tr>
    <td>USPTO Backgrounds</td>
    <td>487</td>
  </tr>
  <tr>
    <td>PubMed Abstracts</td>
    <td>464</td>
  </tr>
  <tr>
    <td>Gutenberg (PG-19)</td>
    <td>251</td>
  </tr>
  <tr>
    <td>DM Mathematics</td>
    <td>151</td>
  </tr>
  <tr>
    <td>EuroParl</td>
    <td>83</td>
  </tr>
  <tr>
    <td>HackerNews</td>
    <td>67</td>
  </tr>
  <tr>
    <td>Ubuntu IRC</td>
    <td>54</td>
  </tr>
  <tr>
    <td>PhilPapers</td>
    <td>51</td>
  </tr>
  <tr>
    <td>NIH ExPorter</td>
    <td>41</td>
  </tr>
  <tr>
    <td>Enron Emails</td>
    <td>22</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>10,000</th>
  </tr>
</table>

Table 5: Number of texts in each text category.

To assign colors to the text categories, we first compute the average text embedding for each category in the Pile using simcse-roberta-large (Gao et al., 2021b). Next, we calculate a tour over the 17 average embedding vectors by solving the traveling salesman problem\footnote{Inspired by Sato (2022); Yamagiwa et al. (2024).}. The TSP is solved using the nearest neighbor method to generate an initial tour, which is then refined using a 2-opt improvement procedure, and Euclidean distance is used as the metric. Based on the adjacency relationships along this tour, we segment the hue circle at equal intervals and color each category accordingly.

G.4 Standard scores

In the three experiments described below, we use standardized values\footnote{By subtracting the mean from each value and dividing by the standard deviation, the data is transformed to have a mean of 0 and a variance of 1.}, or \( Z \)-score normalization, of both the log-likelihood and the benchmark scores calculated for the \( K \) language models.

• In Figures 1 and 11, where we define each language model’s primary text category, we use the average log-likelihood for each category, standardized across all models.

• In Section 4, to determine each language model’s primary task, we standardize each task’s score across the \( K \) models.

• Furthermore, in the data leakage detection described in Section 4, we use the difference between the standardized mean log-likelihood and the standardized 6-TaskMean score as the indicator.

Figure 11: Model maps obtained by double centered log-likelihood matrix \( Q \). These maps correspond to Figure 1. (Top) Colors indicate model types. (Bottom) Colors indicate the text category in which each model attains the highest standardized log-likelihood score among 17 categories.

G.5 Hierarchical clustering settings

Figure 3 displays the double-centered log-likelihood matrix \( Q \), with hierarchical clustering applied to both its rows and columns. We implemented the clustering using SciPy (Virtanen et al., 2020). Distance matrices were computed using scipy.spatial.distance.pdist,
Figure 12: Model maps obtained by double centered log-likelihood matrix \( Q \). These maps correspond to Figure 5. These maps are illustrating model performance. From left to right, the panels show each model’s mean log-likelihood, 6-TaskMean score, and the “primary task,” which refers to the task where each model achieves the highest standardized score among the six tasks, color-coded accordingly. The color bar is clipped at the 10th percentile for mean log-likelihood and 6-TaskMean, with darker colors indicating better performance. In the primary task panel, models with standardized scores below zero across all six tasks are labeled as “All Under 0.”

Figure 13: Model maps obtained by double centered log-likelihood matrix \( Q \) color-coded by (Left) model size and (Right) model creation date.

and clustering was performed using scipy.cluster.hierarchy.linkage. For clustering models, we used sqeuclidean as the metric and median as the linkage method. For clustering texts, we used correlation as the metric and average as the linkage method. For the hierarchical clustering shown in Fig. 4, which presents a dendrogram of 100 language models, we used sqeuclidean as the metric and median as the linkage method. The vertical axis of the dendrogram uses the symmetric logarithmic scale (with a linear threshold of 250) implemented in matplotlib. To ensure that the values on the vertical axis correspond to the Kullback-Leibler divergence, we used the \( q \)-coordinates divided by \( \sqrt{2N} \).

H Additional Model Maps

In this section, we present additional model maps, including a figure that lists all the model names and a map obtained through dimensionality reduction of the double-centered log-likelihood matrix \( Q \).

H.1 Model map via the double centered log-likelihood matrix

In the main text, we use a model map generated by dimensionality reduction of the log-likelihood matrix \( L \). Here, in Figs. 11, 12 and 13, we present model maps obtained by dimensionality reduction of the double-centered log-likelihood matrix \( Q \).

H.2 Model map with model names

We present figures that display the model names corresponding to each point on the model maps defined in Section 4. For both the log-likelihood matrix \( L \) and the double-centered log-likelihood matrix \( Q \), we provide two types of maps: one colored by model type and another colored by primary text category. Figures 14 and 15 show the maps obtained by applying dimensionality reduction to \( L \), while Figures 16 and 17 show the maps obtained using \( Q \).

I Table of Nearest Neighbor Models

Table 6 presents the top 10 nearest neighbors among the 1,018 language models for each of the models highlighted in the top panel of Figure 1. Each sub-table corresponds to a specific model of interest, listing its nearest neighbors in descending order of the KL divergence. From this table, we can see that similar models tend to cluster together, exhibiting relatively small KL divergence values. Additionally, the values in parentheses denote the KL divergence measured in bits per byte of text. This conversion makes it easier to interpret the information cost per byte when comparing predictive distributions of different models.
Figure 14: Model map obtained by dimensionality reduction of the log-likelihood matrix \( L \). Each point on the model map is labeled with the corresponding model name.
<table>
  <tr>
    <th>DM Mathematics</th>
    <th>HackerNews</th>
  </tr>
  <tr>
    <td>ArXiv</td>
    <td>Pile-CC</td>
  </tr>
  <tr>
    <td>PubMed Central</td>
    <td>Wikipedia (en)</td>
  </tr>
  <tr>
    <td>PubMed Abstracts</td>
    <td>Gutenberg (PG-19)</td>
  </tr>
  <tr>
    <td>NIH ExPorter</td>
    <td>Enron Emails</td>
  </tr>
  <tr>
    <td>USPTO Backgrounds</td>
    <td>FreeLaw</td>
  </tr>
  <tr>
    <td>Github</td>
    <td>PhilPapers</td>
  </tr>
  <tr>
    <td>StackExchange</td>
    <td>EuroParl</td>
  </tr>
  <tr>
    <td>Ubuntu IRC</td>
    <td></td>
  </tr>
</table>

![Model map obtained by dimensionality reduction of the log-likelihood matrix \( \mathbf{L} \). Each point on the model map is labeled with the corresponding model name. Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 categories.](page_246_355_1057_1012.png)

Figure 15: Model map obtained by dimensionality reduction of the log-likelihood matrix \( \mathbf{L} \). Each point on the model map is labeled with the corresponding model name. Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 categories.
Figure 16: Model map obtained by dimensionality reduction of the double-centered log-likelihood matrix \( Q \). Each point on the model map is labeled with the corresponding model name.

![Model map obtained by dimensionality reduction of the double-centered log-likelihood matrix Q. Each point on the model map is labeled with the corresponding model name.](page_324_573_1097_682.png)
Figure 17: Model map obtained by dimensionality reduction of the double-centered log-likelihood matrix \( Q \). Each point on the model map is labeled with the corresponding model name. Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 categories.

![Model map obtained by dimensionality reduction of the double-centered log-likelihood matrix Q. Each point on the model map is labeled with the corresponding model name. Colors indicate the model’s “primary text category,” the text category where the model achieves the highest standardized log-likelihood score among 17 categories.](page_324_563_1097_1012.png)
Table 6: Top 10 nearest neighbors among the 1,018 language models for each model labeled in the top panel of Fig. 1. The values indicate the KL divergence measured in bits per byte (bpb), as defined in Section 3.5. These are computed using formula (3) in Section 2, by multiplying the original KL divergence by 0.001484.
Figure 18: Scatter plots of predicted scores versus benchmark scores for test sets across the six benchmark tasks (the dashed line indicates the identity line). Additionally, results for predicting 6-TaskMean (identical to Fig. 8) and the mean log-likelihood are also shown. Each point is color-coded by the mean log-likelihood, with higher mean log-likelihood values generally corresponding to higher task scores. For better visualization, the color bar range is clipped to the 10th–100th percentile.

J Details of Weight Interpolation

In this section, we describe the experimental details concerning the relationship between model coordinates and model weights, as introduced in Section 6.3.

Constructing the weight grid. Let \( p_0 \) denote the base model, and \( p_1 \) and \( p_2 \) denote the fine-tuned models derived from \( p_0 \). We denote the weight parameter vectors of these models as \( W_0, W_1, \) and \( W_2 \), respectively. To construct the weight grid, we merged the model weights using the following linear operation:

\[
W_{\alpha,\beta} = W_0 + \alpha(W_1 - W_0) + \beta(W_2 - W_0),
\]

where the merge ratios \( \alpha, \beta \in \mathbb{R} \) were chosen from 36 evenly spaced combinations within the interval \([0, 1]\): \(\{0.0, 0.2, 0.6, 0.8, 1.0\}\). The original models \( W_0, W_1, \) and \( W_2 \) correspond to \( W_{0,0}, W_{1,0}, \) and \( W_{0,1} \), respectively. When \( \alpha + \beta \leq 1 \), the operation corresponds to linear interpolation between models. Even among models with the same architecture, the sizes of the embedding/unembedding matrices may differ. In such cases, we truncated or reshaped the weight parameters to match the base model.

Computing model coordinates. For each composed model \( p_{\alpha,\beta} \) with weight \( W_{\alpha,\beta} \), we computed the model coordinates \( q_{\alpha,\beta} \) following the method described in Section 2. The tokenizer of the base model was used to ensure consistency. The text data consists of all 10,000 texts prepared in Section 3.1. Additionally, the deterministic algorithms option\(^{20}\) was enabled in the implementation to ensure reproducibility.

Selection of models. We conducted experiments using two base models: Llama-2-7b-hf and mistralai/Mistral-7B-v0.1. For each base model \( p_0 \), we selected the two most downloaded fine-tuned models available on Hugging Face. Specifically, when \( p_0 = \) Llama-2-7b-hf, we set \( p_1 = \) vicuna-7b-v1.5 and \( p_2 = \) Llama-2-7b-chat-hf. When \( p_0 = \) mistralai/Mistral-7B-v0.1, we set \( p_1 = \) HuggingFaceH4/zephyr-7b-beta and \( p_2 = \) mistralai/Mistral-7B-Instruct-v0.1.

Visualization. Figures 10 and 19 show the linearly merged models, visualized in both weight space and log-likelihood space. The corners of the grid are labeled with their corresponding \( (\alpha, \beta) \) values.

In the left panel, we visualized \( W_{\alpha,\beta} \) in the weight space. Since the dimensionality of \( W_{\alpha,\beta} \), i.e., the number of model parameters, is extremely high, we employed a 2D projection method using the norms of the difference vectors \( r_1 = \|W_1 - W_0\|_2, r_2 = \|W_2 - W_0\|_2 \), and the angle between them, \( \phi = \arccos((W_1 - W_0)^T(W_2 - W_0)/r_1 r_2) \). Each point was placed at \( (\alpha r_1 + \beta r_2 \cos \phi, \beta r_2 \sin \phi) \).

In the right panel, we visualized the model coordinates \( q_{\alpha,\beta} \) by Principal Component Analysis

20 torch.use_deterministic_algorithms(True)
Figure 19: Visualization of 36 language models obtained by linearly interpolating pretrained model weights based on mistralai/Mistral-7B-v0.1. Each point is color-coded according to its mean log-likelihood. (Left) Models in the weight parameter space. (Right) Models in the log-likelihood space, represented by the \( q \)-coordinate system.

Figure 20: Scatter plots comparing the actual and predicted mean log-likelihoods for 36 interpolated models derived from Llama-2-7b-hf (left) and Mistral-7B-v0.1 (right). Each point corresponds to a unique combination of merge ratios (\( \alpha, \beta \)) and is color-coded accordingly. The dashed line indicates ideal prediction. Strong correlations are observed for both model sets: Llama-2 shows \( r = 0.969, \rho = 0.996 \), and Mistral shows \( r = 0.973, \rho = 0.975 \). A 2D color map of \( (\alpha, \beta) \) values is shown as an inset.

(PCAs). Each model \( p_{\alpha, \beta} \) was mapped onto the \( q \)-coordinate system to analyze the structure of the interpolated models.

Estimating the mean log-likelihood of interpolated models. To assess the potential of predicting task performance from log-likelihood vectors without explicitly computing them for every interpolated model, we evaluated whether the mean log-likelihoods can be estimated by linearly interpolating those of the base models. Let \( \ell_{\alpha, \beta}(x) \) denote the log-likelihood of input \( x \) computed by the merged model \( p_{\alpha, \beta} \). Let \( \ell_0, \ell_1, \) and \( \ell_2 \) be the log-likelihoods from the base and fine-tuned models \( p_0, p_1, \) and \( p_2 \), respectively. We define the linearly interpolated log-likelihood as:

\[
\hat{\ell}_{\alpha, \beta} = \ell_0 + \alpha(\ell_1 - \ell_0) + \beta(\ell_2 - \ell_0).
\]

Figure 20 shows a strong correlation between the predicted and actual mean log-likelihoods. This result suggests that log-likelihood vectors can be approximated by linear interpolation, enabling efficient prediction of model performance without computing log-likelihoods for every model.

K Details of Model Performance Prediction

This section provides additional details on the prediction of benchmark scores using model coordinates, as discussed in Section 5.

K.1 Details of ridge regression

As described in Section 5.2, ridge regression requires setting a regularization strength parameter, \( \alpha \). To determine \( \alpha \) from \( \{10^1, \ldots, 10^9\} \), we performed a five-fold cross-validation within each training dataset\(^{21}\). As a post-processing step, we clipped the predicted scores to the range [0, 100].

For the setting where the target variable \( v \in \mathbb{R}^K \) was replaced with the mean log-likelihood \( (\bar{\ell}_1, \ldots, \bar{\ell}_K) \in \mathbb{R}^K \), we searched for \( \alpha \) within \( \{10^{-4}, \ldots, 10^4\} \) and did not apply clipping as a post-processing step.

K.2 Details of prediction results

Figure 18 shows scatter plots of predicted scores and actual benchmark scores for each benchmark task, as well as for 6-TaskMean and the mean log-likelihood. As in Fig. 8, the scatter plots show strong correlations for all six benchmark tasks and for the mean log-likelihood.

To account for randomness, we ran five different data splits based on model types when predicting each benchmark score. As explained in Section 5, the final predicted score was the average of these five runs. Additionally, for each split, we computed the correlation coefficients between the predicted scores and the actual benchmark scores. Table 7 presents their mean and standard deviation, and shows a similar trend as Table 2.

K.3 Results from different settings

To investigate potential data leakage due to model types, Table 8 shows the correlation coefficients

\(^{21}\)The training dataset consisted of four out of the five folds obtained by splitting the dataset for each benchmark task.
<table>
  <tr>
    <th></th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Winogrande</th>
    <th>GSM8K</th>
    <th>Average</th>
    <th>mean log-likelihood</th>
  </tr>
  <tr>
    <td>Pearson’s \( r \)</td>
    <td>0.941 ± 0.002</td>
    <td>0.904 ± 0.005</td>
    <td>0.924 ± 0.006</td>
    <td>0.889 ± 0.013</td>
    <td>0.934 ± 0.005</td>
    <td>0.864 ± 0.019</td>
    <td>0.947 ± 0.005</td>
    <td>0.987 ± 0.006</td>
  </tr>
  <tr>
    <td>Spearman’s \( \rho \)</td>
    <td>0.944 ± 0.005</td>
    <td>0.951 ± 0.003</td>
    <td>0.926 ± 0.005</td>
    <td>0.875 ± 0.003</td>
    <td>0.943 ± 0.009</td>
    <td>0.841 ± 0.013</td>
    <td>0.956 ± 0.004</td>
    <td>0.971 ± 0.006</td>
  </tr>
</table>

Table 7: Group 5-fold based on model types using \( Q \): Mean and standard deviation of the correlation coefficients between predicted and actual benchmark scores. Coefficients were obtained by ridge regression under five data splits based on model types. Results for predicting 6-TaskMean and the mean log-likelihood are also included.

<table>
  <tr>
    <th></th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Winogrande</th>
    <th>GSM8K</th>
    <th>Average</th>
    <th>mean log-likelihood</th>
  </tr>
  <tr>
    <td>Pearson’s \( r \)</td>
    <td>0.968 ± 0.001</td>
    <td>0.939 ± 0.004</td>
    <td>0.960 ± 0.002</td>
    <td>0.952 ± 0.001</td>
    <td>0.960 ± 0.003</td>
    <td>0.931 ± 0.001</td>
    <td>0.973 ± 0.001</td>
    <td>0.994 ± 0.001</td>
  </tr>
  <tr>
    <td>Spearman’s \( \rho \)</td>
    <td>0.972 ± 0.001</td>
    <td>0.970 ± 0.002</td>
    <td>0.966 ± 0.001</td>
    <td>0.929 ± 0.001</td>
    <td>0.969 ± 0.001</td>
    <td>0.891 ± 0.004</td>
    <td>0.976 ± 0.001</td>
    <td>0.990 ± 0.000</td>
  </tr>
</table>

Table 8: Random 5-fold using \( Q \): Mean and standard deviation of the correlation coefficients, obtained as in Table 7, but using five random data splits instead of splits based on model types.

<table>
  <tr>
    <th></th>
    <th>ARC</th>
    <th>HellaSwag</th>
    <th>MMLU</th>
    <th>TruthfulQA</th>
    <th>Winogrande</th>
    <th>GSM8K</th>
    <th>Average</th>
    <th>mean log-likelihood</th>
  </tr>
  <tr>
    <td>Pearson’s \( r \)</td>
    <td>0.939 ± 0.002</td>
    <td>0.903 ± 0.007</td>
    <td>0.923 ± 0.006</td>
    <td>0.890 ± 0.013</td>
    <td>0.935 ± 0.006</td>
    <td>0.863 ± 0.018</td>
    <td>0.945 ± 0.005</td>
    <td>1.000 ± 0.000</td>
  </tr>
  <tr>
    <td>Spearman’s \( \rho \)</td>
    <td>0.943 ± 0.005</td>
    <td>0.952 ± 0.003</td>
    <td>0.925 ± 0.005</td>
    <td>0.875 ± 0.003</td>
    <td>0.944 ± 0.008</td>
    <td>0.841 ± 0.013</td>
    <td>0.954 ± 0.004</td>
    <td>1.000 ± 0.000</td>
  </tr>
</table>

Table 9: Group 5-fold based on model types using \( L \): Mean and standard deviation of the correlation coefficients, obtained as in Table 7, but using the matrix \( L \) instead of the matrix \( Q \) in the ridge regression of (5) in Section 5.

using five random data splits. These results demonstrate higher correlation coefficients across all tasks compared to those obtained using splits based on model types (Table 7), suggesting that randomly splitting models may unintentionally simplify the prediction task due to leakage from model types.

As shown in (5) in Section 5, we performed ridge regression using the double-centered log-likelihood matrix \( Q \), derived from the log-likelihood matrix \( L \). To assess the effect of replacing \( Q \) with \( L \), we repeated the analysis using \( L \) and report the resulting means and standard deviations of the correlation coefficients in Table 9. The results obtained with \( L \) (Table 9) show nearly the same trend as those with \( Q \) (Table 7). Note, however, that the mean of each row of \( L \) equals the mean log-likelihood itself, so this specific target can be reproduced exactly. Consequently, the mean correlation coefficients are 1.000 for both Pearson’s \( r \) and Spearman’s \( \rho \).

L Model List

Table 10 lists the 1,018 models used in this study, sorted alphabetically by their names. The BibTeX entries cited for each model were determined through the following procedure.

First, we extracted the BibTeX entries available in each model’s Hugging Face model card\(^{22}\). If the BibTeX entry was missing a year of publication, we filled it in with the model’s creation date\(^{23}\). Additionally, we generated BibTeX entries using the arXiv IDs found in the model card tags by querying the arXiv API\(^{24}\). This process resulted in a set of BibTeX entries for each model.

Next, we manually checked pairs of different BibTeX entries where the title similarity\(^{25}\) was high, or the authors matched, to determine whether they corresponded to the same source. This step allowed us to create groups of BibTeX entries that were considered identical.

Then, for each BibTeX group, we selected a representative entry as follows. Within each group, the entry most frequently cited by the models was chosen as the representative. If multiple candidates met this criterion, we prioritized BibTeX entries generated from arXiv IDs when available. If no such entry existed, we selected the one with the longest string.

Finally, we replaced each model’s BibTeX entry with the representative entry from its corresponding group. Any selected BibTeX entry that contained typos or formatting errors was manually corrected based on compilation errors. If the author information was incomplete, we corrected it manually by checking the source.

Note that for google/codgemma-2b and deepseek-ai/deepseek-llm-7b-base in Table 1, as well as for deepseek-ai/deepseek-coder-1.3b-base and mistralai/Mistral-7B-v0.3 in Table 6, we

\(^{22}\)https://github.com/huggingface/huggingface_hub.
\(^{23}\)https://github.com/sciunto-org/python-bibtexparser.
\(^{24}\)https://github.com/lukasschwab/arxiv.py.
\(^{25}\)We used Python’s difflib.SequenceMatcher.
manually prepared the BibTeX entries for citation based on their respective sources. We also used the same BibTeX entries for all other models that were considered to be of the same type.
Table 10: List of 1018 models. “ID” denotes the alphabetical index; “Model Name” denotes the name of the model; “Model Type” denotes the classification defined in this paper; “Size” denotes the size of the model (B: billion); “Date” denotes the date of model creation; “DLs” denotes the total number of downloads; \( \ell_i \) denotes the mean log-likelihood; “Task” denotes the mean of the 6 benchmark scores (i.e., 6-TaskMean).

<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLs</th>
    <th>\( \ell_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>1</td><td>aaditya/lama3-OpenBioLLM-8B (Singhal et al., 2022, 2023; Nori et al., 2023; Raffaiol et al., 2024; Pal and Sankarasubbu, 2024a)</td><td>llama-3</td><td>8B</td><td>2024-04-20</td><td>9308</td><td>-590.14</td><td>54.06</td></tr>
  <tr><td>2</td><td>aari1995/germo-7b-laser</td><td>mistral</td><td>7B</td><td>2024-01-09</td><td>2351</td><td>-686.18</td><td>62.82</td></tr>
  <tr><td>3</td><td>abacusai/bigstral-12b-32k</td><td>mistral</td><td>12B</td><td>2024-03-06</td><td>5216</td><td>-632.41</td><td>62.17</td></tr>
  <tr><td>4</td><td>abacusai/Giraffe-13b-32k-v3</td><td>llama-2</td><td>13B</td><td>2023-12-06</td><td>1214</td><td>-546.12</td><td>57.24</td></tr>
  <tr><td>5</td><td>abacusai/Llama-3-Smaug-8B (Pal et al., 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-19</td><td>12873</td><td>-565.82</td><td>64.61</td></tr>
  <tr><td>6</td><td>abacusai/Serp-5B-mistral</td><td>mistral</td><td>7B</td><td>2024-05-01</td><td>4050</td><td>-550.17</td><td>61.10</td></tr>
  <tr><td>7</td><td>abhinand/llama-3-OpenBioMed-8B-selp-v0.3</td><td>llama-3</td><td>8B</td><td>2024-05-02</td><td>2713</td><td>-558.51</td><td>62.08</td></tr>
  <tr><td>8</td><td>abhinand/umil-llama-7b-base-v0.1 (Balachandran, 2023)</td><td>llama-2</td><td>7B</td><td>2023-11-08</td><td>1382</td><td>-877.00</td><td>44.52</td></tr>
  <tr><td>9</td><td>abhinand/umil-llama-7b-instruct-v0.1 (Balachandran, 2023)</td><td>llama-2</td><td>7B</td><td>2023-11-08</td><td>3507</td><td>-708.97</td><td>45.52</td></tr>
  <tr><td>10</td><td>abhishekchohan/mistral-7B-forest-dpo</td><td>mistral</td><td>7B</td><td>2024-01-21</td><td>1961</td><td>-561.51</td><td>63.28</td></tr>
  <tr><td>11</td><td>abhishekchohan/Yi-9B-Forest-DPO-v1.0</td><td>llama</td><td>9B</td><td>2024-03-18</td><td>2755</td><td>-528.97</td><td>64.11</td></tr>
  <tr><td>12</td><td>acraSt/Bear-3B</td><td>llama</td><td>3B</td><td>2023-09-02</td><td>1191</td><td>-592.53</td><td>40.18</td></tr>
  <tr><td>13</td><td>acraSt/Griffin-3B</td><td>llama</td><td>3B</td><td>2023-08-18</td><td>1198</td><td>-584.63</td><td>41.13</td></tr>
  <tr><td>14</td><td>acraSt/Mars-3B-V2</td><td>llama</td><td>3B</td><td>2023-08-22</td><td>1234</td><td>-609.79</td><td>42.08</td></tr>
  <tr><td>15</td><td>acraSt/OmniLaMA-3B</td><td>llama</td><td>3B</td><td>2023-08-26</td><td>1201</td><td>-640.98</td><td>38.28</td></tr>
  <tr><td>16</td><td>acraSt/Puma-3B</td><td>llama</td><td>3B</td><td>2023-08-16</td><td>1198</td><td>-584.54</td><td>41.02</td></tr>
  <tr><td>17</td><td>acraSt/RedPajama-INCITE-Chat-Instruct-3B-V1</td><td>gpt_neox</td><td>2B</td><td>2023-07-27</td><td>1202</td><td>-565.14</td><td>39.23</td></tr>
  <tr><td>18</td><td>adam1139/Mistral-7B-AEZAKMI-v1</td><td>mistral</td><td>7B</td><td>2023-11-27</td><td>1199</td><td>-596.56</td><td>54.92</td></tr>
  <tr><td>19</td><td>adonlee/LLAMA-2_13B_SFT_v0</td><td>llama</td><td>13B</td><td>2023-10-03</td><td>1268</td><td>-556.59</td><td>57.31</td></tr>
  <tr><td>20</td><td>adonlee/LLAMA-2_13B_SFT_v1</td><td>llama</td><td>13B</td><td>2023-11-02</td><td>1265</td><td>-548.75</td><td>63.04</td></tr>
  <tr><td>21</td><td>adonlee/LLAMA-2_13B_SFT_v2</td><td>llama</td><td>13B</td><td>2023-11-02</td><td>1241</td><td>-513.43</td><td>63.94</td></tr>
  <tr><td>22</td><td>ahmyconchan/OpenOrcaAYT-13B</td><td>llama-2</td><td>13B</td><td>2023-09-07</td><td>1184</td><td>-569.10</td><td>–</td></tr>
  <tr><td>23</td><td>AIChenKai/TinyLlama-1.1B-Chat-v1.0-x2-MoE</td><td>mistral</td><td>1B</td><td>2024-01-03</td><td>1217</td><td>-619.71</td><td>36.98</td></tr>
  <tr><td>24</td><td>AIJUUD/jud-Mistral-7B</td><td>mistral</td><td>7B</td><td>2024-01-31</td><td>1312</td><td>-564.68</td><td>61.72</td></tr>
  <tr><td>25</td><td>AIJUUD/jud-Mistral-7B-dpo (Lacoste et al., 2019)</td><td>mistral</td><td>7B</td><td>2024-02-07</td><td>3217</td><td>-567.00</td><td>60.89</td></tr>
  <tr><td>26</td><td>ajibawa-2023/carl-7b</td><td>llama</td><td>7B</td><td>2023-07-22</td><td>1212</td><td>-599.09</td><td>46.16</td></tr>
  <tr><td>27</td><td>ajibawa-2023/cde-13B</td><td>llama</td><td>13B</td><td>2023-12-08</td><td>1181</td><td>-604.28</td><td>54.81</td></tr>
  <tr><td>28</td><td>ajibawa-2023/Python-Code-13B</td><td>llama</td><td>13B</td><td>2023-11-11</td><td>1211</td><td>-582.46</td><td>56.61</td></tr>
  <tr><td>29</td><td>ajibawa-2023/uncensored-Frank-13B</td><td>llama</td><td>13B</td><td>2023-12-07</td><td>1177</td><td>-608.08</td><td>60.39</td></tr>
  <tr><td>30</td><td>ajibawa-2023/uncensored-Frank-7B</td><td>llama</td><td>7B</td><td>2023-09-14</td><td>1192</td><td>-577.46</td><td>55.64</td></tr>
  <tr><td>31</td><td>ajibawa-2023/uncensored-Jordan-13B</td><td>llama</td><td>13B</td><td>2023-10-23</td><td>1174</td><td>-578.48</td><td>56.27</td></tr>
  <tr><td>32</td><td>ajibawa-2023/uncensored-Jordan-7B</td><td>llama</td><td>7B</td><td>2023-10-23</td><td>1174</td><td>-637.49</td><td>49.95</td></tr>
  <tr><td>33</td><td>akjindal3244/Mistral-7B-v0.1-Open-Platypus</td><td>mistral</td><td>7B</td><td>2023-10-05</td><td>1286</td><td>-543.69</td><td>58.92</td></tr>
  <tr><td>34</td><td>AlekseyKorshuk/pymalion-6b-vicuna-a-chatml</td><td>gptj</td><td>6B</td><td>2023-06-22</td><td>1169</td><td>-640.90</td><td>48.08</td></tr>
  <tr><td>35</td><td>alifan/Alifan-7B-zeppelin-7B-full</td><td>mistral</td><td>7B</td><td>2023-12-19</td><td>1185</td><td>-732.57</td><td>47.87</td></tr>
  <tr><td>36</td><td>alifan/OLMO-7B-hf (Touvron et al., 2023a; Groeneveld et al., 2024)</td><td>olmo</td><td>1B</td><td>2024-04-12</td><td>15221</td><td>-619.51</td><td>36.78</td></tr>
  <tr><td>37</td><td>alifan/OLMO-7B-hf (Touvron et al., 2023a; Groeneveld et al., 2024)</td><td>olmo</td><td>6B</td><td>2024-04-12</td><td>5407</td><td>-554.02</td><td>43.36</td></tr>
  <tr><td>38</td><td>allknowingroger/MultiverseEx26-7B-selp</td><td>mistral</td><td>7B</td><td>2024-03-30</td><td>3034</td><td>-599.87</td><td>76.80</td></tr>
  <tr><td>39</td><td>alnrg2arg/blockchainlabs_7B_merged_test2_4</td><td>mistral</td><td>7B</td><td>2024-01-17</td><td>1455</td><td>-577.39</td><td>75.28</td></tr>
  <tr><td>40</td><td>alnrg2arg/blockchainlabs_7B_merged_test2_4_prune</td><td>mistral</td><td>7B</td><td>2024-01-18</td><td>1939</td><td>-673.45</td><td>57.91</td></tr>
  <tr><td>41</td><td>aloboun/bun_mistral_7b_v2</td><td>mistral</td><td>7B</td><td>2023-12-20</td><td>1305</td><td>-555.96</td><td>59.76</td></tr>
  <tr><td>42</td><td>aloboun/falcon-1b-cot2</td><td>falcon</td><td>2B</td><td>2023-12-08</td><td>1207</td><td>-715.39</td><td>56.56</td></tr>
  <tr><td>43</td><td>aloboun/falcon-2b-elm3</td><td>llama</td><td>2B</td><td>2023-12-08</td><td>1207</td><td>-604.18</td><td>41.14</td></tr>
  <tr><td>44</td><td>andreaskoepf/llama2-13h-megacode2_min100</td><td>llama-2</td><td>13B</td><td>2023-08-14</td><td>1162</td><td>-571.36</td><td>56.92</td></tr>
  <tr><td>45</td><td>AndronOe/YetAnother_Open-Llama-3B-LoRA</td><td>llama</td><td>3B</td><td>2023-07-21</td><td>1163</td><td>-646.46</td><td>–</td></tr>
  <tr><td>46</td><td>argilla/notus-7b-v1</td><td>mistral</td><td>7B</td><td>2023-11-16</td><td>7660</td><td>-564.14</td><td>60.22</td></tr>
  <tr><td>47</td><td>Artplee/L-MChat-Small</td><td>phi</td><td>2B</td><td>2024-04-11</td><td>2793</td><td>-640.49</td><td>63.14</td></tr>
  <tr><td>48</td><td>Artplee/L-MChat-7b</td><td>mistral</td><td>7B</td><td>2024-04-02</td><td>9634</td><td>-557.41</td><td>69.57</td></tr>
  <tr><td>49</td><td>Aspik101/StableBench-13B-instruct-PL-lora_unload</td><td>llama-2</td><td>13B</td><td>2023-08-13</td><td>1244</td><td>-546.42</td><td>57.40</td></tr>
  <tr><td>50</td><td>Aspik101/StableBench-13B-instruct-PL-lora_unload</td><td>llama-2</td><td>13B</td><td>2023-08-13</td><td>1243</td><td>-546.50</td><td>57.42</td></tr>
  <tr><td>51</td><td>Aspik101/WizardVicuna-Uncensored-3B-instruct-PL-lora_unload</td><td>llama-2</td><td>13B</td><td>2023-08-07</td><td>1161</td><td>-626.59</td><td>39.95</td></tr>
  <tr><td>52</td><td>AtAndDev/CapybaraMarcoroni-7B</td><td>mistral</td><td>7B</td><td>2024-01-03</td><td>1162</td><td>-541.33</td><td>70.32</td></tr>
  <tr><td>53</td><td>athirdpath/NFSW_DPO_Normomaid-7b</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>1280</td><td>-546.50</td><td>61.59</td></tr>
  <tr><td>54</td><td>augmxnt/shisa-base-7b-v1</td><td>mistral</td><td>7B</td><td>2023-11-19</td><td>1188</td><td>-648.62</td><td>51.64</td></tr>
  <tr><td>55</td><td>augmxnt/shisa-gamma-7b-v1</td><td>mistral</td><td>7B</td><td>2023-12-23</td><td>151426</td><td>-590.25</td><td>55.50</td></tr>
  <tr><td>56</td><td>augmxnt/shisa-v7b-v1 (Jain et al., 2023; Raffaiol et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-11-22</td><td>1195</td><td>-643.26</td><td>55.01</td></tr>
  <tr><td>57</td><td>Austins/chromes-hermes-13b-v2</td><td>llama-2</td><td>13B</td><td>2023-08-12</td><td>1227</td><td>-584.56</td><td>56.10</td></tr>
  <tr><td>58</td><td>AzazelLe/TitanShadowExperiment28-7B</td><td>mistral</td><td>7B</td><td>2023-10-18</td><td>3152</td><td>-595.65</td><td>68.86</td></tr>
  <tr><td>59</td><td>AzazelLe/Dumb-Maidlet</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1208</td><td>-560.37</td><td>69.64</td></tr>
  <tr><td>60</td><td>AzazelLe/HalfNSFW_Normomaid-7b</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1201</td><td>-548.86</td><td>68.34</td></tr>
  <tr><td>61</td><td>AzazelLe/Maylin-7b</td><td>mistral</td><td>7B</td><td>2023-12-29</td><td>1208</td><td>-547.63</td><td>62.32</td></tr>
  <tr><td>62</td><td>AzazelLe/Silicon-Medley</td><td>mistral</td><td>7B</td><td>2023-12-29</td><td>1200</td><td>-566.66</td><td>69.49</td></tr>
  <tr><td>63</td><td>AzazelLe/SimMelodicMaid</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1205</td><td>-577.28</td><td>69.70</td></tr>
  <tr><td>64</td><td>AzazelLe/solo-llama-7b</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1207</td><td>-577.15</td><td>67.05</td></tr>
  <tr><td>65</td><td>AzazelLe/Titan-Topp-7b</td><td>mistral</td><td>7B</td><td>2024-01-03</td><td>1197</td><td>-560.94</td><td>62.58</td></tr>
  <tr><td>66</td><td>AzazelLe/XDAN-SlimOrcia</td><td>mistral</td><td>7B</td><td>2023-12-29</td><td>1206</td><td>-575.20</td><td>68.04</td></tr>
  <tr><td>67</td><td>AzazelLe/Yuna-7b-Merge</td><td>mistral</td><td>7B</td><td>2024-01-05</td><td>1201</td><td>-580.65</td><td>71.46</td></tr>
  <tr><td>68</td><td>Azure99/blossom-v1-3b</td><td>bloom</td><td>3B</td><td>2023-07-29</td><td>1228</td><td>-641.52</td><td>36.90</td></tr>
  <tr><td>69</td><td>Azure99/blossom-v2-llama2-7b</td><td>llama-2</td><td>7B</td><td>2023-09-06</td><td>1241</td><td>-572.17</td><td>51.71</td></tr>
  <tr><td>70</td><td>Azure99/blossom-v2-3b</td><td>bloom</td><td>3B</td><td>2023-08-08</td><td>1237</td><td>-654.44</td><td>35.98</td></tr>
  <tr><td>71</td><td>Azure99/blossom-v3-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-11-20</td><td>1319</td><td>-565.31</td><td>62.95</td></tr>
  <tr><td>72</td><td>Azure99/blossom-v3_1-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-11-22</td><td>1317</td><td>-567.17</td><td>62.53</td></tr>
  <tr><td>73</td><td>Azure99/blossom-v4-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-12-26</td><td>1315</td><td>-554.37</td><td>63.61</td></tr>
  <tr><td>74</td><td>BarryFuturemap/NeuralTurboVariant1-7B</td><td>mistral</td><td>7B</td><td>2024-01-22</td><td>1170</td><td>-592.94</td><td>74.83</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>79</td><td>beaugogo/Llama2-7b-openorca-mc-v1</td><td>llama-2</td><td>7B</td><td>2023-08-20</td><td>1177</td><td>-613.56</td><td>52.24</td></tr>
  <tr><td>80</td><td>beaugogo/Llama2-7b-openorca-mc-v2-dpo</td><td>llama-2</td><td>7B</td><td>2023-10-06</td><td>1173</td><td>-608.74</td><td>52.32</td></tr>
  <tr><td>81</td><td>beaugogo/Llama2-7b-sharegpt4</td><td>llama-2</td><td>7B</td><td>2023-08-12</td><td>1183</td><td>-600.22</td><td>51.05</td></tr>
  <tr><td>82</td><td>beaugogo/ptyhia-1.4-deduped-sharegpt</td><td>gpt_neox</td><td>1B</td><td>2023-07-25</td><td>1293</td><td>-557.93</td><td>35.11</td></tr>
  <tr><td>83</td><td>BEE-spoke-data/Mixtral-GQA-400m-v2</td><td>mixtral</td><td>2B</td><td>2023-12-20</td><td>1182</td><td>-794.25</td><td>28.45</td></tr>
  <tr><td>84</td><td>beomi/KoAlpaca-KoRWKV-6B</td><td>rwkv</td><td>6B</td><td>2023-06-02</td><td>2288</td><td>-1099.15</td><td>28.57</td></tr>
  <tr><td>85</td><td>beomi/KoAlpaca-Polyglot-5.8B</td><td>gpt_neox</td><td>6B</td><td>2023-03-16</td><td>4198</td><td>-1309.35</td><td>29.46</td></tr>
  <tr><td>86</td><td>beomi/KoRWKV-6B</td><td>rwkv</td><td>6B</td><td>2023-05-26</td><td>2127</td><td>-1148.78</td><td>28.19</td></tr>
  <tr><td>87</td><td>beomi/Llama2-7b-7k (Lee, 2023)</td><td>llama-2</td><td>7B</td><td>2023-07-20</td><td>5287</td><td>-589.43</td><td>32.22</td></tr>
  <tr><td>88</td><td>beomi/Llama2-7k (Lee, 2024b)</td><td>llama</td><td>7B</td><td>2023-11-30</td><td>4261</td><td>-589.43</td><td>30.27</td></tr>
  <tr><td>89</td><td>beowulfCodeNinja-1.0-OpenChat-7B</td><td>mistral</td><td>7B</td><td>2023-12-20</td><td>6300</td><td>-555.93</td><td>67.40</td></tr>
  <tr><td>90</td><td>berkeley-nest/Stirling-LM-7B-alpha (Zhu et al., 2023a,b)</td><td>mistral</td><td>7B</td><td>2023-11-25</td><td>18366</td><td>-571.63</td><td>67.05</td></tr>
  <tr><td>91</td><td>bhavijawade/SOLAR-10B-OrcadPPO-Jawade</td><td>llama</td><td>10B</td><td>2024-01-06</td><td>1224</td><td>-563.19</td><td>74.27</td></tr>
  <tr><td>92</td><td>bigcode/gpt_bigcode-santacoder</td><td>gpt_bigcode</td><td>1B</td><td>2023-04-06</td><td>40198</td><td>-861.42</td><td>28.49</td></tr>
  <tr><td>93</td><td>bigcode/starcodebase-1b (Shazeer, 2019; Dao et al., 2022; Bavarian et al., 2022; Li et al., 2023a)</td><td>gpt_bigcode</td><td>1B</td><td>2023-07-03</td><td>5247</td><td>-734.99</td><td>30.06</td></tr>
  <tr><td>94</td><td>bigcode/starcodebase-7b (Shazeer, 2019; Dao et al., 2022; Bavarian et al., 2022; Li et al., 2023a)</td><td>gpt_bigcode</td><td>7B</td><td>2023-07-26</td><td>3566</td><td>-652.23</td><td>33.75</td></tr>
  <tr><td>95</td><td>bigcode/starcode2-3b (Bellagy et al., 2020; Dao et al., 2022; Bavarian et al., 2022; Ainslie et al., 2023; Lozhkov et al., 2024)</td><td>starcode2</td><td>3B</td><td>2023-11-29</td><td>445316</td><td>-620.70</td><td>39.25</td></tr>
  <tr><td>96</td><td>bigcode/starcode2-7b (Bellagy et al., 2020; Dao et al., 2022; Bavarian et al., 2022; Ainslie et al., 2023; Lozhkov et al., 2024)</td><td>starcode2</td><td>7B</td><td>2024-02-20</td><td>11834</td><td>-584.03</td><td>42.95</td></tr>
  <tr><td>97</td><td>bigscience/bloomz-3b (Muenninghoff et al., 2023)</td><td>bloom</td><td>3B</td><td>2022-10-08</td><td>7975</td><td>-701.42</td><td>37.03</td></tr>
  <tr><td>98</td><td>bigscience/bloomz-7b1 (Muenninghoff et al., 2023)</td><td>bloom</td><td>7B</td><td>2022-09-27</td><td>12152</td><td>-652.34</td><td>42.21</td></tr>
  <tr><td>99</td><td>bigscience/bloomz-7b1-mt (Muenninghoff et al., 2023)</td><td>bloom</td><td>7B</td><td>2022-09-28</td><td>2427</td><td>-653.08</td><td>42.14</td></tr>
  <tr><td>100</td><td>bigscience/bloom-1b1 (Shoeybi et al., 2020; Dettmers et al., 2022; Press et al., 2022)</td><td>bloom</td><td>1B</td><td>2022-05-19</td><td>11054</td><td>-687.97</td><td>32.47</td></tr>
  <tr><td>101</td><td>bigscience/bloom-1b7 (Shoeybi et al., 2020; Dettmers et al., 2022; Press et al., 2022)</td><td>bloom</td><td>1B</td><td>2022-05-19</td><td>40840</td><td>-657.05</td><td>33.98</td></tr>
  <tr><td>102</td><td>bigscience/bloom-3b (Shoeybi et al., 2020; Dettmers et al., 2022; Press et al., 2022)</td><td>bloom</td><td>3B</td><td>2022-05-19</td><td>13914</td><td>-633.79</td><td>36.07</td></tr>
  <tr><td>103</td><td>bigscience/bloom-7b1 (Shoeybi et al., 2020; Dettmers et al., 2022; Press et al., 2022)</td><td>bloom</td><td>7B</td><td>2022-05-19</td><td>21428</td><td>-604.20</td><td>39.18</td></tr>
  <tr><td>104</td><td>BioMistral/BioMistral-7B (Labrak et al., 2024)</td><td>mistral</td><td>7B</td><td>2024-02-14</td><td>11349</td><td>-652.04</td><td>52.33</td></tr>
  <tr><td>105</td><td>BioMistral/BioMistral-7B-DARE (Yadav et al., 2023a; Labrak et al., 2024; Yu et al., 2024a)</td><td>mistral</td><td>7B</td><td>2024-02-05</td><td>1209</td><td>-586.37</td><td>57.03</td></tr>
  <tr><td>106</td><td>BlueNipples/TimeCrystal-12-13B</td><td>llama-2</td><td>13B</td><td>2023-11-11</td><td>1282</td><td>-567.72</td><td>59.26</td></tr>
  <tr><td>107</td><td>bofenghuang/vigogne-2-13b-instruct</td><td>llama-2</td><td>13B</td><td>2023-07-26</td><td>1203</td><td>-534.88</td><td>55.24</td></tr>
  <tr><td>108</td><td>bofenghuang/vigogne-7b-chat</td><td>llama-2</td><td>7B</td><td>2023-07-20</td><td>1170</td><td>-555.45</td><td>45.45</td></tr>
  <tr><td>109</td><td>bofenghuang/vigogne-7b-instruct</td><td>llama-2</td><td>7B</td><td>2023-07-20</td><td>1275</td><td>-555.11</td><td>52.02</td></tr>
  <tr><td>110</td><td>bofenghuang/vigogne-7b-instruct</td><td>llama-1</td><td>7B</td><td>2023-03-22</td><td>1167</td><td>-570.41</td><td>47.76</td></tr>
  <tr><td>111</td><td>bofenghuang/vigostral-7b-chat</td><td>mistral</td><td>7B</td><td>2023-09-29</td><td>4118</td><td>-535.15</td><td>59.18</td></tr>
  <tr><td>112</td><td>BramVanroy/GETje-7b-ultra (Vanroy, 2024)</td><td>mistral</td><td>7B</td><td>2024-01-27</td><td>1597</td><td>-674.95</td><td>52.61</td></tr>
  <tr><td>113</td><td>Brouz/Slepenco</td><td>llama</td><td>1B</td><td>2023-09-08</td><td>1201</td><td>-545.39</td><td>56.59</td></tr>
  <tr><td>114</td><td>budcosystem/boom-1b</td><td>llama</td><td>1B</td><td>2023-10-03</td><td>1267</td><td>-846.13</td><td>48.14</td></tr>
  <tr><td>115</td><td>CalderaAI/13B-BlueMethod</td><td>llama-1</td><td>13B</td><td>2023-07-20</td><td>1173</td><td>-590.13</td><td>52.32</td></tr>
  <tr><td>116</td><td>CalderaAI/13B-LeGerdenain-L2</td><td>llama-1</td><td>13B</td><td>2023-08-03</td><td>1200</td><td>-551.69</td><td>55.13</td></tr>
  <tr><td>117</td><td>CalderaAI/13B-Outputless</td><td>llama</td><td>13B</td><td>2023-07-20</td><td>1197</td><td>-1133.56</td><td>49.54</td></tr>
  <tr><td>118</td><td>CalderaAI/13B-Thorns-32</td><td>llama</td><td>13B</td><td>2023-09-06</td><td>1207</td><td>-580.08</td><td>54.72</td></tr>
  <tr><td>119</td><td>Cartinoc5930/Llama2_init_Mistral</td><td>llama-2</td><td>7B</td><td>2024-01-16</td><td>1174</td><td>-528.82</td><td>60.98</td></tr>
  <tr><td>120</td><td>castorini/rank_vicuna_7b_v1_fp16 (Touvron et al., 2023b; Pradeep et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-09-27</td><td>1165</td><td>-720.69</td><td>44.36</td></tr>
  <tr><td>121</td><td>cedar-ai/FinanceConnect-13B (CedarAI, 2023)</td><td>llama</td><td>13B</td><td>2023-11-28</td><td>1270</td><td>-661.62</td><td>49.34</td></tr>
  <tr><td>122</td><td>Changelly/K2SS-10B-11b-v1.0</td><td>llama-2</td><td>10B</td><td>2024-03-03</td><td>2290</td><td>-811.23</td><td>36.67</td></tr>
  <tr><td>123</td><td>Changelly/K2S3_test_24001</td><td>llama-2</td><td>13B</td><td>2024-02-14</td><td>2346</td><td>-561.61</td><td>56.67</td></tr>
  <tr><td>124</td><td>chargoddard/storytime-13b</td><td>llama-2</td><td>13B</td><td>2023-09-22</td><td>1166</td><td>-599.45</td><td>56.64</td></tr>
  <tr><td>125</td><td>chickencasser/llama2-platypus-llama2-chat-13B-hf</td><td>llama-2</td><td>13B</td><td>2023-09-28</td><td>1216</td><td>-536.92</td><td>54.11</td></tr>
  <tr><td>126</td><td>CHIH-HUNG/llama-2-13b-dolphin_20w</td><td>llama-2</td><td>13B</td><td>2023-08-29</td><td>1187</td><td>-533.73</td><td>55.06</td></tr>
  <tr><td>127</td><td>CHIH-HUNG/llama-2-13b-dolphin_5w</td><td>llama-2</td><td>13B</td><td>2023-08-25</td><td>1175</td><td>-533.03</td><td>55.53</td></tr>
  <tr><td>128</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_TEST_2.2w</td><td>llama-2</td><td>13B</td><td>2023-09-04</td><td>1174</td><td>-530.90</td><td>53.20</td></tr>
  <tr><td>129</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_3.3w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-19</td><td>1166</td><td>-533.48</td><td>53.32</td></tr>
  <tr><td>130</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_3.3w-r44-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-19</td><td>1166</td><td>-533.53</td><td>53.35</td></tr>
  <tr><td>131</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_3.3w-r44-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-09-19</td><td>1170</td><td>-536.50</td><td>53.62</td></tr>
  <tr><td>132</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_3.8w-r8-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-20</td><td>1165</td><td>-533.51</td><td>53.71</td></tr>
  <tr><td>133</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_3.8w-r8-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-09-20</td><td>1170</td><td>-535.00</td><td>53.99</td></tr>
  <tr><td>134</td><td>CHIH-HUNG/llama-2-13b-FINETUNE3_add1015k_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-24</td><td>1165</td><td>-536.24</td><td>53.43</td></tr>
  <tr><td>135</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-10-08</td><td>1161</td><td>-531.46</td><td>54.88</td></tr>
  <tr><td>136</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_compare15k_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-10-08</td><td>1169</td><td>-532.00</td><td>53.94</td></tr>
  <tr><td>137</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-22</td><td>1165</td><td>-533.56</td><td>53.52</td></tr>
  <tr><td>138</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r16-gate_up_down-test1</td><td>llama-2</td><td>13B</td><td>2023-10-07</td><td>1165</td><td>-531.27</td><td>53.66</td></tr>
  <tr><td>139</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r16-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-09-22</td><td>1164</td><td>-534.36</td><td>53.68</td></tr>
  <tr><td>140</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r44-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-21</td><td>1174</td><td>-537.05</td><td>53.48</td></tr>
  <tr><td>141</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r44-q_k_v_o_gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-21</td><td>1163</td><td>-540.32</td><td>53.23</td></tr>
  <tr><td>142</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-21</td><td>1173</td><td>-533.65</td><td>53.58</td></tr>
  <tr><td>143</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_3.8w-r8-q_k_v_o_gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-21</td><td>1171</td><td>-533.89</td><td>54.06</td></tr>
  <tr><td>144</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-09-25</td><td>1165</td><td>-537.12</td><td>52.88</td></tr>
  <tr><td>145</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-10-04</td><td>1161</td><td>-534.88</td><td>54.34</td></tr>
  <tr><td>146</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r44-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-10-01</td><td>1172</td><td>-536.02</td><td>53.32</td></tr>
  <tr><td>147</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r44-q_k_v_o_gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-10-04</td><td>1163</td><td>-541.34</td><td>53.88</td></tr>
  <tr><td>148</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-10-04</td><td>1165</td><td>-531.92</td><td>53.62</td></tr>
  <tr><td>149</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r16-q_k_v_o</td><td>llama-2</td><td>13B</td><td>2023-10-02</td><td>1167</td><td>-534.88</td><td>54.64</td></tr>
  <tr><td>150</td><td>CHIH-HUNG/llama-2-13b-FINETUNE4_4.5w-r44-q_k_v_o_gate_up_down</td><td>llama-2</td><td>13B</td><td>2023-10-04</td><td>1164</td><td>-540.68</td><td>53.69</td></tr>
  <tr><td>151</td><td>CHIH-HUNG/llama-2-13b-OpenOrca_5v2</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>1178</td><td>-533.15</td><td>55.80</td></tr>
  <tr><td>152</td><td>CHIH-HUNG/llama-2-13b-Open_Platypus_and_ccp_2.6w-3_epoch</td><td>llama-2</td><td>13B</td><td>2023-09-05</td><td>1165</td><td>-533.19</td><td>53.80</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>154</td><td>chinchilla/Ti-7b-200k-dpo</td><td>llama</td><td>6B</td><td>2023-12-01</td><td>1187</td><td>-573.79</td><td>51.93</td></tr>
  <tr><td>155</td><td>circulars/Llama-2-13b-orca-v1</td><td>llama-2</td><td>13B</td><td>2023-08-01</td><td>1228</td><td>-540.82</td><td>57.05</td></tr>
  <tr><td>156</td><td>circulars/Llama-2-7b-orca-v1</td><td>llama-2</td><td>7B</td><td>2023-08-01</td><td>1235</td><td>-565.19</td><td>53.56</td></tr>
  <tr><td>157</td><td>cbrain/Llama-2-7b-ft-instruct-es</td><td>llama-2</td><td>7B</td><td>2023-08-09</td><td>2062</td><td>-559.81</td><td>49.63</td></tr>
  <tr><td>158</td><td>CobraMamba/ramba-gpt-3b-v4</td><td>llama</td><td>3B</td><td>2023-09-05</td><td>1183</td><td>-605.83</td><td>41.24</td></tr>
  <tr><td>159</td><td>codellama/CodeLlama-13b-hf</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>8432</td><td>-582.07</td><td>43.35</td></tr>
  <tr><td>160</td><td>codellama/CodeLlama-13b-Instruct-hf</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>25248</td><td>-579.86</td><td>45.82</td></tr>
  <tr><td>161</td><td>codellama/CodeLlama-13b-Python-hf</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>25248</td><td>-587.68</td><td>37.00</td></tr>
  <tr><td>162</td><td>codellama/CodeLlama-13b-ft</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>66040</td><td>-590.81</td><td>39.81</td></tr>
  <tr><td>163</td><td>codellama/CodeLlama-7b-ft</td><td>llama-2</td><td>7B</td><td>2023-08-24</td><td>133523</td><td>-598.06</td><td>38.05</td></tr>
  <tr><td>164</td><td>codellama/CodeLlama-7b-Python-hf</td><td>llama-2</td><td>7B</td><td>2023-08-24</td><td>5138</td><td>-605.60</td><td>36.42</td></tr>
  <tr><td>165</td><td>cognitivecomputations/dolphin-2.2.1-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-10-30</td><td>7292</td><td>-546.62</td><td>65.01</td></tr>
  <tr><td>166</td><td>cognitivecomputations/dolphin-2.6-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-12-27</td><td>1418</td><td>-559.17</td><td>64.92</td></tr>
  <tr><td>167</td><td>cognitivecomputations/dolphin-2.6-mistral-7b-dpo</td><td>mistral</td><td>7B</td><td>2023-12-31</td><td>1236</td><td>-565.34</td><td>67.20</td></tr>
  <tr><td>168</td><td>cognitivecomputations/dolphin-2.6-mistral-7b-dpo-laser</td><td>(Sharma et al., 2023)</td><td>mistral</td><td>7B</td><td>2024-01-01</td><td>2070</td><td>-558.72</td><td>67.28</td></tr>
  <tr><td>169</td><td>cognitivecomputations/dolphin-2.9-llama-3-8b</td><td>llama-3</td><td>8B</td><td>2024-04-20</td><td>130977</td><td>-640.35</td><td>69.52</td></tr>
  <tr><td>170</td><td>cognitivecomputations/dolphin-2.9-llama-3-8b</td><td>llama-3</td><td>8B</td><td>2024-04-20</td><td>130977</td><td>-640.35</td><td>69.52</td></tr>
  <tr><td>171</td><td>cognitivecomputations/dolphin-2.9-1.1-y-5b</td><td>llama</td><td>8B</td><td>2024-05-18</td><td>4880</td><td>-508.79</td><td>68.92</td></tr>
  <tr><td>172</td><td>cognitivecomputations/Llama-3-8B-Instruct-abilitated-v2</td><td>llama-3</td><td>8B</td><td>2024-05-09</td><td>8728</td><td>-581.93</td><td>66.00</td></tr>
  <tr><td>173</td><td>cognitivecomputations/openchat-3.5-010-laser</td><td>mistral</td><td>7B</td><td>2024-01-27</td><td>6155</td><td>-554.46</td><td>69.46</td></tr>
  <tr><td>174</td><td>cognitivecomputations/TinyDolphin-2.8-1.1b</td><td>llama</td><td>1B</td><td>2024-01-21</td><td>1678</td><td>-683.06</td><td>36.34</td></tr>
  <tr><td>175</td><td>cognitivecomputations/WestLake-7B-v2-laser</td><td>mistral</td><td>7B</td><td>2024-01-26</td><td>3980</td><td>-591.19</td><td>74.78</td></tr>
  <tr><td>176</td><td>ContextualAI/archangel-stft-kto_1llama-3</td><td>(Ethayarajh et al., 2023)</td><td>llama</td><td>13B</td><td>2023-12-03</td><td>1214</td><td>-542.49</td><td>52.87</td></tr>
  <tr><td>177</td><td>cookinai/BrunnHermes</td><td>mistral</td><td>7B</td><td>2023-12-11</td><td>1193</td><td>-551.92</td><td>52.42</td></tr>
  <tr><td>178</td><td>cookinai/calm-14</td><td>mistral</td><td>7B</td><td>2023-12-11</td><td>1189</td><td>-552.22</td><td>72.68</td></tr>
  <tr><td>179</td><td>CorinnaMaj-4B-Dolly</td><td>gpt</td><td>6B</td><td>2023-03-28</td><td>1173</td><td>-502.52</td><td>59.30</td></tr>
  <tr><td>180</td><td>crumb/apricot-wildflower-20</td><td>mistral</td><td>7B</td><td>2023-12-19</td><td>1179</td><td>-531.99</td><td>59.74</td></tr>
  <tr><td>181</td><td>cyberagent/open-calm-7b</td><td>(Andonian et al., 2021)</td><td>gpt_neox</td><td>7B</td><td>2023-05-15</td><td>24168</td><td>-1002.50</td><td>58.21</td></tr>
  <tr><td>182</td><td>cypienai/cymist-2-v01-SFT</td><td>(Lacoste et al., 2019)</td><td>mistral</td><td>7B</td><td>2024-05-12</td><td>2754</td><td>-628.53</td><td>51.71</td></tr>
  <tr><td>183</td><td>cypienai/cymist-2-v02-SFT</td><td>(Lacoste et al., 2019)</td><td>mistral</td><td>7B</td><td>2024-05-22</td><td>2717</td><td>-537.73</td><td>62.57</td></tr>
  <tr><td>184</td><td>Dampish/Dante-2.8B</td><td>gpt_neox</td><td>4B</td><td>2023-05-09</td><td>1237</td><td>-671.96</td><td>–</td></tr>
  <tr><td>185</td><td>Dampish/Stitcher-4B-V0</td><td>(Black et al., 2022)</td><td>gpt_neox</td><td>4B</td><td>2023-06-03</td><td>1245</td><td>-866.96</td><td>36.15</td></tr>
  <tr><td>186</td><td>Danielbrldz/Barcenas-3B-ORPO</td><td>llama-3</td><td>8B</td><td>2024-04-29</td><td>12814</td><td>-557.50</td><td>72.50</td></tr>
  <tr><td>187</td><td>Danielbrldz/Barcenas-13b</td><td>llama-2</td><td>13B</td><td>2023-09-09</td><td>1178</td><td>-558.81</td><td>55.83</td></tr>
  <tr><td>188</td><td>Danielbrldz/Barcenas-3b</td><td>llama-2</td><td>3B</td><td>2023-11-15</td><td>1167</td><td>-554.05</td><td>41.74</td></tr>
  <tr><td>189</td><td>Danielbrldz/Barcenas-7b</td><td>llama</td><td>7B</td><td>2023-08-25</td><td>1175</td><td>-603.96</td><td>50.87</td></tr>
  <tr><td>190</td><td>Danielbrldz/CodeBarcenas-7b</td><td>llama-2</td><td>7B</td><td>2023-09-03</td><td>1174</td><td>-656.48</td><td>40.09</td></tr>
  <tr><td>191</td><td>danielpark/gorani-100k-Llama-2-13b-instruct</td><td>llama-2</td><td>13B</td><td>2023-10-04</td><td>1176</td><td>-1410.79</td><td>29.69</td></tr>
  <tr><td>192</td><td>databricks/dolly-v2-7b-1h</td><td>(Conover et al., 2023)</td><td>gpt_neox</td><td>12B</td><td>2023-04-13</td><td>3087</td><td>-565.97</td><td>39.46</td></tr>
  <tr><td>193</td><td>databricks/dolly-v2-7b-1h</td><td>(Conover et al., 2023)</td><td>gpt_neox</td><td>12B</td><td>2023-04-13</td><td>2143</td><td>-556.97</td><td>39.24</td></tr>
  <tr><td>194</td><td>DBCLAB/Llama-3-instruction-constructionsafety-layerturning (AI@Meta, 2024; Lee, 2024a; Lee and Ahn, 2024)</td><td>llama-3</td><td>8B</td><td>2024-05-22</td><td>2712</td><td>-658.58</td><td>56.32</td></tr>
  <tr><td>195</td><td>Deathsquad10/TinyLlama-Remix</td><td>llama</td><td>1B</td><td>2023-11-26</td><td>1213</td><td>-812.12</td><td>34.00</td></tr>
  <tr><td>196</td><td>Deathsquad10/TinyLlama-remix</td><td>llama</td><td>1B</td><td>2024-01-06</td><td>1191</td><td>-622.67</td><td>37.09</td></tr>
  <tr><td>197</td><td>Deathsquad10/TinyLlama-remix-V2</td><td>llama</td><td>1B</td><td>2024-01-11</td><td>1171</td><td>-675.63</td><td>39.91</td></tr>
  <tr><td>198</td><td>Deci/DeciLM-7B-instruct (AI4Research Team, 2023)</td><td>deci</td><td>7B</td><td>2023-12-10</td><td>9769</td><td>-570.53</td><td>63.19</td></tr>
  <tr><td>199</td><td>DeepMount00/Llama-3-8b-Ita</td><td>llama-3</td><td>8B</td><td>2024-05-01</td><td>179401</td><td>-578.66</td><td>73.65</td></tr>
  <tr><td>200</td><td>DeepMount00/Mistral-Ita-7b</td><td>mistral</td><td>7B</td><td>2023-11-08</td><td>2927</td><td>-579.30</td><td>58.26</td></tr>
  <tr><td>201</td><td>deepseek-a/DeepSeek-Coder-V2-Lite-Base (Dai et al., 2024)</td><td>deepseek</td><td>16B</td><td>2024-06-14</td><td>13642</td><td>-527.73</td><td>–</td></tr>
  <tr><td>202</td><td>deepseek-a/DeepSeek-Coder-V2-Lite-Instruct (Dai et al., 2024)</td><td>deepseek</td><td>16B</td><td>2024-06-14</td><td>142758</td><td>-569.70</td><td>–</td></tr>
  <tr><td>203</td><td>deepseek-a/deepseek-coder-1.3b-base (Guo et al., 2024)</td><td>deepseek</td><td>1B</td><td>2023-10-28</td><td>4890</td><td>-718.82</td><td>–</td></tr>
  <tr><td>204</td><td>deepseek-a/deepseek-coder-1.3b-instruct (Guo et al., 2024)</td><td>deepseek</td><td>1B</td><td>2023-10-28</td><td>2894</td><td>-718.82</td><td>32.40</td></tr>
  <tr><td>205</td><td>deepseek-a/deepseek-coder-6.7b-Instruct (Guo et al., 2024)</td><td>deepseek</td><td>6B</td><td>2023-10-23</td><td>3848</td><td>-487.51</td><td>40.87</td></tr>
  <tr><td>206</td><td>deepseek-a/deepseek-coder-6.7b-Instruct (Guo et al., 2024)</td><td>deepseek</td><td>6B</td><td>2023-10-29</td><td>2882</td><td>-866.56</td><td>43.57</td></tr>
  <tr><td>207</td><td>deepseek-a/deepseek-coder-7b-base-v1.5 (Guo et al., 2024)</td><td>deepseek</td><td>7B</td><td>2024-01-25</td><td>895</td><td>-570.98</td><td>–</td></tr>
  <tr><td>208</td><td>deepseek-a/deepseek-coder-7b-instruct-v1.5 (Guo et al., 2024)</td><td>deepseek</td><td>7B</td><td>2024-01-25</td><td>28928</td><td>-604.47</td><td>50.89</td></tr>
  <tr><td>209</td><td>deepseek-a/deepseek-llm-7b-base (DeepSeek-Al et al., 2024a)</td><td>deepseek</td><td>7B</td><td>2023-11-29</td><td>19180</td><td>-552.39</td><td>–</td></tr>
  <tr><td>210</td><td>deepseek-a/deepseek-llm-7b-chat (DeepSeek-Al et al., 2024a)</td><td>deepseek</td><td>7B</td><td>2023-11-29</td><td>34271</td><td>-594.23</td><td>59.38</td></tr>
  <tr><td>211</td><td>deepseek-a/deepseek-math-7b-base (Shao et al., 2024)</td><td>deepseek</td><td>7B</td><td>2024-02-05</td><td>37614</td><td>-566.93</td><td>57.61</td></tr>
  <tr><td>212</td><td>deepseek-a/deepseek-math-7b-instruct (Shao et al., 2024)</td><td>deepseek</td><td>7B</td><td>2024-02-05</td><td>8352</td><td>-591.55</td><td>51.48</td></tr>
  <tr><td>213</td><td>deepseek-a/deepseek-math-7b-instruct (Shao et al., 2024)</td><td>deepseek</td><td>7B</td><td>2024-02-05</td><td>17616</td><td>-607.25</td><td>49.54</td></tr>
  <tr><td>214</td><td>deepseek-a/deepseek-moe-16b-base (Da et al., 2024)</td><td>deepseek</td><td>16B</td><td>2024-01-08</td><td>13924</td><td>-544.49</td><td>–</td></tr>
  <tr><td>215</td><td>deepseek-a/deepseek-moe-16b-chat (Da et al., 2024)</td><td>deepseek</td><td>16B</td><td>2024-01-09</td><td>8852</td><td>-578.23</td><td>–</td></tr>
  <tr><td>216</td><td>deepseek-a/DeepSeek-Prover-V1 (Xin et al., 2024a)</td><td>deepseek</td><td>7B</td><td>2024-08-16</td><td>114</td><td>-791.21</td><td>–</td></tr>
  <tr><td>217</td><td>deepseek-a/DeepSeek-Prover-V1.5-Base (Xin et al., 2024b)</td><td>deepseek</td><td>7B</td><td>2024-08-15</td><td>230</td><td>-554.13</td><td>–</td></tr>
  <tr><td>218</td><td>deepseek-a/DeepSeek-Prover-V1.5-SFT (Xin et al., 2024b)</td><td>deepseek</td><td>7B</td><td>2024-08-15</td><td>12223</td><td>-672.91</td><td>–</td></tr>
  <tr><td>219</td><td>deepseek-a/DeepSeek-Prover-V1.5-SFT (Xin et al., 2024b)</td><td>deepseek</td><td>7B</td><td>2024-08-15</td><td>6459</td><td>-670.56</td><td>–</td></tr>
  <tr><td>220</td><td>deepseek-a/DeepSeek-R1-Distill-Qwen-14B (DeepSeek-Al et al., 2025)</td><td>deepseek</td><td>2B</td><td>2025-01-20</td><td>13948</td><td>-605.82</td><td>–</td></tr>
  <tr><td>221</td><td>deepseek-a/DeepSeek-R1-Distill-Qwen-15B (DeepSeek-Al et al., 2025)</td><td>deepseek</td><td>2B</td><td>2025-01-20</td><td>20094</td><td>-854.97</td><td>–</td></tr>
  <tr><td>222</td><td>deepseek-a/DeepSeek-R1-Distill-Qwen-7B (DeepSeek-Al et al., 2025)</td><td>deepseek</td><td>7B</td><td>2025-01-20</td><td>20118</td><td>-758.13</td><td>–</td></tr>
  <tr><td>223</td><td>deepseek-a/DeepSeek-V2-Lite (DeepSeek-Al et al., 2024b)</td><td>deepseek</td><td>16B</td><td>2024-05-15</td><td>25474</td><td>-533.54</td><td>–</td></tr>
  <tr><td>224</td><td>deepseek-a/DeepSeek-V2-Lite-Chat (DeepSeek-Al et al., 2024b)</td><td>deepseek</td><td>16B</td><td>2024-05-15</td><td>18056</td><td>-588.92</td><td>–</td></tr>
  <tr><td>225</td><td>deepseek-a/ESFT-vanilla-lite (Wang et al., 2024c)</td><td>deepseek</td><td>16B</td><td>2024-07-04</td><td>270</td><td>-550.46</td><td>–</td></tr>
  <tr><td>226</td><td>Delos/Mistral-Python-7b-Orpo-1</td><td>llama-2</td><td>7B</td><td>2023-10-09</td><td>1185</td><td>-561.93</td><td>51.02</td></tr>
  <tr><td>227</td><td>dhmelzter/Llama-2-13b-ft-hs_el5_1024_r_64_alpha_16_merged</td><td>llama-2</td><td>13B</td><td>2023-09-14</td><td>1188</td><td>-542.35</td><td>54.16</td></tr>
  <tr><td>228</td><td>dhmelzter/Llama-2-13b-ft-hs_wiki_1024_r_64_alpha_16_merged</td><td>llama-2</td><td>13B</td><td>2023-09-14</td><td>1198</td><td>-536.93</td><td>52.94</td></tr>
  <tr><td>229</td><td>dhmelzter/Llama-7b-SFT_el5_wiki65k_1024_r_64_alpha_16_merged</td><td>llama</td><td>6B</td><td>2023-08-25</td><td>1257</td><td>-568.38</td><td>50.00</td></tr>
  <tr><td>230</td><td>diffnearnhard/Psychfighter-Normoid-ties-13B</td><td>llama</td><td>13B</td><td>2023-12-28</td><td>1215</td><td>-569.13</td><td>59.47</td></tr>
  <tr><td>231</td><td>digitous/Alpaca13b</td><td>llama-1</td><td>1B</td><td>2023-04-13</td><td>1182</td><td>-547.35</td><td>52.39</td></tr>
  <tr><td>232</td><td>digitous/13B-Chimera</td><td>llama-1</td><td>1B</td><td>2023-05-23</td><td>1359</td><td>-562.36</td><td>54.92</td></tr>
  <tr><td>233</td><td>Doctor-Shotgun/CalliopeDS-L2-13B (Yadav et al., 2023a)</td><td>llama-2</td><td>13B</td><td>2023-09-16</td><td>1428</td><td>-568.72</td><td>56.34</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>239</td><td>Doctor/Shotgun/CalliopeDS-v2-1.2-13B</td><td>llama-2</td><td>13B</td><td>2023-09-28</td><td>1250</td><td>-572.65</td><td>57.12</td></tr>
  <tr><td>240</td><td>Deepo/Nope/You_can_cry_Snowman-13B</td><td>llama</td><td>13B</td><td>2023-12-27</td><td>1203</td><td>-577.68</td><td>69.46</td></tr>
  <tr><td>241</td><td>dotvgnsht/perry-7b</td><td>llama</td><td>7B</td><td>2023-09-28</td><td>1171</td><td>-618.90</td><td>49.55</td></tr>
  <tr><td>242</td><td>dreamgen/WizardLM-2.7B (Xu et al., 2023a; Luo et al., 2023, 2025)</td><td>mistral</td><td>7B</td><td>2024-04-16</td><td>2533</td><td>-641.93</td><td>63.75</td></tr>
  <tr><td>243</td><td>dvruete/oastt-pythia-12b-flash-attn-5000-steps</td><td>gpt_neox</td><td>12B</td><td>2023-03-12</td><td>1175</td><td>-552.37</td><td>40.73</td></tr>
  <tr><td>244</td><td>dvruete/oastt-pythia-12b-pretrained-sft</td><td>gpt_neox</td><td>12B</td><td>2023-04-03</td><td>1170</td><td>-552.58</td><td>41.48</td></tr>
  <tr><td>245</td><td>dvruete/oastt-pythia-12b-reference</td><td>gpt_neox</td><td>12B</td><td>2023-04-03</td><td>1174</td><td>-557.88</td><td>40.33</td></tr>
  <tr><td>246</td><td>eachade/vicuna-13b-1.1</td><td>llama-1</td><td>13B</td><td>2023-04-13</td><td>1336</td><td>-646.61</td><td>53.29</td></tr>
  <tr><td>247</td><td>eachade/vicuna-7b-1.1</td><td>llama-1</td><td>7B</td><td>2023-04-13</td><td>1388</td><td>-613.88</td><td>50.37</td></tr>
  <tr><td>248</td><td>EleutherAI/Pecan-7b-9b</td><td>llama</td><td>7B</td><td>2023-05-10</td><td>1011</td><td>-581.92</td><td>48.30</td></tr>
  <tr><td>249</td><td>EleutherAI/gpt-j-6b (Gao et al., 2020; Wang, 2021; Wang and Komatsuzaki, 2021; Su et al., 2023a)</td><td>gptj</td><td>6B</td><td>2022-03-02</td><td>241455</td><td>-479.14</td><td>40.10</td></tr>
  <tr><td>250</td><td>EleutherAI/gpt-neo-1.3B (Gao et al., 2020; Black et al., 2021)</td><td>gpt_neo</td><td>1B</td><td>2022-03-02</td><td>243332</td><td>-545.00</td><td>33.58</td></tr>
  <tr><td>251</td><td>EleutherAI/gpt-neo-2.7B (Gao et al., 2020; Black et al., 2021)</td><td>gpt_neo</td><td>2B</td><td>2022-03-02</td><td>205503</td><td>-520.30</td><td>36.20</td></tr>
  <tr><td>252</td><td>EleutherAI/lemma_7b (Azerbayev et al., 2024)</td><td>llama-2</td><td>7B</td><td>2023-09-12</td><td>4671</td><td>-554.43</td><td>48.75</td></tr>
  <tr><td>253</td><td>EleutherAI/polyglot-ko-12.8b (Kim et al., 2022; Su et al., 2023a; Ko et al., 2023)</td><td>gpt_neox</td><td>13B</td><td>2022-10-14</td><td>2941</td><td>-667.65</td><td>53.33</td></tr>
  <tr><td>254</td><td>EleutherAI/pythia-1b-deduped (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>1B</td><td>2023-02-14</td><td>17983</td><td>-551.07</td><td>32.78</td></tr>
  <tr><td>255</td><td>EleutherAI/pythia-12b (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>12B</td><td>2023-02-28</td><td>9192</td><td>-475.42</td><td>38.82</td></tr>
  <tr><td>256</td><td>EleutherAI/pythia-12b-deduped (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>12B</td><td>2023-02-27</td><td>9737</td><td>-477.81</td><td>39.70</td></tr>
  <tr><td>257</td><td>EleutherAI/pythia-1.4b (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>1B</td><td>2023-02-09</td><td>22152</td><td>-531.68</td><td>34.75</td></tr>
  <tr><td>258</td><td>EleutherAI/pythia-1.4b-deduped (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>1B</td><td>2023-02-09</td><td>12730</td><td>-534.52</td><td>35.00</td></tr>
  <tr><td>259</td><td>EleutherAI/pythia-2.8b-deduped (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>2B</td><td>2023-02-10</td><td>11649</td><td>-507.42</td><td>36.72</td></tr>
  <tr><td>260</td><td>EleutherAI/pythia-6.9b-deduped (Gao et al., 2020; Biderman et al., 2022, 2023)</td><td>gpt_neox</td><td>6B</td><td>2023-02-25</td><td>9824</td><td>-490.70</td><td>39.30</td></tr>
  <tr><td>261</td><td>elinas/chronos-13b-v2</td><td>llama-2</td><td>13B</td><td>2023-08-02</td><td>1944</td><td>-595.38</td><td>55.25</td></tr>
  <tr><td>262</td><td>elliotthwang/elliot_Llama-2-7b-hf</td><td>llama-2</td><td>6B</td><td>2023-10-09</td><td>1181</td><td>-553.66</td><td>50.20</td></tr>
  <tr><td>263</td><td>elyza/ELYZA-japanese-Llama-2-13b (Sasaki et al., 2023b; Touvron et al., 2023b)</td><td>llama-2</td><td>13B</td><td>2023-12-25</td><td>1200</td><td>-587.25</td><td>56.14</td></tr>
  <tr><td>264</td><td>elyza/ELYZA-japanese-Llama-2-13b-instruct (Sasaki et al., 2023b; Touvron et al., 2023b)</td><td>llama-2</td><td>13B</td><td>2023-12-25</td><td>1710</td><td>-594.98</td><td>54.72</td></tr>
  <tr><td>265</td><td>elyza/ELYZA-japanese-Llama-2-7b (Touvron et al., 2023b; Sasaki et al., 2023a)</td><td>llama-2</td><td>7B</td><td>2023-08-28</td><td>2356</td><td>-630.66</td><td>48.70</td></tr>
  <tr><td>266</td><td>elyza/ELYZA-japanese-Llama-2-7b-fast (Touvron et al., 2023b; Sasaki et al., 2023a)</td><td>llama-2</td><td>7B</td><td>2023-08-28</td><td>1766</td><td>-640.71</td><td>47.67</td></tr>
  <tr><td>267</td><td>elyza/ELYZA-japanese-Llama-2-7b-fast-instruct (Touvron et al., 2023b; Sasaki et al., 2023a)</td><td>llama-2</td><td>7B</td><td>2023-08-28</td><td>2387</td><td>-629.72</td><td>49.15</td></tr>
  <tr><td>268</td><td>elyza/ELYZA-japanese-Llama-2-7b-instruct (Touvron et al., 2023b; Sasaki et al., 2023a)</td><td>llama-2</td><td>7B</td><td>2023-08-28</td><td>6296</td><td>-625.45</td><td>49.78</td></tr>
  <tr><td>269</td><td>Enno-AI/enmodata-rav-pajamathur-13b-peft</td><td>llama</td><td>13B</td><td>2023-09-29</td><td>1215</td><td>-613.07</td><td>55.40</td></tr>
  <tr><td>270</td><td>erichzz/falcon-7w-1b-chat</td><td>falcon</td><td>1B</td><td>2023-12-05</td><td>1319</td><td>-722.18</td><td>37.38</td></tr>
  <tr><td>271</td><td>erichzz/falcon-7w-1b-instruct-openorca</td><td>falcon</td><td>1B</td><td>2023-11-24</td><td>1585</td><td>-752.98</td><td>37.63</td></tr>
  <tr><td>272</td><td>euclase/falcon_1b_stage1</td><td>falcon</td><td>1B</td><td>2023-09-15</td><td>2119</td><td>-734.28</td><td>37.25</td></tr>
  <tr><td>273</td><td>euclase/falcon_1b_stage2</td><td>falcon</td><td>1B</td><td>2023-09-17</td><td>4016</td><td>-717.40</td><td>37.59</td></tr>
  <tr><td>274</td><td>euclase/Ferret_7B</td><td>mistral</td><td>7B</td><td>2023-10-28</td><td>1172</td><td>-634.24</td><td>53.87</td></tr>
  <tr><td>275</td><td>Eurdem/Define_llama3_2x8B</td><td>mistral</td><td>13B</td><td>2024-05-10</td><td>9076</td><td>-571.19</td><td>71.73</td></tr>
  <tr><td>276</td><td>Expert68/llama2_13b_instruct_vesion2</td><td>llama-2</td><td>13B</td><td>2023-10-14</td><td>1184</td><td>-561.35</td><td>55.41</td></tr>
  <tr><td>277</td><td>facebook/opt-1b-1.3b (Brown et al., 2023)</td><td>opt</td><td>1B</td><td>2023-01-26</td><td>8604</td><td>-712.00</td><td>35.21</td></tr>
  <tr><td>278</td><td>facebook/opt-1b-1.3b (Brown et al., 2020; Zhang et al., 2022b)</td><td>opt</td><td>1B</td><td>2022-05-11</td><td>1911</td><td>-652.18</td><td>36.06</td></tr>
  <tr><td>279</td><td>facebook/opt-2.7b (Brown et al., 2020; Zhang et al., 2022b)</td><td>opt</td><td>2B</td><td>2022-05-11</td><td>139758</td><td>-700.64</td><td>34.60</td></tr>
  <tr><td>280</td><td>facebook/opt-6.7b (Brown et al., 2020; Zhang et al., 2022b)</td><td>opt</td><td>6B</td><td>2022-05-11</td><td>61450</td><td>-673.03</td><td>36.74</td></tr>
  <tr><td>281</td><td>facebook/xglm-1.7B (Lin et al., 2022b)</td><td>xglm</td><td>1B</td><td>2023-02-02</td><td>2388</td><td>-723.91</td><td>31.42</td></tr>
  <tr><td>282</td><td>facebook/xglm-4.5B (Lin et al., 2022b)</td><td>xglm</td><td>5B</td><td>2023-02-02</td><td>1436</td><td>-669.92</td><td>34.31</td></tr>
  <tr><td>283</td><td>facebook/xglm-7.5B (Lin et al., 2022b)</td><td>xglm</td><td>7B</td><td>2023-02-02</td><td>4654</td><td>-663.38</td><td>36.38</td></tr>
  <tr><td>284</td><td>fairmind/Meta-Llama-8B-Instruct-ablated-v3</td><td>llama-3</td><td>8B</td><td>2024-04-09</td><td>9975</td><td>-572.33</td><td>67.27</td></tr>
  <tr><td>285</td><td>fairmind/Meta-Llama-8B-Instruct-ablated-v3</td><td>ph3</td><td>8B</td><td>2024-05-22</td><td>5430</td><td>-560.05</td><td>61.02</td></tr>
  <tr><td>286</td><td>FairMind/Llama-3-8B-4bit-UltraChat-Ita</td><td>llama-3</td><td>8B</td><td>2024-05-03</td><td>5014</td><td>-543.07</td><td>61.54</td></tr>
  <tr><td>287</td><td>FairMind/Phi-i-mini-4k-instruct-4bit-Ita</td><td>mistral</td><td>4B</td><td>2024-05-02</td><td>2746</td><td>-634.71</td><td>66.61</td></tr>
  <tr><td>288</td><td>flbigit/LUNA-SOLARKraut-IM-Instruct</td><td>llama</td><td>10B</td><td>2023-12-22</td><td>1180</td><td>-582.98</td><td>73.79</td></tr>
  <tr><td>289</td><td>flbigit/una-cybertron-7b-v2-b16 (Murias, 2023)</td><td>mistral</td><td>7B</td><td>2023-12-02</td><td>1440</td><td>-568.27</td><td>69.67</td></tr>
  <tr><td>290</td><td>flbigit/UNA-POLAR-10.7B-InstructMath-v2</td><td>llama</td><td>10B</td><td>2024-01-02</td><td>1180</td><td>-560.34</td><td>74.07</td></tr>
  <tr><td>291</td><td>feidie/Meatamath-7b</td><td>llama-2</td><td>7B</td><td>2023-11-24</td><td>1198</td><td>-724.03</td><td>55.81</td></tr>
  <tr><td>292</td><td>FelixChao/CodeLLama-3B-Finetune-1v</td><td>llama-2</td><td>13B</td><td>2023-08-12</td><td>1181</td><td>-642.19</td><td>54.19</td></tr>
  <tr><td>293</td><td>FelixChao/llama-2-13b-math1</td><td>llama-2</td><td>13B</td><td>2023-08-15</td><td>1197</td><td>-607.04</td><td>54.19</td></tr>
  <tr><td>294</td><td>FinancialSupport/saiga-7b</td><td>mistral</td><td>7B</td><td>2023-12-28</td><td>3902</td><td>-558.71</td><td>64.51</td></tr>
  <tr><td>295</td><td>fireballon/baihuachun-vicuna-chinese-7b</td><td>llama-1</td><td>7B</td><td>2023-06-18</td><td>1210</td><td>-702.08</td><td>46.06</td></tr>
  <tr><td>296</td><td>FlagAlpha/Llama2-Chinese-7b-Chat</td><td>llama-2</td><td>7B</td><td>2023-07-23</td><td>1383</td><td>-627.19</td><td>51.13</td></tr>
  <tr><td>297</td><td>FlagAlpha/Llama3-Chinese-8B-Instruct</td><td>llama-3</td><td>8B</td><td>2024-04-23</td><td>1978</td><td>-557.17</td><td>63.50</td></tr>
  <tr><td>298</td><td>FredheimSt/Guanaco-3B-Uncooled-v2</td><td>gpt_neox</td><td>2B</td><td>2023-04-11</td><td>2189</td><td>-631.88</td><td>38.98</td></tr>
  <tr><td>299</td><td>FreedheimIntelligence/AceGPT-7B</td><td>llama</td><td>7B</td><td>2023-09-14</td><td>2965</td><td>-565.76</td><td>47.47</td></tr>
  <tr><td>300</td><td>Freeheelin/free-llama-4po-v0.2 (Kim et al., 2024b)</td><td>llama-3</td><td>8B</td><td>2024-05-09</td><td>1296</td><td>-693.44</td><td>43.03</td></tr>
  <tr><td>301</td><td>gagan3012/MetaModelv2</td><td>llama</td><td>10B</td><td>2024-01-03</td><td>1203</td><td>-560.16</td><td>74.24</td></tr>
  <tr><td>302</td><td>gagan3012/MetaModelv3</td><td>llama</td><td>10B</td><td>2024-01-05</td><td>1207</td><td>-561.14</td><td>74.39</td></tr>
  <tr><td>303</td><td>garage-bAInd/Platytype-2-13B (Hu et al., 2022; Touvron et al., 2023b; Lee et al., 2024)</td><td>llama</td><td>13B</td><td>2023-08-05</td><td>3716</td><td>-533.92</td><td>54.89</td></tr>
  <tr><td>304</td><td>garage-bAInd/Platytype-2-7B (Hu et al., 2022; Touvron et al., 2023b; Lee et al., 2024)</td><td>llama</td><td>6B</td><td>2023-08-22</td><td>6198</td><td>-554.08</td><td>49.97</td></tr>
  <tr><td>305</td><td>GeneZC/MiniChat-1.5-3B (Jain et al., 2023; Rafailov et al., 2024; Zhang et al., 2024a)</td><td>llama</td><td>3B</td><td>2023-11-26</td><td>1601</td><td>-599.51</td><td>50.23</td></tr>
  <tr><td>306</td><td>GeneZC/MiniChat-2-3B (Jain et al., 2023; Rafailov et al., 2024; Zhang et al., 2024a)</td><td>llama</td><td>3B</td><td>2023-12-27</td><td>4167</td><td>-607.10</td><td>51.49</td></tr>
  <tr><td>307</td><td>GeneZC/MiniChat-3B (Zhang et al., 2024a)</td><td>llama</td><td>3B</td><td>2023-11-11</td><td>1633</td><td>-585.38</td><td>45.31</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>311</td>
    <td>GeneZC/MiniMA-2-3B (Zhang et al., 2024a)</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-12-27</td>
    <td>1492</td>
    <td>-526.37</td>
    <td>44.75</td>
  </tr>
  <tr>
    <td>312</td>
    <td>GeneZC/MiniMA-3B (Zhang et al., 2024a)</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-11-11</td>
    <td>1494</td>
    <td>-537.57</td>
    <td>41.44</td>
  </tr>
  <tr>
    <td>313</td>
    <td>georgesung/llama2_7b_chat_uncensored</td>
    <td>llama-2</td>
    <td>7B</td>
    <td>2023-07-20</td>
    <td>2602</td>
    <td>-568.34</td>
    <td>49.67</td>
  </tr>
  <tr>
    <td>314</td>
    <td>ghost-x/ghost-7b-alpha</td>
    <td>mistral</td>
    <td>7B</td>
    <td>2024-04-13</td>
    <td>4854</td>
    <td>-662.59</td>
    <td>57.65</td>
  </tr>
  <tr>
    <td>315</td>
    <td>glaiave/glaiave-coder-7b</td>
    <td>llama-2</td>
    <td>7B</td>
    <td>2023-09-17</td>
    <td>1222</td>
    <td>-692.31</td>
    <td>41.56</td>
  </tr>
  <tr>
    <td>316</td>
    <td>google/codegemma-2b (CodeGemma Team et al., 2024)</td>
    <td>gemma</td>
    <td>2B</td>
    <td>2024-03-21</td>
    <td>8798</td>
    <td>-793.92</td>
    <td>32.19</td>
  </tr>
  <tr>
    <td>317</td>
    <td>google/codegemma-7b (CodeGemma Team et al., 2024)</td>
    <td>gemma</td>
    <td>8B</td>
    <td>2024-03-21</td>
    <td>3779</td>
    <td>-556.79</td>
    <td>56.73</td>
  </tr>
  <tr>
    <td>318</td>
    <td>google/codegemma-11.7b (Joshi et al., 2019; Bisk et al., 2019; Zellers et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>8B</td>
    <td>2024-03-21</td>
    <td>11227</td>
    <td>-1040.98</td>
    <td>58.28</td>
  </tr>
  <tr>
    <td>319</td>
    <td>google/gemma-11.7b (Joshi et al., 2019; Bisk et al., 2019; Zellers et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>8B</td>
    <td>2024-03-26</td>
    <td>19835</td>
    <td>-1355.60</td>
    <td>60.09</td>
  </tr>
  <tr>
    <td>320</td>
    <td>google/gemma-2b (Joshi et al., 2017; Mihaylov et al., 2018; Zhao et al., 2018; Rudinger et al., 2018; Zellers et al., 2019; Bisk et al., 2019; Sap et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Cobbe et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Lin et al., 2022a; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>2B</td>
    <td>2024-02-08</td>
    <td>256798</td>
    <td>-581.35</td>
    <td>46.51</td>
  </tr>
  <tr>
    <td>321</td>
    <td>google/gemma-2b-it (Joshi et al., 2017; Mihaylov et al., 2018; Zhao et al., 2018; Rudinger et al., 2018; Zellers et al., 2019; Bisk et al., 2019; Sap et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Cobbe et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Lin et al., 2022a; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>2B</td>
    <td>2024-02-08</td>
    <td>103851</td>
    <td>-886.70</td>
    <td>42.75</td>
  </tr>
  <tr>
    <td>322</td>
    <td>google/gemma-7b (Joshi et al., 2017; Mihaylov et al., 2018; Zhao et al., 2018; Rudinger et al., 2018; Zellers et al., 2019; Bisk et al., 2019; Sap et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Cobbe et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Lin et al., 2022a; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>8B</td>
    <td>2024-02-08</td>
    <td>72047</td>
    <td>-539.12</td>
    <td>63.75</td>
  </tr>
  <tr>
    <td>323</td>
    <td>google/gemma-7b-it (Joshi et al., 2017; Mihaylov et al., 2018; Zhao et al., 2018; Rudinger et al., 2018; Zellers et al., 2019; Bisk et al., 2019; Sap et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Cobbe et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Lin et al., 2022a; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>gemma</td>
    <td>8B</td>
    <td>2024-02-13</td>
    <td>66776</td>
    <td>-1327.43</td>
    <td>53.56</td>
  </tr>
  <tr>
    <td>324</td>
    <td>google/recurrentgemma-2b (Joshi et al., 2017; Mihaylov et al., 2018; Zhao et al., 2018; Rudinger et al., 2018; Zellers et al., 2019; Bisk et al., 2019; Sap et al., 2019; Clark et al., 2019; Collet, 2019; Sakaguchi et al., 2019; Talmor et al., 2019; Gehman et al., 2020; Austin et al., 2021; Hendrycks et al., 2021; Cobbe et al., 2021; Chen et al., 2021; Dhamala et al., 2021; Parrish et al., 2022; Lin et al., 2022a; Hartvigsen et al., 2022; Srivastava et al., 2023; Zhong et al., 2023; Gemini Team et al., 2024)</td>
    <td>recurrent_gemma</td>
    <td>2B</td>
    <td>2024-04-08</td>
    <td>4046</td>
    <td>-1033.09</td>
    <td>40.86</td>
  </tr>
  <tr>
    <td>325</td>
    <td>gradiental/Llama-3-8B-Instruct-Gradient1048k (Ding et al., 2023; Peng et al., 2023b; Pekelis et al., 2024; Alifanov, 2024; Liu et al., 2025)</td>
    <td>llama-3</td>
    <td>8B</td>
    <td>2024-04-29</td>
    <td>6854</td>
    <td>-564.98</td>
    <td>59.84</td>
  </tr>
  <tr>
    <td>326</td>
    <td>gradiental/Llama-3-8B-Instruct-Gradient264k (Ding et al., 2023; Peng et al., 2023b; Pekelis et al., 2024; Alifanov, 2024; Liu et al., 2025)</td>
    <td>llama-3</td>
    <td>8B</td>
    <td>2024-04-25</td>
    <td>12536</td>
    <td>-555.17</td>
    <td>60.26</td>
  </tr>
  <tr>
    <td>327</td>
    <td>GritLM/GridLM-7B (Muenninghoff et al., 2025)</td>
    <td>mistral</td>
    <td>7B</td>
    <td>2024-02-11</td>
    <td>72991</td>
    <td>-564.58</td>
    <td>61.41</td>
  </tr>
  <tr>
    <td>328</td>
    <td>Gryphie/Mythologic-L1-12b</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-08-03</td>
    <td>1198</td>
    <td>-570.07</td>
    <td>56.19</td>
  </tr>
  <tr>
    <td>329</td>
    <td>Gryphie/Mythologic-L12-13b</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-08-10</td>
    <td>1435</td>
    <td>-583.44</td>
    <td>56.09</td>
  </tr>
  <tr>
    <td>330</td>
    <td>Gryphie/Mythologic-L12-13b</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-08-10</td>
    <td>1174</td>
    <td>-564.44</td>
    <td>56.31</td>
  </tr>
  <tr>
    <td>331</td>
    <td>gritml/llama2-7b-instruct-sharded</td>
    <td>llama-2</td>
    <td>6B</td>
    <td>2023-07-21</td>
    <td>3230</td>
    <td>-640.07</td>
    <td>50.58</td>
  </tr>
  <tr>
    <td>332</td>
    <td>gywlm/lm2-13b-chinese-v2</td>
    <td>llama-2</td>
    <td>13B</td>
    <td>2023-08-22</td>
    <td>1163</td>
    <td>-719.40</td>
    <td>49.58</td>
  </tr>
  <tr>
    <td>333</td>
    <td>habano/TinyLlama-1.1B-instruct-full-1r1-5-v0.1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-19</td>
    <td>1178</td>
    <td>-679.96</td>
    <td>35.58</td>
  </tr>
  <tr>
    <td>334</td>
    <td>habano/TinyLlama-1.1B-intermediate-step-715k-1.5T-1r-5-1epochs-aost1-top1-instruct-V1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-22</td>
    <td>1279</td>
    <td>-658.65</td>
    <td>34.98</td>
  </tr>
  <tr>
    <td>335</td>
    <td>habano/TinyLlama-1.1B-intermediate-step-715k-1.5T-1r-5-2.2epochs-aost1-top1-instruct-V1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-20</td>
    <td>1283</td>
    <td>-648.88</td>
    <td>35.45</td>
  </tr>
  <tr>
    <td>336</td>
    <td>habano/TinyLlama-1.1B-intermediate-step-715k-1.5T-1r-5-epochs-aost1-top1-instruct-V1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-21</td>
    <td>1277</td>
    <td>-651.34</td>
    <td>35.42</td>
  </tr>
  <tr>
    <td>337</td>
    <td>habano/TinyLlama-1.1B-intermediate-step-715k-1.5T-1r-5-epochs-aost1-top1-instruct-V1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-21</td>
    <td>1288</td>
    <td>-646.55</td>
    <td>35.28</td>
  </tr>
  <tr>
    <td>338</td>
    <td>habano/TinyLlama-1.1B-intermediate-step-715k-1.5T-1r-5-epochs-aost1-top1-instruct-V1</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-21</td>
    <td>1288</td>
    <td>-646.55</td>
    <td>35.28</td>
  </tr>
  <tr>
    <td>339</td>
    <td>hakurec/nommygpt-3B</td>
    <td>gpt_neox</td>
    <td>12B</td>
    <td>2023-04-09</td>
    <td>1177</td>
    <td>-602.56</td>
    <td>38.63</td>
  </tr>
  <tr>
    <td>340</td>
    <td>haoranxu/ALMA-13B (Xu et al., 2024b,a)</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-09-17</td>
    <td>2012</td>
    <td>-563.34</td>
    <td>50.16</td>
  </tr>
  <tr>
    <td>341</td>
    <td>haoranxu/ALMA-13B-Pretrain (Xu et al., 2024b,a)</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-09-17</td>
    <td>5510</td>
    <td>-558.03</td>
    <td>51.68</td>
  </tr>
  <tr>
    <td>342</td>
    <td>haoranxu/ALMA-7B (Xu et al., 2024b,a)</td>
    <td>llama</td>
    <td>7B</td>
    <td>2023-09-17</td>
    <td>1292</td>
    <td>-599.26</td>
    <td>45.32</td>
  </tr>
  <tr>
    <td>343</td>
    <td>harborwater/open-llama-3b-clause-30k</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-11-21</td>
    <td>1200</td>
    <td>-603.17</td>
    <td>40.93</td>
  </tr>
  <tr>
    <td>344</td>
    <td>health36/HealtK-1.1B-V1-Chat-dp0</td>
    <td>llama</td>
    <td>IB</td>
    <td>2023-11-05</td>
    <td>2790</td>
    <td>-796.10</td>
    <td>33.00</td>
  </tr>
  <tr>
    <td>345</td>
    <td>heegy/UAI2A2-7b-hf (Touvron et al., 2023b)</td>
    <td>llama-2</td>
    <td>7B</td>
    <td>2023-08-04</td>
    <td>3330</td>
    <td>-596.21</td>
    <td>52.98</td>
  </tr>
  <tr>
    <td>346</td>
    <td>heegy/UAI2A2-7b-hf (Touvron et al., 2023b)</td>
    <td>llama-2</td>
    <td>7B</td>
    <td>2023-08-04</td>
    <td>3463</td>
    <td>-541.31</td>
    <td>52.27</td>
  </tr>
  <tr>
    <td>347</td>
    <td>heegy/UAI1A-13b-hf</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-08-01</td>
    <td>3225</td>
    <td>-544.21</td>
    <td>52.61</td>
  </tr>
  <tr>
    <td>348</td>
    <td>heegy/RedTulu-Uncensored-3B-0719</td>
    <td>gpt_neox</td>
    <td>3B</td>
    <td>2023-07-23</td>
    <td>1187</td>
    <td>-703.96</td>
    <td>39.19</td>
  </tr>
  <tr>
    <td>349</td>
    <td>heegy/WizardVicuna-213b-hf (Touvron et al., 2023b)</td>
    <td>llama-2</td>
    <td>13B</td>
    <td>2023-08-07</td>
    <td>6481</td>
    <td>-612.15</td>
    <td>51.05</td>
  </tr>
  <tr>
    <td>350</td>
    <td>heegy/WizardVicuna-open-llama-3b-v2</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-08-25</td>
    <td>9364</td>
    <td>-671.17</td>
    <td>38.77</td>
  </tr>
  <tr>
    <td>351</td>
    <td>heegy/WizardVicuna-Uncensored-3B-0719</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-07-23</td>
    <td>1168</td>
    <td>-638.46</td>
    <td>39.73</td>
  </tr>
  <tr>
    <td>352</td>
    <td>heegy/WizardVicuna-3B-0719</td>
    <td>llama</td>
    <td>3B</td>
    <td>2023-07-23</td>
    <td>3356</td>
    <td>-653.19</td>
    <td>38.41</td>
  </tr>
  <tr>
    <td>353</td>
    <td>HenryJJ/Instruction_Yi-6B_Dolly15K</td>
    <td>mistral</td>
    <td>7B</td>
    <td>2024-01-12</td>
    <td>1419</td>
    <td>-574.85</td>
    <td>48.55</td>
  </tr>
  <tr>
    <td>354</td>
    <td>HenryJJ/Instruction_Yi-6B_Dolly15K</td>
    <td>llama</td>
    <td>6B</td>
    <td>2024-01-06</td>
    <td>1170</td>
    <td>-524.87</td>
    <td>56.85</td>
  </tr>
  <tr>
    <td>355</td>
    <td>HenryJJ/Instruction_Yi-6B_Dolly_Coalpaca</td>
    <td>llama</td>
    <td>6B</td>
    <td>2024-01-07</td>
    <td>1177</td>
    <td>-527.06</td>
    <td>56.11</td>
  </tr>
  <tr>
    <td>356</td>
    <td>hfl/chinese-alpaca-2-13b</td>
    <td>llama</td>
    <td>13B</td>
    <td>2023-08-14</td>
    <td>1308</td>
    <td>-592.75</td>
    <td>54.71</td>
  </tr>
  <tr>
    <td>357</td>
    <td>hfl/chinese-llama-2-13b</td>
    <td>llama-2</td>
    <td>13B</td>
    <td>2023-08-11</td>
    <td>1176</td>
    <td>-665.59</td>
    <td>52.00</td>
  </tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLLs</th>
    <th>\( \hat{E}_t \)</th>
    <th>Task</th>
  </tr>
  <tr><td>358</td><td>hfl/chinese-llama-2-1-3b</td><td>llama-2</td><td>1B</td><td>2023-10-08</td><td>2269</td><td>-1132.44</td><td>28.97</td></tr>
  <tr><td>359</td><td>HHGZ/GALILEE-7B (Sainz et al., 2024)</td><td>llama-2</td><td>7B</td><td>2023-09-25</td><td>1394</td><td>-633.26</td><td>37.48</td></tr>
  <tr><td>360</td><td>hiyougA/Baichuan2-7B-Base-LlaMAfied</td><td>llama-2</td><td>7B</td><td>2023-09-08</td><td>1218</td><td>-530.80</td><td>48.99</td></tr>
  <tr><td>361</td><td>hiyougA/Baichuan2-7B-Chat-LlaMAfied</td><td>llama-2</td><td>7B</td><td>2023-09-09</td><td>1211</td><td>-609.32</td><td>51.42</td></tr>
  <tr><td>362</td><td>hoskinson-center/tedordGPT-v0-1.6-7B</td><td>gpt_neox</td><td>6B</td><td>2023-04-02</td><td>1187</td><td>-974.98</td><td>29.72</td></tr>
  <tr><td>363</td><td>hpcai-tech/Colossal-LlAMA-2-7b-base (Li et al., 2023b; Touvron et al., 2023c; Dao, 2023)</td><td>llama-2</td><td>7B</td><td>2023-09-18</td><td>1192</td><td>-690.99</td><td>51.39</td></tr>
  <tr><td>364</td><td>HuggingFace=addltion-model-fineweb-v1 (Lacoste et al., 2019)</td><td>llama</td><td>1B</td><td>2023-01-20</td><td>2228</td><td>-771.42</td><td>36.76</td></tr>
  <tr><td>365</td><td>HuggingFace/Global-server-7b-tr (Yamazaki & Amodei, 2023)</td><td>mistral</td><td>7B</td><td>2023-10-21</td><td>7463</td><td>-551.64</td><td>59.98</td></tr>
  <tr><td>366</td><td>HuggingFace/zh/7p/hr-7b-alpha (Ding et al., 2023; Tunstall et al., 2023; Rafaiol et al., 2024; Cui et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-10-09</td><td>12011</td><td>-559.75</td><td>56.50</td></tr>
  <tr><td>367</td><td>HuggingFace/zh/jeyphr-7b-beta (Ding et al., 2023; Tunstall et al., 2023; Rafaiol et al., 2024; Cui et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-10-26</td><td>29019</td><td>-570.40</td><td>59.08</td></tr>
  <tr><td>368</td><td>huggerRec-llama-1-3b</td><td>llama-1</td><td>13B</td><td>2023-04-03</td><td>6345</td><td>-541.90</td><td>51.33</td></tr>
  <tr><td>369</td><td>huge/slowHlama2-7b</td><td>llama-1</td><td>6B</td><td>2023-04-03</td><td>192383</td><td>-562.06</td><td>46.37</td></tr>
  <tr><td>370</td><td>HWER/lJama2-7b-sharegrept</td><td>llama-2</td><td>7B</td><td>2023-08-04</td><td>1200</td><td>-660.22</td><td>51.05</td></tr>
  <tr><td>371</td><td>HWER/python-1.4e-dedupded-sharegrept</td><td>gpt_neox</td><td>1B</td><td>2023-10-18</td><td>1200</td><td>-557.33</td><td>35.11</td></tr>
  <tr><td>372</td><td>hyunsokko/kio-en-llama2-13b</td><td>llama-2</td><td>13B</td><td>2023-10-30</td><td>3351</td><td>-550.60</td><td>51.27</td></tr>
  <tr><td>373</td><td>hyunsokko/i-kt-food-llama2-13b</td><td>llama-2</td><td>13B</td><td>2023-10-40</td><td>3364</td><td>-775.26</td><td>43.62</td></tr>
  <tr><td>374</td><td>hyunsokko/doc-llama2-7B</td><td>llama-2</td><td>13B</td><td>2023-10-40</td><td>3312</td><td>-831.38</td><td>40.75</td></tr>
  <tr><td>375</td><td>hwu/Cameliae-v7B-R8 (Housby et al., 2019; Detmers et al., 2023; Komatsu et al., 2023; Wu et al., 2024)</td><td>cameliidae</td><td>13B</td><td>2024-01-10</td><td>1886</td><td>-535.46</td><td>59.40</td></tr>
  <tr><td>376</td><td>uwu/Cameliae-x7B (Housby et al., 2019; Detmers et al., 2023; Komatsu et al., 2023; Wu et al., 2024)</td><td>cameliidae</td><td>7B</td><td>2024-01-10</td><td>1901</td><td>-565.23</td><td>54.47</td></tr>
  <tr><td>377</td><td>xboah12ngm-gm-assst-1cm-1024-12B</td><td>gpt_neox</td><td>12B</td><td>2025-01-27</td><td>1185</td><td>-429.39</td><td>34.65</td></tr>
  <tr><td>378</td><td>xcboah12ngm-gm-assst-1cn-704-viewer2-300Bw</td><td>llama-1</td><td>7B</td><td>2023-09-25</td><td>1141</td><td>-1042.99</td><td>34.12</td></tr>
  <tr><td>379</td><td>subKAI/dogTTS-2-7B-Base-GCN-Instructions</td><td>gpt_neox</td><td>1B</td><td>2024-02-10</td><td>1100</td><td>-740.84</td><td>37.65</td></tr>
  <tr><td>380</td><td>hT42H2/strip-mlan-2.5f-b-1.7ll芩ba-</td><td>gpt_neox</td><td>1B</td><td>2024-03-13</td><td>15748</td><td>-324.68</td><td>1A.58</td></tr>
  <tr><td>381</td><td>hT42H2/strip-octts-sailiated-5.12.b-9b/kg</td><td>gpt_neox</td><td>9B</td><td>2024-03-10</td><td>8191</td><td>-952.71</td><td>38.62</td></tr>
  <tr><td>382</td><td>Jemile/a/track-as convergence-dpo</td><td>llama-3</td><td>8B</td><td>2024-01-54</td><td>6805</td><td>-576.43</td><td>67.92</td></tr>
  <tr><td>383</td><td>b sandy/neMiNTE-C7B/SuladariAR et al., 2024)</td><td>mistral</td><td>7B</td><td>2024-02-30</td><td>11156</td><td>-541.12</td><td>64.00</td></tr>
  <tr><td>384</td><td>/nnuidas/RlHermeMoe-27B-ht</td><td>mistral</td><td>12B</td><td>2024-01-31</td><td>1127</td><td>-533.10</td><td>64.08</td></tr>
  <tr><td>385</td><td>b shraze/airoprc-llama2-13b-hfêt</td><td>llama-2</td><td>13B</td><td>2023-10-06</td><td>21161</td><td>-549.57</td><td>48.97</td></tr>
  <tr><td>386</td><td><a href="https://pennlinc.github.io/mټemlle/">IdeaCCNL-ZiYa-LLAMA-13B-v1 (IDEACCNL-2021; Yang et al., 2022; Zhang et al., 2022a)</a></td><td>llama-2</td><td>13B</td><td>2025-16-16</td><td>1244</td><td>-11974.97</td><td>29.82</td></tr>
  <tr><td>387</td><td>ideaCCNL-ZiYa-LLAMA-13B-v1 (IDEA-CCNL, 2021; Yang et al., 2022;</td><td>llama-4</td><td>13B</td><td>2025-16-16</td><td>1234</td><td>-11974.97</td><td>29.82</td></tr>
  <tr><td>388</td><td>Ignos/les0csorpus/Green-Node-Alpaca-7B-v1</td><td>mistral</td><td>7B</td><td>2025-12-19</td><td>1329</td><td>-599.65</td><td>51.28</td></tr>
  <tr><td>389</td><td>agnosis0157/Petsqlintegration-10-BP</td><td>illama</td><td>10B</td><td>2023-11-25</td><td>1908</td><td>-575.01</td><td>56.79</td></tr>
  <tr><td>390</td><td>Igpos/Mistral7FT-7B-v1</td><td>mistral</td><td>7B</td><td>2024-01-27</td><td>2104</td><td>-524.62</td><td>38.87</td></tr>
  <tr><td>391</td><td>KaurDev/Athena-v4</td><td>llama</td><td>12B</td><td>2024-03-18</td><td>4092</td><td>-448.06</td><td>61.11</td></tr>
  <tr><td>392</td><td>KaurDev/Athena-v4</td><td>llama</td><td>12B</td><td>2024-03-18</td><td>4255</td><td>-566.11</td><td>52.12</td></tr>
  <tr><td>393</td><td>kaurstt-Institute/BgG-PFT-Inject-v0.2</td><td>mistral</td><td>7B</td><td>2024-03-03</td><td>3265</td><td>-594.56</td><td>63.08</td></tr>
  <tr><td>394</td><td>telantar-ball-eh3-v3 (Mukherjee et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-11-14</td><td>3978</td><td>-635.71</td><td>59.50</td></tr>
  <tr><td>395</td><td>telantar-ewhat-v7-6-v2 (Yu et al., 2024b)</td><td>mistral</td><td>7B</td><td>2024-01-17</td><td>6202</td><td>-587.74</td><td>48.29</td></tr>
  <tr><td>396</td><td>InYet-Context/SALlikSDBC7SΤOR (Inschlinken-Vainanien-10.7bv2.1</td><td>llama</td><td>7B</td><td>2022-10-20</td><td>166266</td><td>-4927.55</td><td>96.83</td></tr>
  <tr><td>397</td><td>tiiliurpng athletes67B-ht</td><td>llama-2</td><td>12B</td><td>2023-09-28</td><td>2200</td><td>-586.33</td><td>50.48</td></tr>
  <tr><td>398</td><td>tiiliurpng allergies7Ba-eic (Shi et al., 2024)</td><td>llama</td><td>12B</td><td>2023-12-08</td><td>2086</td><td>-560.36</td><td>47.93</td></tr>
  <tr><td>399</td><td>jae74/openhermes_dpo_notower_021</td><td>mistral</td><td>7B</td><td>2024-01-02</td><td>11161</td><td>-561.25</td><td>63.78</td></tr>
  <tr><td>400</td><td>jem-hq/trinity-v1</td><td>mistral</td><td>7B</td><td>2023-12-14</td><td>1192</td><td>-559.84</td><td>74.80</td></tr>
  <tr><td>401</td><td>Jayans9288/aeroso_m3</td><td>llama</td><td>10B</td><td>2024-03-01</td><td>2766</td><td>-585.53</td><td>62.21</td></tr>
  <tr><td>402</td><td>jh4sworld carbonllama10.7B v1</td><td>llama</td><td>12B</td><td>2024-12-08</td><td>12241+zz</td><td>-655.64</td><td>38.5</td></tr>
  <tr><td>403</td><td>jonabartel/notebookess2.0-1.8.1</td><td>llama</td><td>10B</td><td>2024-01-15</td><td>192146+</td><td>-561.41</td><td>46.37</td></tr>
  <tr><td>404</td><td> JonasUlrafall/Vixastrong-1126v2</td><td>llama</td><td>12B</td><td>2024-03-21</td><td>3094-144</td><td>-561.47</td><td>34.76</td></tr>
  <tr><td>405</td><td>JonasUlnark/Fachine-13.0-c1,18</td><td>llama</td><td>12B</td><td>2024-01-29</td><td>2287+51</td><td>-541.71</td><td>31.06</td></tr>
  <tr><td>406</td><td>JonhnVolfsnew/BLL-ding-2d</td><td>mistral</td><td>7A</td><td>2024-04-08</td><td>2682</td><td>-591.07</td><td>60.32</td></tr>
  <tr><td>407</td><td>johnsnowbands/SeMdentiamatives8-v2.0</td><td>llama-3</td><td>13B</td><td>2024-03-30</td><td>11813</td><td>-569.01</td><td>61.93</td></tr>
  <tr><td>408</td><td>jondurbin/athorros-3.1-shrub-150B-7b</td><td>llama</td><td>12B</td><td>2025-03-12</td><td>1471-84</td><td>-614.18</td><td>67.25</td></tr>
  <tr><td>409</td><td>jondurbin/ataros-gfz-2B-19thtuluito-</td><td>llama</td><td>12B</td><td>2025-03-27</td><td>1531-814-8</td><td>-584.55</td><td>35.95</td></tr>
  <tr><td>410</td><td>Jondurbrenpvoro-15.18tens-2.0</td><td>llama</td><td>12B</td><td>2025-02-28</td><td>2332</td><td>-554.56</td><td>53.34</td></tr>
  <tr><td>411</td><td>jondurbin/Adj812-v0.1</td><td>llama-3</td><td>8B</td><td>2024-04-24</td><td>9780</td><td>-512.36</td><td>67.84</td></tr>
  <tr><td>412</td><td>nctrisgan/flowers/Tiny-Llama-3T-Cinder-v1.2</td><td>llama</td><td>1B</td><td>2023-12-31</td><td>1413+</td><td>-7770.5</td><td>35.26</td></tr>
  <tr><td>413</td><td>joechershoaugeung-7B-MSMuan7ABC-7HT聊天</td><td>llama-2</td><td>7B</td><td>2023-08-04</td><td>1212</td><td>-646.77</td><td>51.99</td></tr>
  <tr><td>414</td><td>JosehusCheungequipment-LL-LLMAisher-TB</td><td>llama-2</td><td>7B</td><td>2023-08-30</td><td>1186</td><td>-788.03</td><td>40.00</td></tr>
  <tr><td>415</td><td>JosehusCheunglsevven-7/+mLMAAFied-TB- charset</td><td>llama-2</td><td>13B</td><td>2024-05-17</td><td>11900</td><td>-625.44</td><td>51.59</td></tr>
  <tr><td>416</td><td>jpetrepzj me_10 (Chillacentlass en Trainbons 2023b)</td><td>llama-2</td><td>13B</td><td>2023-07-27</td><td>12615</td><td>-574.88</td><td>55.07</td></tr>
  <tr><td>417</td><td>jrexticesleV-62-llIBClustersject.LeoVitasso-Type TP-CN (%) 2v.</td><td>llama-2</td><td>13B</td><td>2023-07-04</td><td>11815</td><td>-693.64</td><td>47.65</td></tr>
  <tr><td>418</td><td>junelec/wizericUserInfo-7mAhja13B</td><td>llama-1</td><td>7B</td><td>2024-03-09</td><td>12758</td><td>-590.27</td><td>56.35</td></tr>
  <tr><td>419</td><td>Keepblog/Mistr(r-zeScrphy-FetOA</td><td>mistral</td><td>7B</td><td>2024-03-09</td><td>12837.2</td><td>-6841.16</td><td>57.37</td></tr>
  <tr><td>420</td><td>Kaist-in/astraße_girts-params-v6 (hong strengthen_2024)</td><td>mistral</td><td>7B</td><td>2004-03-29</td><td>4211</td><td>-660.46</td><td>63.36</td></tr>
  <tr><td>421</td><td>hhohmtSjURIreiara-7MBorn</td><td>t5-large-qerffk</td><td>18B</td><td>2023-08-19</td><td>13934</td><td>-5644.12</td><td>41.94</td></tr>
  <tr><td>422</td><td>kobte/MCSQenhkarinsften-7kS1</td><td>azyawclesslyahn-7m</td><td>12B</td><td>2024-04-24</td><td>2168</td><td>-670.04</td><td>74.05</td></tr>
  <tr><td>423</td><td>Keks/Llama25-kbh-7gear (Touvron et al., 2023b)</td><td>llama-2</td><td>1B</td><td>2023-07-24</td><td>1317</td><td>-13776.65</td><td>39.47</td></tr>
  <tr><td>424</td><td>kubieizi/grailggdima-choao-gradd-12-13B</td><td>llama-2</td><td>13B</td><td>2023-05-03</td><td>2756</td><td>-537.10</td><td>36.53</td></tr>
  <tr><td>425</td><td>Kabstebio/Mstral:azionioytv-Beta ShotsRELP</td><td>mistral</td><td>7B</td><td>2023-03-09</td><td>2762</td><td>-603.34</td><td>57.73</td></tr>
  <tr><td>426</td><td>KahstBoth[Mistnerlt-2 الجمهوري</td><td>mistral</td><td>7B</td><td>2024-03-03</td><td>4281</td><td>-546.06</td><td>61.95</td></tr>
  <tr><td>427</td><td>Kekmodelburgin-1072-7Thicts</td><td>llama</td><td>18B</td><td>2023-08-29</td><td>13035</td><td>-54061.20</td><td>94.9</td></tr>
  <tr><td>428</td><td>KeksLerna-1cinZomel-121BB1</td><td>mistral</td><td>7B</td><td>2023-04-20</td><td>9128</td><td>-678.30</td><td>40.27</td></tr>
  <tr><td>429</td><td>kingbi/borchomaisionn-abaco-grad-12-13B</td><td>llama-2</td><td>13B</td><td>2023-08-04</td><td>11862</td><td>-382.73</td><td>55.50</td></tr>
  <tr><td>430</td><td>klandyi/MentalLlama-7hlrZH-Y ~(Yang et al., 2024b)</td><td>llama</td><td>7B</td><td>2023-09-26</td><td>2634</td><td>-640.26</td><td>51.17</td></tr>
  <tr><td>431</td><td>KnutJaegersberg/dceceaun-13b</td><td>llama</td><td>13B</td><td>2023-09-20</td><td>281</td><td>-339.47</td><td>53.63</td></tr>
  <tr><td>432</td><td>KnutJaegersberg/daloentJC-e2f-taf</td><td>falcon</td><td>1B</td><td>2023-12-04</td><td>2009</td><td>-978.54</td><td>55.02</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>437</td><td>KnutJaegersberg/LLongMA-3b-LIMA</td><td>llama</td><td>3B</td><td>2023-09-03</td><td>2010</td><td>-617.15</td><td>38.51</td></tr>
  <tr><td>438</td><td>KnutJaegersberg/MistralInstructLongsh</td><td>mistral</td><td>7B</td><td>2023-11-15</td><td>1970</td><td>-543.74</td><td>53.62</td></tr>
  <tr><td>439</td><td>KnutJaegersberg/Owen-1-8B-Llama</td><td>llama</td><td>1B</td><td>2024-01-03</td><td>2839</td><td>-760.12</td><td>44.75</td></tr>
  <tr><td>440</td><td>KnutJaegersberg/Walter-Falcon-1B</td><td>falcon</td><td>1B</td><td>2023-12-09</td><td>2014</td><td>-1010.10</td><td>34.07</td></tr>
  <tr><td>441</td><td>KnutJaegersberg/webMistral-7B</td><td>mistral</td><td>7B</td><td>2023-11-17</td><td>1972</td><td>-554.62</td><td>53.97</td></tr>
  <tr><td>442</td><td>KoboldAI/GPTJ-6B-Adventure</td><td>gptj</td><td>6B</td><td>2022-03-02</td><td>1297</td><td>-661.77</td><td>35.95</td></tr>
  <tr><td>443</td><td>KoboldAI/GPTJ-6B-Janeway (Gao et al., 2020; Wang and Komatsuzaki, 2021)</td><td>gptj</td><td>6B</td><td>2022-03-02</td><td>4382</td><td>-496.37</td><td>39.54</td></tr>
  <tr><td>444</td><td>KoboldAI/GPTJ-6B-Shinen (Gao et al., 2020; Wang and Komatsuzaki, 2021)</td><td>gptj</td><td>6B</td><td>2022-03-02</td><td>1498</td><td>-498.02</td><td>39.60</td></tr>
  <tr><td>445</td><td>KoboldAI/GPTJ-6B-Skein (Lacoste et al., 2019; Wang, 2021)</td><td>gptj</td><td>6B</td><td>2022-03-02</td><td>1236</td><td>-507.40</td><td>40.02</td></tr>
  <tr><td>446</td><td>KoboldAI/LaMA2-13B-Holmos</td><td>llama-2</td><td>13B</td><td>2023-08-14</td><td>1255</td><td>-563.47</td><td>54.52</td></tr>
  <tr><td>447</td><td>KoboldAI/LaMA2-13B-Tiefighter</td><td>llama-2</td><td>13B</td><td>2023-10-18</td><td>2321</td><td>-599.88</td><td>54.51</td></tr>
  <tr><td>448</td><td>KoboldAI/OTP-13B-Erebos (Zhang et al., 2022b)</td><td>opt</td><td>13B</td><td>2022-09-09</td><td>6998</td><td>-652.81</td><td>39.61</td></tr>
  <tr><td>449</td><td>KoboldAI/OTP-13B-Nervous-Mix (Zhang et al., 2022b)</td><td>opt</td><td>13B</td><td>2023-02-13</td><td>1657</td><td>-646.08</td><td>39.61</td></tr>
  <tr><td>450</td><td>KoboldAI/OTP-13B-Nervy-v2 (Zhang et al., 2022b)</td><td>opt</td><td>13B</td><td>2022-09-09</td><td>4259</td><td>-659.38</td><td>39.53</td></tr>
  <tr><td>451</td><td>KoboldAI/OTP-27B-Erebos (Zhang et al., 2022b)</td><td>opt</td><td>27B</td><td>2023-02-19</td><td>5349</td><td>-692.73</td><td>39.66</td></tr>
  <tr><td>452</td><td>KoboldAI/OTP-27B-Nervous-Mix</td><td>opt</td><td>27B</td><td>2023-02-09</td><td>1464</td><td>-686.59</td><td>36.88</td></tr>
  <tr><td>453</td><td>KoboldAI/OTP-6B-nervy-v2 (Zhang et al., 2022b)</td><td>opt</td><td>6B</td><td>2022-06-26</td><td>4974</td><td>-566.52</td><td>38.72</td></tr>
  <tr><td>454</td><td>KoboldAI/OTP-6.7B-Erebos (Zhang et al., 2022b)</td><td>opt</td><td>6B</td><td>2022-09-15</td><td>5587</td><td>-641.11</td><td>39.09</td></tr>
  <tr><td>455</td><td>KoboldAI/OTP-6.7B-Nervous-Mix</td><td>opt</td><td>6B</td><td>2023-02-13</td><td>1585</td><td>-642.68</td><td>38.83</td></tr>
  <tr><td>456</td><td>kodono/Solar-M-SakuraSolar-SLERP</td><td>llama</td><td>10B</td><td>2024-01-12</td><td>3254</td><td>-562.65</td><td>74.29</td></tr>
  <tr><td>457</td><td>kodono/Solar-OrcadDPO-Solar-Instruct-SLERP</td><td>llama</td><td>10B</td><td>2024-01-12</td><td>3266</td><td>-560.11</td><td>74.35</td></tr>
  <tr><td>458</td><td>Korabit/Llama-2-7b-chat-hf-atr-100step-flan</td><td>llama-2</td><td>7B</td><td>2023-12-03</td><td>1208</td><td>-658.08</td><td>52.28</td></tr>
  <tr><td>459</td><td>Korabit/Llama-2-7b-chat-hf-atr-100step-flan-v2</td><td>llama-2</td><td>7B</td><td>2023-12-03</td><td>1208</td><td>-658.08</td><td>52.28</td></tr>
  <tr><td>460</td><td>Korabit/Llama-2-7b-chat-hf-atr-100step-v2</td><td>llama-2</td><td>7B</td><td>2023-11-22</td><td>1205</td><td>-655.71</td><td>50.89</td></tr>
  <tr><td>461</td><td>Korabit/Llama-2-7b-chat-hf-atr-200step-flan</td><td>llama-2</td><td>7B</td><td>2023-11-30</td><td>1201</td><td>-647.72</td><td>52.62</td></tr>
  <tr><td>462</td><td>Korabit/Llama-2-7b-chat-hf-atr-200step-flan-v2</td><td>llama-2</td><td>7B</td><td>2023-12-03</td><td>1211</td><td>-648.67</td><td>52.75</td></tr>
  <tr><td>463</td><td>Korabit/Llama-2-7b-chat-hf-atr-200step-merged</td><td>llama-2</td><td>7B</td><td>2023-11-21</td><td>1222</td><td>-653.05</td><td>52.26</td></tr>
  <tr><td>464</td><td>Korabit/Llama-2-7b-chat-hf-atr-300step-v2</td><td>llama-2</td><td>7B</td><td>2023-11-22</td><td>1203</td><td>-658.69</td><td>50.21</td></tr>
  <tr><td>465</td><td>Korabit/Llama-2-7b-chat-hf-atr-300step-flan-v2</td><td>llama-2</td><td>7B</td><td>2023-12-03</td><td>1208</td><td>-647.99</td><td>52.41</td></tr>
  <tr><td>466</td><td>Korabit/Llama-2-7b-chat-hf-atr-300step-flan-v2</td><td>llama-2</td><td>7B</td><td>2023-12-03</td><td>1208</td><td>-647.99</td><td>52.28</td></tr>
  <tr><td>467</td><td>Korabit/Llama-2-7b-chat-hf-atr-768-logic-Coder-v7.5</td><td>mistral</td><td>27B</td><td>2024-04-07</td><td>4047</td><td>-569.72</td><td>72.28</td></tr>
  <tr><td>468</td><td>Kukedlc/NeuralLLama-3-8B-DT-v1</td><td>llama-3</td><td>8B</td><td>2024-05-11</td><td>5707</td><td>-571.65</td><td>72.52</td></tr>
  <tr><td>469</td><td>Kukedlc/NeuralLLama-3-8B-ORPO-v0.3</td><td>llama-3</td><td>8B</td><td>2024-05-14</td><td>7985</td><td>-562.41</td><td>72.66</td></tr>
  <tr><td>470</td><td>Kukedlc/NeuralSynthesis-7B-v0.1</td><td>mistral</td><td>7B</td><td>2024-04-06</td><td>8910</td><td>-596.75</td><td>76.80</td></tr>
  <tr><td>471</td><td>Kukedlc/NeuralSynthesis-7B-v0.3</td><td>mistral</td><td>7B</td><td>2024-04-07</td><td>3114</td><td>-597.52</td><td>76.70</td></tr>
  <tr><td>472</td><td>Kukedlc/NeuralSynthesis-7B-v0.4-sleep</td><td>mistral</td><td>7B</td><td>2024-04-12</td><td>3067</td><td>-597.78</td><td>76.76</td></tr>
  <tr><td>473</td><td>kyujinpy/Sakura-SOLAR-Instruct</td><td>llama</td><td>10B</td><td>2023-12-24</td><td>4271</td><td>-559.44</td><td>74.40</td></tr>
  <tr><td>474</td><td>kyujinpy/Sakura-SOLAR-Instruct-DPO-v1</td><td>llama</td><td>10B</td><td>2023-12-25</td><td>3284</td><td>-562.45</td><td>74.13</td></tr>
  <tr><td>475</td><td>kyujinpy/Sakura-SOLAR-Math-Instruct-DPO-v1</td><td>llama</td><td>10B</td><td>2023-09-19</td><td>1172</td><td>-555.04</td><td>50.45</td></tr>
  <tr><td>476</td><td>Lazycuber/L-2.7b-Basic-Gianaco-Uncensored</td><td>llama-3</td><td>9B</td><td>2024-05-15</td><td>2230</td><td>-558.58</td><td>54.76</td></tr>
  <tr><td>477</td><td>lcw99/lama-3-10b-it-kor-exented-chang</td><td>llama-3</td><td>9B</td><td>2024-05-21</td><td>2234</td><td>-561.88</td><td>63.76</td></tr>
  <tr><td>478</td><td>lcw99/lama-3-10b-it-kor-exented-chang-pro8</td><td>llama-3</td><td>9B</td><td>2024-05-27</td><td>2236</td><td>-564.03</td><td>63.70</td></tr>
  <tr><td>479</td><td>lcw99/lama-3-8b-it-kor-exented-chang</td><td>llama-3</td><td>8B</td><td>2024-05-02</td><td>2259</td><td>-527.86</td><td>66.27</td></tr>
  <tr><td>480</td><td>LDCC/LDCC-SOLAR-10.7B (Kim et al., 2024b)</td><td>llama</td><td>10B</td><td>2024-01-03</td><td>5248</td><td>-599.71</td><td>71.40</td></tr>
  <tr><td>481</td><td>lemon-minit/gemma-ko-7b-instruct-v0.62</td><td>gemma</td><td>8B</td><td>2024-03-06</td><td>2337</td><td>-580.64</td><td>59.02</td></tr>
  <tr><td>482</td><td>lemon-minit/gemma-ko-7b-instruct-v0.71</td><td>gemma</td><td>8B</td><td>2024-04-03</td><td>7543</td><td>-589.67</td><td>69.25</td></tr>
  <tr><td>483</td><td>lemon-minit/gemma-7b-openhemes-v0.80</td><td>gemma</td><td>8B</td><td>2024-04-09</td><td>2232</td><td>-611.63</td><td>59.23</td></tr>
  <tr><td>484</td><td>LeoLM/leo-hessianai-7b</td><td>llama</td><td>7B</td><td>2023-08-22</td><td>4502</td><td>-591.31</td><td>47.72</td></tr>
  <tr><td>485</td><td>LeoLM/leo-hessianai-7b-chat</td><td>llama</td><td>7B</td><td>2023-09-10</td><td>3924</td><td>-735.02</td><td>49.29</td></tr>
  <tr><td>486</td><td>lex-fue/Detector</td><td>mistral</td><td>7B</td><td>2024-01-18</td><td>1926</td><td>-577.38</td><td>51.15</td></tr>
  <tr><td>487</td><td>lgalves/mistral-13b-chat-platypus</td><td>llama-2</td><td>13B</td><td>2023-09-06</td><td>1193</td><td>-570.00</td><td>53.92</td></tr>
  <tr><td>488</td><td>lgalves/mistral-7b-hf_open-platypus</td><td>llama-2</td><td>7B</td><td>2023-08-30</td><td>1204</td><td>-564.47</td><td>49.73</td></tr>
  <tr><td>489</td><td>lgalves/mistral-7b_hf_open_platypus</td><td>mistral</td><td>7B</td><td>2023-10-10</td><td>1183</td><td>-539.54</td><td>58.19</td></tr>
  <tr><td>490</td><td>lgalves/tinylama-1.1b-chat-v0.3_platypus</td><td>llama</td><td>1B</td><td>2023-10-09</td><td>1193</td><td>-665.44</td><td>34.50</td></tr>
  <tr><td>491</td><td>lightblue/suzume-Llama-3-8B-multilingual (Devine, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-23</td><td>16442</td><td>-557.50</td><td>65.55</td></tr>
  <tr><td>492</td><td>lighternal/Llama-3-merge-biomed-8b (Yadav et al., 2023a; Yu et al., 2024a)</td><td>llama-3</td><td>8B</td><td>2024-05-28</td><td>2731</td><td>-575.20</td><td>66.30</td></tr>
  <tr><td>493</td><td>liminerity/M7T-7b</td><td>mistral</td><td>7B</td><td>2024-04-30</td><td>4186</td><td>-599.03</td><td>58.82</td></tr>
  <tr><td>494</td><td>liminerity/M7T-7b-chat</td><td>llama-2</td><td>7B</td><td>2023-10-20</td><td>4079</td><td>-584.59</td><td>52.59</td></tr>
  <tr><td>495</td><td>lim-agents/orca-code-7b-v1.0 (Gou et al., 2024)</td><td>llama-2</td><td>7B</td><td>2023-10-08</td><td>1242</td><td>-682.23</td><td>40.21</td></tr>
  <tr><td>500</td><td>llmsys/longchat-13b-16k</td><td>llama-1</td><td>13B</td><td>2023-06-28</td><td>12228</td><td>-621.21</td><td>49.64</td></tr>
  <tr><td>501</td><td>llmsys/longchat-v1.5-32k</td><td>llama</td><td>7B</td><td>2023-08-01</td><td>5114</td><td>-605.92</td><td>47.95</td></tr>
  <tr><td>503</td><td>llmsys/vicuna-13b-delta-v1.1 (Zheng et al., 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-04-12</td><td>1216</td><td>-1396.42</td><td>53.28</td></tr>
  <tr><td>504</td><td>llmsys/vicuna-13b-v1.1 (Zheng et al., 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-04-12</td><td>2662</td><td>-646.46</td><td>53.28</td></tr>
  <tr><td>505</td><td>llmsys/vicuna-13b-v1.3 (Zheng et al., 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-04-12</td><td>1193</td><td>-610.80</td><td>52.27</td></tr>
  <tr><td>506</td><td>llmsys/vicuna-13b-v1.5-16k (Touvron et al., 2023b; Zheng et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-07-29</td><td>48653</td><td>-585.07</td><td>55.41</td></tr>
  <tr><td>507</td><td>llmsys/vicuna-13b-v1.5-16k (Touvron et al., 2023b; Zheng et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-08-01</td><td>2980</td><td>-595.73</td><td>54.97</td></tr>
  <tr><td>509</td><td>llmsys/vicuna-7b-v1.3 (Zheng et al., 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>7B</td><td>2023-06-18</td><td>31436</td><td>-621.79</td><td>49.78</td></tr>
  <tr><td>510</td><td>llmsys/vicuna-7b-v1.5 (Touvron et al., 2023b; Zheng et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-07-29</td><td>368410</td><td>-608.63</td><td>52.06</td></tr>
  <tr><td>511</td><td>Locutusque/Hercules-2.5-Mistral-7B</td><td>mistral</td><td>7B</td><td>2023-07-31</td><td>3362</td><td>-622.49</td><td>51.42</td></tr>
  <tr><td>512</td><td>Locutusque/Hercules-2.5-Mistral-7B</td><td>mistral</td><td>7B</td><td>2024-02-10</td><td>1953</td><td>-553.61</td><td>63.59</td></tr>
  <tr><td>513</td><td>Locutusque/Oreca-2.13b-SFT-v6</td><td>llama</td><td>13B</td><td>2023-11-25</td><td>2797</td><td>-582.16</td><td>59.09</td></tr>
  <tr><td>514</td><td>Locutusque/Oreca-2.13b-SFT_v5</td><td>llama</td><td>13B</td><td>2023-12-22</td><td>2319</td><td>-674.35</td><td>56.15</td></tr>
  <tr><td>515</td><td>Locutusque/Oreca-2.13b-SFT_v5</td><td>llama</td><td>13B</td><td>2023-12-13</td><td>2191</td><td>-602.83</td><td>56.77</td></tr>
  <tr><td>517</td><td>LTC-AI-Labs/L2-7b-Base-WVG-Uncensored</td><td>llama</td><td>7B</td><td>2023-09-23</td><td>1241</td><td>-556.11</td><td>50.63</td></tr>
  <tr><td>518</td><td>LTC-AI-Labs/L2-7b-Beluga-WVG-Test</td><td>llama</td><td>7B</td><td>2023-10-03</td><td>1222</td><td>-567.10</td><td>52.04</td></tr>
  <tr><td>519</td><td>LTC-AI-Labs/L2-7b-Hermes-Syntha</td><td>llama-2</td><td>7B</td><td>2023-11-23</td><td>1232</td><td>-562.42</td><td>52.21</td></tr>
  <tr><td>520</td><td>LTC-AI-Labs/L2-7b-Hermes-WVG-Test</td><td>llama</td><td>7B</td><td>2023-09-27</td><td>1225</td><td>-570.23</td><td>51.35</td></tr>
  <tr><td>521</td><td>LTC-AI-Labs/L2-7b-Syntha-WVG-Test</td><td>llama</td><td>7B</td><td>2023-09-28</td><td>1233</td><td>-576.22</td><td>51.25</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>522</td><td>luffycodes/vicuna-13b-v1-dot5-ep2-v-rag-w-simple (Sonkar et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-08-21</td><td>1165</td><td>-603.30</td><td>55.40</td></tr>
  <tr><td>523</td><td>luffycodes/vicuna-class-shishya-13b-ep3 (Sonkar et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-12-21</td><td>2902</td><td>-633.43</td><td>48.52</td></tr>
  <tr><td>524</td><td>luffycodes/vicuna-class-shishya-7b-ep3 (Sonkar et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-12-14</td><td>3501</td><td>-649.39</td><td>46.14</td></tr>
  <tr><td>525</td><td>luffycodes/vicuna-class-tutor-13b-ep3 (Sonkar et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-12-21</td><td>1514</td><td>-605.75</td><td>55.88</td></tr>
  <tr><td>526</td><td>luffycodes/vicuna-class-tutor-7b-ep3 (Sonkar et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-12-15</td><td>3735</td><td>-643.38</td><td>51.45</td></tr>
  <tr><td>527</td><td>lu-vae/llama2-13B-sharegpt4-orca-openplattypus-8w</td><td>llama-2</td><td>13B</td><td>2023-09-14</td><td>1190</td><td>-544.97</td><td>55.75</td></tr>
  <tr><td>528</td><td>lu-vae/llama2-13B-sharegpt4-test</td><td>llama-2</td><td>13B</td><td>2023-09-07</td><td>1195</td><td>-557.27</td><td>55.69</td></tr>
  <tr><td>529</td><td>lyogavin/Animla-7b-100k</td><td>llama-2</td><td>7B</td><td>2023-09-14</td><td>1207</td><td>-611.42</td><td>42.98</td></tr>
  <tr><td>530</td><td>13utterfly/llama2-7b-layla</td><td>llama-2</td><td>7B</td><td>2023-12-12</td><td>1178</td><td>-560.12</td><td>52.05</td></tr>
  <tr><td>531</td><td>13utterfly/llama2-7b-layla-v1</td><td>llama-2</td><td>7B</td><td>2023-12-12</td><td>1178</td><td>-568.02</td><td>52.21</td></tr>
  <tr><td>532</td><td>13utterfly/minima-3b-layla-v2</td><td>llama-2</td><td>3B</td><td>2023-12-19</td><td>1176</td><td>-547.03</td><td>43.39</td></tr>
  <tr><td>533</td><td>13utterfly/mistral-7b-v0.1-layla-v1</td><td>mistral</td><td>7B</td><td>2023-10-31</td><td>1206</td><td>-602.61</td><td>57.56</td></tr>
  <tr><td>534</td><td>13utterfly/mistral-7b-v0.1-layla-v2</td><td>mistral</td><td>7B</td><td>2023-12-16</td><td>1199</td><td>-567.42</td><td>57.60</td></tr>
  <tr><td>535</td><td>13utterfly/open-llama-3b-v2-layla</td><td>llama</td><td>3B</td><td>2023-08-18</td><td>1172</td><td>-832.18</td><td>40.25</td></tr>
  <tr><td>536</td><td>macadellec/laser-dolphin-mixtial-2x7b-dpo (Gao et al., 2021a; Sharma et al., 2023)</td><td>mistral</td><td>12B</td><td>2024-01-08</td><td>1275</td><td>-594.20</td><td>61.16</td></tr>
  <tr><td>537</td><td>macadellec/Westf_ake-7b-v2-laser-truthy-dpo</td><td>mistral</td><td>7B</td><td>2024-01-27</td><td>5064</td><td>-593.71</td><td>75.37</td></tr>
  <tr><td>538</td><td>malhajar/Mistral-7B-v0.2-medtron-turkish</td><td>mistral</td><td>7B</td><td>2024-01-05</td><td>3865</td><td>-641.84</td><td>63.34</td></tr>
  <tr><td>539</td><td>martyr/llama2-megamerge-dare-13b-v2</td><td>llama-2</td><td>13B</td><td>2023-12-17</td><td>1213</td><td>-594.59</td><td>57.94</td></tr>
  <tr><td>540</td><td>martyr/mistral-megamerge-dare-7b</td><td>mistral</td><td>7B</td><td>2023-12-14</td><td>1203</td><td>-638.88</td><td>48.93</td></tr>
  <tr><td>541</td><td>martyr/solar-megamerge-dare-10.7b-v1</td><td>llama</td><td>10B</td><td>2023-12-31</td><td>1190</td><td>-533.45</td><td>68.79</td></tr>
  <tr><td>542</td><td>matsoo/lab-weblab-10b</td><td>llama</td><td>10B</td><td>2023-08-04</td><td>1596</td><td>-485.56</td><td>38.59</td></tr>
  <tr><td>543</td><td>matsoo/lab-weblab-10b-instruction-sft</td><td>llama</td><td>10B</td><td>2023-08-04</td><td>1253</td><td>-515.77</td><td>39.13</td></tr>
  <tr><td>544</td><td>Maywell/PPO-PT-Franklin</td><td>opt</td><td>7B</td><td>2023-08-26</td><td>1237</td><td>-1386.66</td><td>57.87</td></tr>
  <tr><td>545</td><td>Maywell/PPO-PT-Franklin</td><td>opt</td><td>6B</td><td>2023-08-15</td><td>1193</td><td>-729.78</td><td>35.84</td></tr>
  <tr><td>546</td><td>Maywell/PVO1-10.7B-Mistral-v0.2</td><td>mistral</td><td>7B</td><td>2023-11-24</td><td>3267</td><td>-681.01</td><td>64.58</td></tr>
  <tr><td>547</td><td>Maywell/PVO1-10.7B-Mistral-v0.2</td><td>mistral</td><td>10B</td><td>2023-12-13</td><td>3228</td><td>-578.51</td><td>64.25</td></tr>
  <tr><td>548</td><td>Maywell/Synatra-RP-Orcas-2-7b-v0.1</td><td>llama</td><td>6B</td><td>2023-11-21</td><td>3278</td><td>-632.02</td><td>59.65</td></tr>
  <tr><td>549</td><td>Maywell/Synatra-V0.1-7B-Instruct</td><td>mistral</td><td>7B</td><td>2023-10-09</td><td>3381</td><td>-623.00</td><td>55.86</td></tr>
  <tr><td>550</td><td>Maywell/Synatra-10.7B-v0.4</td><td>llama</td><td>10B</td><td>2023-12-27</td><td>3282</td><td>-532.85</td><td>65.48</td></tr>
  <tr><td>551</td><td>MaziyarPanahii/Llama-3-BB-Instruct-v0.4</td><td>mistral</td><td>7B</td><td>2023-10-29</td><td>4337</td><td>-574.59</td><td>60.55</td></tr>
  <tr><td>552</td><td>MaziyarPanahii/Llama-3-BB-Instruct-v0.8</td><td>mistral</td><td>7B</td><td>2023-10-29</td><td>8560</td><td>-582.37</td><td>62.06</td></tr>
  <tr><td>553</td><td>MaziyarPanahii/Llama-3-BB-Instruct-v0.4</td><td>llama-3</td><td>8B</td><td>2024-05-01</td><td>1407</td><td>-569.30</td><td>73.40</td></tr>
  <tr><td>554</td><td>MaziyarPanahii/Llama-3-BB-Instruct-v0.8</td><td>llama-3</td><td>8B</td><td>2024-05-01</td><td>7281</td><td>-576.95</td><td>73.17</td></tr>
  <tr><td>555</td><td>MaziyarPanahii/Mistral-7B-Instruct-v0.3</td><td>mistral</td><td>7B</td><td>2024-05-30</td><td>6241</td><td>-575.60</td><td>73.29</td></tr>
  <tr><td>556</td><td>MaziyarPanahii/Mistral-7B-Instruct-v0.3</td><td>mistral</td><td>7B</td><td>2024-05-22</td><td>5230</td><td>-549.67</td><td>65.21</td></tr>
  <tr><td>557</td><td>MaziyarPanahii/Mistral-7B-v0.3</td><td>mistral</td><td>7B</td><td>2024-05-22</td><td>5758</td><td>-531.44</td><td>60.40</td></tr>
  <tr><td>558</td><td>medalpac/medalpac-7b (Li et al., 2023d)</td><td>llama-1</td><td>1B</td><td>2023-03-29</td><td>8136</td><td>-628.95</td><td>48.45</td></tr>
  <tr><td>559</td><td>MediaTek-Research/hfz-7b-Instruct-v1.0 (Hsu et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-07-16</td><td>3065</td><td>-559.09</td><td>64.91</td></tr>
  <tr><td>560</td><td>meta-llama/Llama-2-13b-hf (Touvron et al., 2023b)</td><td>llama-2</td><td>13B</td><td>2023-07-13</td><td>120871</td><td>-530.82</td><td>55.69</td></tr>
  <tr><td>561</td><td>meta-llama/Llama-2-7b-chat-hf (Touvron et al., 2023b)</td><td>llama-2</td><td>6B</td><td>2023-07-13</td><td>140224</td><td>-566.54</td><td>50.74</td></tr>
  <tr><td>562</td><td>meta-llama/Llama-2-7b-hf (Touvron et al., 2023b)</td><td>llama-2</td><td>6B</td><td>2023-07-13</td><td>1294737</td><td>-549.87</td><td>50.97</td></tr>
  <tr><td>563</td><td>meta-llama/Meta-Llama-3-8B (AI@Meta, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-17</td><td>675850</td><td>-514.33</td><td>62.62</td></tr>
  <tr><td>564</td><td>meta-llama/Meta-Llama-3-8B-Instruct (AI@Meta, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-17</td><td>208011</td><td>-573.95</td><td>66.87</td></tr>
  <tr><td>565</td><td>meta-math/MathMath-Lemma-7B (Yu et al., 2024b; Azerbayev et al., 2024)</td><td>llama</td><td>7B</td><td>2023-11-11</td><td>13212</td><td>-612.53</td><td>59.19</td></tr>
  <tr><td>566</td><td>meta-math/MathMath-Lemma-7B (Jiang et al., 2023; Yu et al., 2024b)</td><td>mistral</td><td>7B</td><td>2023-11-14</td><td>3022</td><td>-532.53</td><td>57.87</td></tr>
  <tr><td>567</td><td>meta-math/MathMath-13B (Mitra et al., 2023)</td><td>llama</td><td>13B</td><td>2023-11-14</td><td>13616</td><td>-651.29</td><td>58.64</td></tr>
  <tr><td>568</td><td>microsoft/Orcas-2-7b (Mitra et al., 2023)</td><td>llama</td><td>7B</td><td>2023-11-14</td><td>11084</td><td>-684.33</td><td>54.55</td></tr>
  <tr><td>570</td><td>microsoft/ph1-1.5 (Li et al., 2023c)</td><td>phi</td><td>1B</td><td>2023-09-10</td><td>112476</td><td>-724.13</td><td>47.69</td></tr>
  <tr><td>571</td><td>microsoft/ph-2</td><td>phi</td><td>2B</td><td>2023-12-13</td><td>237268</td><td>-621.44</td><td>61.33</td></tr>
  <tr><td>572</td><td>microsoft/Phi-3-medium-128k-instruct</td><td>phi3</td><td>13B</td><td>2024-05-07</td><td>34054</td><td>-544.42</td><td>73.00</td></tr>
  <tr><td>573</td><td>microsoft/Phi-3-medium-4k-instruct</td><td>phi3</td><td>13B</td><td>2024-05-07</td><td>35987</td><td>-552.43</td><td>73.45</td></tr>
  <tr><td>574</td><td>migtsisera/Synthesia-1.7B-v1.3 (Mukherjee et al., 2023; Tissera, 2023)</td><td>mistral</td><td>7B</td><td>2023-09-28</td><td>3336</td><td>-541.46</td><td>59.34</td></tr>
  <tr><td>575</td><td>migtsisera/Synthesia-1.7B-v1.5</td><td>mistral</td><td>7B</td><td>2023-10-07</td><td>1228</td><td>-537.58</td><td>59.59</td></tr>
  <tr><td>576</td><td>migtsisera/Synthesia-7b-v3.0</td><td>mistral</td><td>7B</td><td>2023-12-08</td><td>1204</td><td>-531.17</td><td>61.99</td></tr>
  <tr><td>577</td><td>migtsisera/Tess-XS-v1.1</td><td>llama-3</td><td>8B</td><td>2023-11-22</td><td>1213</td><td>-546.97</td><td>59.39</td></tr>
  <tr><td>578</td><td>migtsisera/Tess-2.0-1.Llama-3-8B</td><td>llama-3</td><td>8B</td><td>2024-05-05</td><td>3314</td><td>-523.68</td><td>64.81</td></tr>
  <tr><td>579</td><td>migtsisera/Tess-7b-v1.4</td><td>llama-3</td><td>8B</td><td>2023-12-04</td><td>1234</td><td>-599.60</td><td>62.19</td></tr>
  <tr><td>580</td><td>Mihaiu/Metis-v4.3</td><td>mistral</td><td>7B</td><td>2023-12-16</td><td>1279</td><td>-567.64</td><td>65.44</td></tr>
  <tr><td>581</td><td>mindy-labs/mindy-7b-v2</td><td>mistral</td><td>7B</td><td>2023-12-14</td><td>1228</td><td>-558.38</td><td>72.11</td></tr>
  <tr><td>582</td><td>MistralAI/DPO_test02</td><td>mistral</td><td>7B</td><td>2023-10-30</td><td>3467</td><td>-547.35</td><td>62.23</td></tr>
  <tr><td>584</td><td>mistralai/Mistral-7B-Instruct-v0.1 (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-09-27</td><td>1370245</td><td>-593.06</td><td>54.96</td></tr>
  <tr><td>585</td><td>mistralai/Mistral-7B-Instruct-v0.2 (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-12-11</td><td>3565248</td><td>-574.91</td><td>65.71</td></tr>
  <tr><td>586</td><td>mistralai/Mistral-7B-v0.1 (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-09-20</td><td>1750089</td><td>-527.60</td><td>60.97</td></tr>
  <tr><td>587</td><td>mistralai/Mistral-7B-v0.3 (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2024-05-22</td><td>1443065</td><td>-531.44</td><td>60.28</td></tr>
  <tr><td>588</td><td>mistralai/community/Mistral-7B-v0.2</td><td>mistral</td><td>7B</td><td>2024-03-23</td><td>30074</td><td>-532.63</td><td>60.41</td></tr>
  <tr><td>589</td><td>milabone/AlphaMonarch-7B</td><td>mistral</td><td>7B</td><td>2024-02-14</td><td>12804</td><td>-593.46</td><td>75.99</td></tr>
  <tr><td>590</td><td>milabone/Beagle14-7B</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1986</td><td>-565.01</td><td>67.76</td></tr>
  <tr><td>591</td><td>milabone/ChimeraLlama-3-8B-v2</td><td>llama-3</td><td>8B</td><td>2024-04-22</td><td>2086</td><td>-565.81</td><td>69.69</td></tr>
  <tr><td>592</td><td>milabone/ChimeraLlama-3-8B-v3</td><td>llama-3</td><td>8B</td><td>2024-05-01</td><td>5718</td><td>-568.78</td><td>70.06</td></tr>
  <tr><td>593</td><td>milabone/Daredevil-SB-ablateder</td><td>llama</td><td>8B</td><td>2024-05-26</td><td>9010</td><td>-565.64</td><td>71.82</td></tr>
  <tr><td>594</td><td>milabone/GML-Mistral-merged-v1</td><td>mistral</td><td>8B</td><td>2023-12-27</td><td>1166</td><td>-1384.76</td><td>48.54</td></tr>
  <tr><td>595</td><td>milabone/Marcorol14-7B-sleep</td><td>mistral</td><td>7B</td><td>2023-12-29</td><td>3792</td><td>-554.34</td><td>73.01</td></tr>
  <tr><td>596</td><td>milabone/NeuralMarcorol14-7B</td><td>mistral</td><td>7B</td><td>2024-01-06</td><td>2015</td><td>-566.69</td><td>73.57</td></tr>
  <tr><td>597</td><td>milabone/NeuralMonarch-7B</td><td>mistral</td><td>7B</td><td>2024-01-24</td><td>13480</td><td>-591.00</td><td>76.15</td></tr>
  <tr><td>598</td><td>milabone/13B-1.7B-v1.1</td><td>mistral</td><td>7B</td><td>2023-12-15</td><td>3317</td><td>-597.75</td><td>60.40</td></tr>
  <tr><td>599</td><td>MonologAI/Myth_solar_10.7b_v3.0</td><td>llama</td><td>10B</td><td>2024-04-26</td><td>9855</td><td>-673.46</td><td>67.61</td></tr>
  <tr><td>600</td><td>monologAI/openinstruction-mistral-7b</td><td>mistral</td><td>7B</td><td>2023-11-20</td><td>1191</td><td>-537.83</td><td>63.64</td></tr>
  <tr><td>601</td><td>mosaicml/mpt-7b (Henry et al., 2020; Shoeybi et al., 2020; Dao et al., 2022; Press et al., 2022; Touvron et al., 2023a; Chen et al., 2023a; MosaicML NLP Team, 2023b)</td><td>mpt</td><td>7B</td><td>2023-05-04</td><td>88069</td><td>-647.13</td><td>45.39</td></tr>
  <tr><td>602</td><td>mosaicml/mpt-7b-chat (Henry et al., 2020; Press et al., 2022; Dao et al., 2022; MosaicML NLP Team, 2023b)</td><td>mpt</td><td>7B</td><td>2023-05-05</td><td>8599</td><td>-569.62</td><td>44.83</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>604</td><td>mosaiclm/mpt-7b-storywriter (Press et al., 2022; Dao et al., 2022; Chen et al., 2023a; MosaicML NLP Team, 2023b)</td><td>mpt</td><td>7B</td><td>2023-05-04</td><td>1920</td><td>-650.05</td><td>39.31</td></tr>
  <tr><td>605</td><td>mosaiclm/mpt-7b-8k (Henry et al., 2020; Shoeybi et al., 2020; Dao et al., 2022; Press et al., 2022; Touvron et al., 2023a; Chen et al., 2023a; MosaicML NLP Team, 2023b)</td><td>mpt</td><td>7B</td><td>2023-06-30</td><td>2106</td><td>-546.57</td><td>47.24</td></tr>
  <tr><td>606</td><td>mosaiclm/mpt-7b-8k-chat (Henry et al., 2020; Press et al., 2022; Dao et al., 2022; MosaicML NLP Team, 2023a)</td><td>mpt</td><td>7B</td><td>2023-06-22</td><td>1304</td><td>-586.09</td><td>47.78</td></tr>
  <tr><td>607</td><td>mpt-7b-8k-2-code-7b (Manuel Romero, 2023)</td><td>llama-2</td><td>7B</td><td>2023-07-26</td><td>1313</td><td>-572.92</td><td>49.95</td></tr>
  <tr><td>608</td><td>MTSAI/multi verse_model</td><td>mistral</td><td>7B</td><td>2023-04-07</td><td>6357</td><td>-598.96</td><td>76.74</td></tr>
  <tr><td>609</td><td>mwitderrick/open_llama_3b_code_instruct_0.1</td><td>llama</td><td>3B</td><td>2023-12-11</td><td>1252</td><td>-599.99</td><td>39.72</td></tr>
  <tr><td>610</td><td>NekoPunchBBB/Llama-2-13b-hf_Open-Platypus-QLoRA-multigpu</td><td>llama-2</td><td>13B</td><td>2023-09-15</td><td>1208</td><td>-536.60</td><td>54.40</td></tr>
  <tr><td>611</td><td>Neko-Institute-of-Science/metharme-7b</td><td>llama-1</td><td>6B</td><td>2023-04-30</td><td>1161</td><td>-562.36</td><td>47.48</td></tr>
  <tr><td>612</td><td>Neko-Institute-of-Science/pegmation-7b</td><td>llama-1</td><td>6B</td><td>2023-04-30</td><td>1188</td><td>-561.13</td><td>46.04</td></tr>
  <tr><td>613</td><td>overseerai/llama-3-Laminiatd-8B-v0.1</td><td>llama-3</td><td>8B</td><td>2023-02-11</td><td>1409</td><td>-545.54</td><td>48.55</td></tr>
  <tr><td>614</td><td>NewStar/Koss-7B-v0.2</td><td>mistral</td><td>7B</td><td>2023-12-21</td><td>1179</td><td>-532.64</td><td>51.78</td></tr>
  <tr><td>615</td><td>NewStar/Koss-7B-chat</td><td>llama</td><td>7B</td><td>2023-10-01</td><td>1180</td><td>-660.75</td><td>50.37</td></tr>
  <tr><td>616</td><td>NewStar/Starlight-7B (Clark et al., 2018; Zellers et al., 2019; Hendrycks et al., 2021a; Gao et al., 2021a; Lin et al., 2022a; Beeching et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-09-11</td><td>1180</td><td>-549.87</td><td>49.73</td></tr>
  <tr><td>617</td><td>NEXTNewChattingAI/shark_tank_7b_v2</td><td>mistral</td><td>7B</td><td>2023-12-23</td><td>1208</td><td>-618.14</td><td>66.54</td></tr>
  <tr><td>618</td><td>NEXTNewChattingAI/shark_tank_7b_v2</td><td>mistral</td><td>7B</td><td>2023-12-17</td><td>1224</td><td>-546.93</td><td>71.10</td></tr>
  <tr><td>619</td><td>Nexusflow/NexusRaven-V2-13B (Nexusflow.ai team, 2023; Rozière et al., 2024)</td><td>llama</td><td>13B</td><td>2023-12-04</td><td>3966</td><td>-593.55</td><td>48.21</td></tr>
  <tr><td>620</td><td>Nexusflow/Starling-LM-7B-beta (Ziegler et al., 2020; Zhu et al., 2023a)</td><td>mistral</td><td>7B</td><td>2023-04-19</td><td>4847</td><td>-559.22</td><td>69.88</td></tr>
  <tr><td>621</td><td>NickyNicky/Mistral-7B-OpenOrca-oast_top1_2023-08-25-v2 (Xiao et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1236</td><td>-560.11</td><td>61.65</td></tr>
  <tr><td>622</td><td>NickyNicky/Mistral-7B-OpenOrca-oast_top1_2023-08-25-v3 (Dao et al., 2023b; Xiao et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-10-13</td><td>1227</td><td>-568.73</td><td>61.26</td></tr>
  <tr><td>623</td><td>nlguyu/ColorShadow-7B</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1214</td><td>-552.87</td><td>68.34</td></tr>
  <tr><td>624</td><td>nlguyu/ColorShadow-7B-v2</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1214</td><td>-569.63</td><td>66.88</td></tr>
  <tr><td>625</td><td>nlguyu/ColorShadow-7B-v3</td><td>mistral</td><td>7B</td><td>2023-12-30</td><td>1222</td><td>-561.80</td><td>67.29</td></tr>
  <tr><td>626</td><td>Norquiall/llama-2-7b-claude-chat</td><td>llama-2</td><td>7B</td><td>2023-08-11</td><td>1221</td><td>-557.34</td><td>50.98</td></tr>
  <tr><td>627</td><td>Norquiall/llama-2-7b-claude-chat-rp</td><td>llama-2</td><td>7B</td><td>2023-08-14</td><td>1218</td><td>-557.09</td><td>51.25</td></tr>
  <tr><td>628</td><td>Norquiall/Mistral-7B-claude-instruct</td><td>mistral</td><td>7B</td><td>2023-09-28</td><td>1248</td><td>-534.51</td><td>59.27</td></tr>
  <tr><td>629</td><td>NousResearch/Codellama-13b-hf</td><td>llama</td><td>13B</td><td>2023-08-24</td><td>6382</td><td>-592.02</td><td>68.53</td></tr>
  <tr><td>630</td><td>NousResearch/Codellama-7b-hf</td><td>llama</td><td>7B</td><td>2023-08-24</td><td>9428</td><td>-597.05</td><td>80.81</td></tr>
  <tr><td>631</td><td>NousResearch/Hermes-2-Pro-Llama-3B ("Teknium" et al., 2024b)</td><td>llama-3</td><td>8B</td><td>2024-04-30</td><td>26797</td><td>-576.90</td><td>68.73</td></tr>
  <tr><td>632</td><td>NousResearch/Hermes-2-Pro-Mistral-7B ("interstellarinjia" et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-04-31</td><td>14586</td><td>-620.93</td><td>67.35</td></tr>
  <tr><td>633</td><td>NousResearch/Hermes-2-Theta-Llama-3B ("Teknium" et al., 2024a)</td><td>llama-3</td><td>8B</td><td>2024-05-05</td><td>12392</td><td>-565.47</td><td>69.21</td></tr>
  <tr><td>634</td><td>NousResearch/Meta-Llama-3B-Instruct (AI@Meta, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-18</td><td>104388</td><td>-573.95</td><td>67.10</td></tr>
  <tr><td>635</td><td>NousResearch/Nous-Hermes-Llama-13b</td><td>llama-2</td><td>13B</td><td>2023-07-20</td><td>39412</td><td>-586.34</td><td>57.35</td></tr>
  <tr><td>636</td><td>NousResearch/Nous-Hermes-Llama-7b</td><td>llama-2</td><td>7B</td><td>2023-07-20</td><td>12017</td><td>-560.97</td><td>58.87</td></tr>
  <tr><td>637</td><td>NousResearch/Nous-Hermes-2-Mistral-7B-DPO ("Teknium" et al., 2024c)</td><td>mistral</td><td>7B</td><td>2023-06-03</td><td>1536</td><td>-605.76</td><td>54.04</td></tr>
  <tr><td>638</td><td>NousResearch/Nous-Hermes-2-SOLAR-10.7B</td><td>llama</td><td>10B</td><td>2024-02-18</td><td>8725</td><td>-565.26</td><td>68.10</td></tr>
  <tr><td>640</td><td>NousResearch/Yarn-Mistral-7b-128k (Peng et al., 2023b)</td><td>mistral</td><td>7B</td><td>2023-10-31</td><td>20727</td><td>-532.22</td><td>59.42</td></tr>
  <tr><td>641</td><td>NousResearch/Yarn-Mistral-7b-64k (Peng et al., 2023b)</td><td>mistral</td><td>7B</td><td>2023-10-31</td><td>10368</td><td>-532.16</td><td>59.63</td></tr>
  <tr><td>642</td><td>NTQAIX/Ncode-CQ-7B-orpo (Hong et al., 2024)</td><td>qwen2</td><td>7B</td><td>2024-04-24</td><td>12700</td><td>-683.75</td><td>48.98</td></tr>
  <tr><td>643</td><td>nvidia/Llama-3-ChatQA-1.5-B8 (Liu et al., 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-28</td><td>10608</td><td>-515.86</td><td>56.71</td></tr>
  <tr><td>644</td><td>occulum/Helios-14-7B-slerp</td><td>mistral</td><td>7B</td><td>2023-12-21</td><td>1215</td><td>-538.31</td><td>42.25</td></tr>
  <tr><td>645</td><td>occulum/Helios-10-7B-v2</td><td>llama</td><td>7B</td><td>2023-12-21</td><td>1218</td><td>-1381.38</td><td>42.19</td></tr>
  <tr><td>646</td><td>OEVortex/EMO-2B</td><td>gemma</td><td>2B</td><td>2024-04-28</td><td>4137</td><td>-976.17</td><td>44.26</td></tr>
  <tr><td>648</td><td>oh-yeontack/llama-2-13B-LoRA-assemble</td><td>llama-2</td><td>13B</td><td>2023-09-13</td><td>3303</td><td>-568.38</td><td>57.91</td></tr>
  <tr><td>649</td><td>oh-yeontack/llama-2-7B-LoRA-assemble</td><td>llama-2</td><td>7B</td><td>2023-09-13</td><td>1337</td><td>-614.33</td><td>52.26</td></tr>
  <tr><td>650</td><td>openaccess-ai-collective/DP0OpenHermes-11B</td><td>mistral</td><td>10B</td><td>2023-12-03</td><td>1185</td><td>-591.33</td><td>66.83</td></tr>
  <tr><td>651</td><td>openaccess-ai-collective/tracklobe-7b (Mukherjee et al., 2023; Longpre et al., 2023; Lian et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-10-07</td><td>1198</td><td>-530.33</td><td>61.16</td></tr>
  <tr><td>652</td><td>openaccess-ai-collective/manticoe-13b</td><td>llama-1</td><td>13B</td><td>2023-05-17</td><td>1224</td><td>-553.22</td><td>54.86</td></tr>
  <tr><td>653</td><td>openaccess-ai-collective/manticoe-13b-chat-pyg</td><td>llama-1</td><td>13B</td><td>2023-05-22</td><td>2148</td><td>-553.85</td><td>54.13</td></tr>
  <tr><td>654</td><td>openaccess-ai-collective/minotaur-13b</td><td>llama-1</td><td>13B</td><td>2023-06-06</td><td>1202</td><td>-577.38</td><td>53.97</td></tr>
  <tr><td>655</td><td>openaccess-ai-collective/minotaur-13b-fixed</td><td>llama-1</td><td>13B</td><td>2023-06-12</td><td>1194</td><td>-568.52</td><td>55.19</td></tr>
  <tr><td>656</td><td>openaccess-ai-collective/mistral-7b-smilcaboros</td><td>mistral</td><td>7B</td><td>2023-10-13</td><td>1186</td><td>-569.20</td><td>61.18</td></tr>
  <tr><td>657</td><td>openaccess-ai-collective/wizard-mega-13b</td><td>llama-1</td><td>13B</td><td>2023-05-14</td><td>2168</td><td>-565.46</td><td>54.27</td></tr>
  <tr><td>658</td><td>OpenAssistant/llama2-13b-orca-8k-3119 (Mukherjee et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-07-24</td><td>1218</td><td>-518.71</td><td>58.88</td></tr>
  <tr><td>659</td><td>OpenAssistant/assst-sft-1-pythia-12b</td><td>llama-2</td><td>13B</td><td>2023-07-24</td><td>1270</td><td>-531.87</td><td>55.09</td></tr>
  <tr><td>660</td><td>OpenAssistant/assst-sft-4-pythia-12b-epoch-3.5</td><td>gpt_neox</td><td>12B</td><td>2023-03-09</td><td>22359</td><td>-615.20</td><td>40.77</td></tr>
  <tr><td>661</td><td>OpenAssistant/assst-sft-12-pythia-12b-epoch-3.5</td><td>gpt_neox</td><td>12B</td><td>2023-04-03</td><td>458718</td><td>-572.27</td><td>41.31</td></tr>
  <tr><td>662</td><td>OpenAssistant/assst-sft-12b-sft-8v7-epoch-3</td><td>gpt_neox</td><td>12B</td><td>2023-05-07</td><td>1323</td><td>-524.49</td><td>42.21</td></tr>
  <tr><td>663</td><td>OpenAssistant/stablechat-7b-sft-v7-epoch-3</td><td>gpt_neox</td><td>12B</td><td>2023-04-20</td><td>1215</td><td>-764.93</td><td>34.85</td></tr>
  <tr><td>664</td><td>openemb/UltrALM-13b-v2.0</td><td>llama</td><td>13B</td><td>2023-09-22</td><td>1174</td><td>-554.24</td><td>58.72</td></tr>
  <tr><td>665</td><td>OpenBuddy/openbuddy-llama2-13b-v9-bf16</td><td>llama</td><td>13B</td><td>2023-08-23</td><td>1207</td><td>-562.43</td><td>52.31</td></tr>
  <tr><td>666</td><td>OpenBuddy/openbuddy-llama2-13b-v11-bf16</td><td>llama</td><td>13B</td><td>2023-08-23</td><td>1199</td><td>-606.17</td><td>52.93</td></tr>
  <tr><td>667</td><td>OpenBuddy/openbuddy-llama2-13b-v11-bf16</td><td>llama</td><td>13B</td><td>2023-08-24</td><td>1210</td><td>-595.93</td><td>55.28</td></tr>
  <tr><td>668</td><td>OpenBuddy/openbuddy-llama2-13b-v1-pf16</td><td>llama-2</td><td>13B</td><td>2023-07-25</td><td>7339</td><td>-587.40</td><td>57.76</td></tr>
  <tr><td>669</td><td>OpenBuddy/openbuddy-llama3-8b-v21.1-8k</td><td>llama-3</td><td>8B</td><td>2024-04-20</td><td>2293</td><td>-578.14</td><td>65.31</td></tr>
  <tr><td>670</td><td>OpenBuddy/openbuddy-mistral-7b-v20.3-32k</td><td>mistral</td><td>7B</td><td>2023-04-27</td><td>2302</td><td>-642.78</td><td>62.73</td></tr>
  <tr><td>671</td><td>OpenBuddy/openbuddy-mistral-7b-v13</td><td>mistral</td><td>7B</td><td>2023-10-10</td><td>1335</td><td>-643.47</td><td>53.50</td></tr>
  <tr><td>672</td><td>OpenBuddy/openbuddy-mistral-7b-v13-base</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1197</td><td>-677.88</td><td>51.99</td></tr>
  <tr><td>673</td><td>OpenBuddy/openbuddy-mistral-7b-v13-base</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1217</td><td>-638.52</td><td>52.62</td></tr>
  <tr><td>674</td><td>OpenBuddy/openbuddy-llama3-8b-v10-bf16</td><td>llama-3</td><td>8B</td><td>2023-07-03</td><td>1209</td><td>-654.65</td><td>59.31</td></tr>
  <tr><td>675</td><td>OpenBuddy/openbuddy-llama3-8b-v10-bf16</td><td>llama</td><td>7B</td><td>2023-08-10</td><td>1206</td><td>-794.41</td><td>36.87</td></tr>
  <tr><td>676</td><td>OpenBuddy/openbuddy-llama3-7b-v12-bf16</td><td>llama</td><td>7B</td><td>2023-09-19</td><td>3153</td><td>-854.52</td><td>45.28</td></tr>
  <tr><td>677</td><td>OpenBuddy/openbuddy-zephyr-7b-v14.1</td><td>mistral</td><td>7B</td><td>2023-11-06</td><td>3392</td><td>-657.03</td><td>51.86</td></tr>
  <tr><td>678</td><td>openchat/openchat-3.5-0106 (Wang et al., 2024a; OpenAI et al., 2024)</td><td>mistral</td><td>7B</td><td>2024-01-07</td><td>26858</td><td>-554.42</td><td>69.30</td></tr>
  <tr><td>679</td><td>openchat/openchat-3.5-0106-gemma (Wang et al., 2024a)</td><td>gemma</td><td>8B</td><td>2024-03-09</td><td>7817</td><td>-937.52</td><td>69.42</td></tr>
  <tr><td>680</td><td>openchat/openchat-3.5-1210 (Wang et al., 2024a; OpenAI et al., 2024)</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>2123</td><td>-583.16</td><td>68.89</td></tr>
  <tr><td>681</td><td>openchat/openchat-3.6-8b-20240522 (Wang et al., 2024a)</td><td>llama-3</td><td>8B</td><td>2024-05-07</td><td>10464</td><td>-561.80</td><td>68.14</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>682</td><td>openlm-research/open_13b (Geng and Liu, 2023; Together Computer, 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-06-15</td><td>2373</td><td>-582.20</td><td>47.26</td></tr>
  <tr><td>683</td><td>openlm-research/open_13b_3b (Geng and Liu, 2023; Together Computer, 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>3B</td><td>2023-06-07</td><td>157405</td><td>-612.46</td><td>38.26</td></tr>
  <tr><td>684</td><td>openlm-research/open_13b_v2 (Geng and Liu, 2023; Together Computer, 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>3B</td><td>2023-07-16</td><td>21275</td><td>-578.08</td><td>40.28</td></tr>
  <tr><td>685</td><td>openlm-research/open_7b_1b (Geng and Liu, 2023; Together Computer, 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>7B</td><td>2023-06-07</td><td>44968</td><td>-610.20</td><td>42.31</td></tr>
  <tr><td>686</td><td>openlm-research/open_13b_7b_v2 (Geng and Liu, 2023; Together Computer, 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>7B</td><td>2023-07-06</td><td>2792</td><td>-556.22</td><td>44.26</td></tr>
  <tr><td>687</td><td>OpenModels/all-gemma-1.1-7b-t (Joshi et al., 2017; Zhao et al., 2018; Mishaylov et al., 2018; Zellers et al., 2019; Clark et al., 2019; Chelrier, 2019; Sakaguchi et al., 2020; Tamura et al., 2019; Bisk et al., 2019; Sap et al., 2019; Hendrycks et al., 2021; Cohen et al., 2021; Chen et al., 2021; Anstain et al., 2021; Parrish et al., 2022; Zhong et al., 2023; Srivastava et al., 2023; Gemini Team et al., 2024)</td><td>gemma</td><td>8B</td><td>2024-04-06</td><td>5128</td><td>-1355.60</td><td>59.78</td></tr>
  <tr><td>688</td><td>OpenPipe/mistral-ft-optimized-1218</td><td>mistral</td><td>7B</td><td>2023-12-17</td><td>1405</td><td>-546.73</td><td>71.94</td></tr>
  <tr><td>689</td><td>OpenPipe/mistral-ft-optimized-1227</td><td>mistral</td><td>7B</td><td>2023-12-27</td><td>6273</td><td>-564.28</td><td>70.54</td></tr>
  <tr><td>690</td><td>openhaigpt/openhaigpt-1.0-beta-13b-chat-hf</td><td>llama</td><td>13B</td><td>2023-12-18</td><td>1225</td><td>-612.51</td><td>50.45</td></tr>
  <tr><td>691</td><td>OpenOrca/LongOrca-13B-16k (Wang et al., 2023a; Mukherjee et al., 2023; Dale et al., 2023; Touvron et al., 2023b; Longpre et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-08-16</td><td>1217</td><td>-531.07</td><td>56.59</td></tr>
  <tr><td>692</td><td>OpenOrca/LongOrca-7B-16k (Wang et al., 2023a; Lian et al., 2023c; Mukherjee et al., 2023; Touvron et al., 2023b; Longpre et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-08-05</td><td>1223</td><td>-583.06</td><td>53.02</td></tr>
  <tr><td>693</td><td>OpenOrca/Mistral-7B (Mukherjee et al., 2023; Longpre et al., 2023; Lian et al., 2023d)</td><td>mistral</td><td>7B</td><td>2023-09-29</td><td>18070</td><td>-575.75</td><td>60.17</td></tr>
  <tr><td>694</td><td>OpenOrca/Mistral-7B-SlimOrca (Mukherjee et al., 2023; Longpre et al., 2023; Lian et al., 2023e)</td><td>mistral</td><td>7B</td><td>2023-10-08</td><td>3696</td><td>-569.42</td><td>60.37</td></tr>
  <tr><td>695</td><td>OpenOrca/Mistral-7B-SlimOrca (Mukherjee et al., 2023; Longpre et al., 2023; Lian et al., 2023f)</td><td>mistral</td><td>7B</td><td>2023-07-31</td><td>2172</td><td>-547.85</td><td>56.70</td></tr>
  <tr><td>696</td><td>OpenOrca/OpenOrca-Platypus-2-13B (Hu et al., 2022; Wang et al., 2023a; Lee et al., 2023; Mukherjee et al., 2023; Touvron et al., 2023b; Longpre et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-08-11</td><td>6853</td><td>-557.72</td><td>57.28</td></tr>
  <tr><td>697</td><td>OpenOrca/OpenOrca-Platypus-2-13B (Hu et al., 2022; Wang et al., 2023a; Lee et al., 2023; Mukherjee et al., 2023; Touvron et al., 2023b; Longpre et al., 2023)</td><td>llama-2</td><td>13B</td><td>2023-07-12</td><td>1240</td><td>-679.87</td><td>51.38</td></tr>
  <tr><td>698</td><td>OpenOrca/OpenOrca-Preview-1-13B (Mukherjee et al., 2023; Longpre et al., 2023; Lian et al., 2023a; Touvron et al., 2023a)</td><td>llama-1</td><td>8B</td><td>2024-04-23</td><td>9653</td><td>-564.93</td><td>66.18</td></tr>
  <tr><td>699</td><td>OpenOrca/OpenOrca-Preview-1-13B (Mukherjee et al., 2023; Touvron et al., 2023a)</td><td>llama-1</td><td>8B</td><td>2024-08-09</td><td>4717</td><td>-546.48</td><td>57.24</td></tr>
  <tr><td>700</td><td>pankajmathur/orca_mini_v3_13b (Mukherjee et al., 2023; Touvron et al., 2023b; Mathur, 2023b)</td><td>llama</td><td>7B</td><td>2023-08-07</td><td>2338</td><td>-572.18</td><td>53.47</td></tr>
  <tr><td>701</td><td>pankajmathur/orca_mini_v3_7b (Mukherjee et al., 2023; Touvron et al., 2023b; Mathur, 2023c; Touvron et al., 2023a)</td><td>llama</td><td>8B</td><td>2024-04-19</td><td>2305</td><td>-552.92</td><td>66.77</td></tr>
  <tr><td>702</td><td>PanopterKR/Wikiverse-Llama-3-KO-8B-Instruct (AI@Meta, 2024; AI@Wakanda)</td><td>llama-3</td><td>13B</td><td>2023-08-11</td><td>1176</td><td>-549.39</td><td>51.94</td></tr>
  <tr><td>703</td><td>pe-nlp/llama-2-13b-vicuna-wizard</td><td>llama-2</td><td>13B</td><td>2023-04-14</td><td>1168</td><td>-546.61</td><td>53.29</td></tr>
  <tr><td>704</td><td>pillowtalks/a-delta13b</td><td>llama-1</td><td>13B</td><td>2023-02-09</td><td>1351</td><td>-505.96</td><td>40.31</td></tr>
  <tr><td>705</td><td>Pirr/pirr/13b-deduped-green_devil</td><td>gpt_neox</td><td>7B</td><td>2023-12-11</td><td>1162</td><td>-549.18</td><td>69.18</td></tr>
  <tr><td>706</td><td>PistachioAI/Synatra-MCS-7B-107B-Mercer-V1.0</td><td>mistral</td><td>10B</td><td>2023-12-20</td><td>2264</td><td>-530.70</td><td>71.88</td></tr>
  <tr><td>707</td><td>princeton-nlp/Sheared-LLaMA-1.3B-ShardGPT (Xia et al., 2024)</td><td>llama-2</td><td>1B</td><td>2023-11-22</td><td>1737</td><td>-721.70</td><td>37.14</td></tr>
  <tr><td>708</td><td>princeton-nlp/Sheared-LLaMA-2.7B (Xia et al., 2024)</td><td>llama-2</td><td>2B</td><td>2023-10-10</td><td>2590</td><td>-610.40</td><td>40.84</td></tr>
  <tr><td>709</td><td>princeton-nlp/Sheared-LLaMA-2.7B-ShardGPT (Xia et al., 2024)</td><td>llama-2</td><td>2B</td><td>2023-11-22</td><td>1866</td><td>-681.47</td><td>42.11</td></tr>
  <tr><td>710</td><td>project-baize/baize-v2-13b (Xu et al., 2023b)</td><td>llama-1</td><td>13B</td><td>2023-05-23</td><td>2149</td><td>-578.69</td><td>52.94</td></tr>
  <tr><td>711</td><td>project-baize/baize-v2-7b (Xu et al., 2023b)</td><td>llama-1</td><td>7B</td><td>2023-05-23</td><td>1212</td><td>-638.05</td><td>46.72</td></tr>
  <tr><td>712</td><td>PygmalionAI/Almethane-1.3b</td><td>gpt_neox</td><td>1B</td><td>2023-06-02</td><td>1235</td><td>-541.34</td><td>35.04</td></tr>
  <tr><td>713</td><td>PygmalionAI/Almethane-1.3b</td><td>llama-2</td><td>1B</td><td>2023-05-23</td><td>2393</td><td>-555.46</td><td>56.48</td></tr>
  <tr><td>714</td><td>PygmalionAI/Alpygmalion-2-13b</td><td>gpt_neox</td><td>1B</td><td>2023-12-25</td><td>1519</td><td>-963.34</td><td>11.14</td></tr>
  <tr><td>715</td><td>PygmalionAI/Alpygmalion-2-13b</td><td>llama-2</td><td>1B</td><td>2023-09-04</td><td>2083</td><td>-540.27</td><td>55.12</td></tr>
  <tr><td>716</td><td>PygmalionAI/Alpygmalion-2-7b</td><td>llama-2</td><td>6B</td><td>2023-09-04</td><td>2063</td><td>-559.12</td><td>51.11</td></tr>
  <tr><td>717</td><td>PygmalionAI/Alpygmalion-2-7b</td><td>gpt_neo</td><td>2B</td><td>2023-01-05</td><td>1986</td><td>-559.32</td><td>33.98</td></tr>
  <tr><td>718</td><td>PygmalionAI/Alpygmalion-6b</td><td>gptj</td><td>6B</td><td>2023-01-07</td><td>4312</td><td>-555.36</td><td>38.47</td></tr>
  <tr><td>719</td><td>qnguyen3/Master-Yi-9B</td><td>llama</td><td>8B</td><td>2024-05-18</td><td>8810</td><td>-522.22</td><td>67.44</td></tr>
  <tr><td>720</td><td>quantumaihk/llama-2-7b-hf-guanco-1k</td><td>llama-2</td><td>7B</td><td>2023-08-06</td><td>1166</td><td>-605.59</td><td>50.13</td></tr>
  <tr><td>721</td><td>quantumaihk/QuantumLM-7B</td><td>llama</td><td>7B</td><td>2023-07-23</td><td>1192</td><td>-625.18</td><td>49.51</td></tr>
  <tr><td>722</td><td>quantumaihk/QuantumLM-7B</td><td>mistral</td><td>7B</td><td>2023-07-17</td><td>1171</td><td>-552.86</td><td>68.68</td></tr>
  <tr><td>723</td><td>Qwen/CodeQwen-5.7B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>1B</td><td>2024-04-15</td><td>77169</td><td>-691.05</td><td>43.26</td></tr>
  <tr><td>724</td><td>Qwen/Qwen1.5-1.8B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>1B</td><td>2024-01-22</td><td>137256</td><td>-610.58</td><td>46.55</td></tr>
  <tr><td>725</td><td>Qwen/Qwen1.5-1.8B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>3B</td><td>2024-01-22</td><td>6431</td><td>-553.86</td><td>57.05</td></tr>
  <tr><td>726</td><td>Qwen/Qwen1.5-4B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>3B</td><td>2024-01-30</td><td>5525</td><td>-601.85</td><td>46.79</td></tr>
  <tr><td>727</td><td>Qwen/Qwen1.5-4B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>7B</td><td>2024-01-22</td><td>122768</td><td>-533.48</td><td>61.76</td></tr>
  <tr><td>728</td><td>Qwen/Qwen1.5-7B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>7B</td><td>2024-01-30</td><td>2611</td><td>-606.15</td><td>50.15</td></tr>
  <tr><td>729</td><td>Qwen/Qwen1.5-7B-Chat (Bai et al., 2023)</td><td>qwen2</td><td>7B</td><td>2024-01-31</td><td>4610</td><td>-581.07</td><td>53.80</td></tr>
  <tr><td>730</td><td>Qwen/Qwen2-1.5B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2024-06-04</td><td>37173</td><td>-507.30</td><td>68.40</td></tr>
  <tr><td>731</td><td>Qwen/Qwen2-7B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2023-12-03</td><td>1226</td><td>-549.82</td><td>67.73</td></tr>
  <tr><td>732</td><td>Qwen/Qwen2-7B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2023-12-03</td><td>1237</td><td>-550.18</td><td>69.09</td></tr>
  <tr><td>733</td><td>Qwen/Qwen2-7B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2023-12-12</td><td>1233</td><td>-564.26</td><td>70.73</td></tr>
  <tr><td>734</td><td>Qwen/Qwen2-7B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2024-05-03</td><td>1246</td><td>-582.52</td><td>63.62</td></tr>
  <tr><td>735</td><td>Qwen/Qwen2-7B (Yang et al., 2024a)</td><td>qwen2</td><td>7B</td><td>2023-10-14</td><td>1166</td><td>-613.77</td><td>50.68</td></tr>
  <tr><td>736</td><td>rishraj/CatPFT-Base (Acharya, 2023)</td><td>mistral</td><td>7B</td><td>2023-07-31</td><td>3269</td><td>-594.93</td><td>32.14</td></tr>
  <tr><td>737</td><td>RoverX/llama-2-7b-small-shards-Samantha-V1-SFT</td><td>llama-2</td><td>7B</td><td>2024-01-17</td><td>601</td><td>-1107.39</td><td>28.28</td></tr>
  <tr><td>738</td><td>RuehleLabara/LogoS-7Bx2-MoE-13B-v0.2</td><td>mistral</td><td>12B</td><td>2024-01-21</td><td>3055</td><td>-589.76</td><td>77.15</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>746</td><td>ruslanmvai-medical-model-32bit</td><td>llama</td><td>8B</td><td>2024-05-13</td><td>2810</td><td>-557.43</td><td>67.67</td></tr>
  <tr><td>747</td><td>ruslanmvai/Medical-Llama-3B</td><td>llama-3</td><td>8B</td><td>2024-04-21</td><td>5457</td><td>-514.00</td><td>60.61</td></tr>
  <tr><td>749</td><td>rwkv/2go-bruins-v2.1 (Murias, 2023)</td><td>mistral</td><td>7B</td><td>2023-12-14</td><td>1193</td><td>-561.68</td><td>74.50</td></tr>
  <tr><td>749</td><td>rwkv/2go-bruins-v2.1.1 (Murias, 2023)</td><td>mistral</td><td>7B</td><td>2023-12-14</td><td>1205</td><td>-562.22</td><td>74.95</td></tr>
  <tr><td>750</td><td>RWKV/rwkv-raver-1b5</td><td>rwkv</td><td>1B</td><td>2023-05-04</td><td>1570</td><td>-526.30</td><td>33.56</td></tr>
  <tr><td>751</td><td>saberaI/zro1.5_3B</td><td>gpt_neox</td><td>2B</td><td>2023-12-25</td><td>1200</td><td>-695.87</td><td>38.02</td></tr>
  <tr><td>752</td><td>Salesforce/codgen-GB-multi (Nijkamp et al., 2023)</td><td>codegen</td><td>6B</td><td>2022-04-13</td><td>1651</td><td>-733.14</td><td>32.43</td></tr>
  <tr><td>753</td><td>Salesforce/codgen-GB-multi (Nijkamp et al., 2023)</td><td>codegen</td><td>6B</td><td>2022-04-13</td><td>1176</td><td>-478.78</td><td>40.00</td></tr>
  <tr><td>755</td><td>samir-fama/FernandoGPT-v1</td><td>mistral</td><td>7B</td><td>2023-12-28</td><td>1209</td><td>-551.87</td><td>87.87</td></tr>
  <tr><td>755</td><td>samir-fama/FernandoGPT-v1</td><td>mistral</td><td>7B</td><td>2023-12-28</td><td>1215</td><td>-552.06</td><td>73.11</td></tr>
  <tr><td>756</td><td>SanjiWatsuki/Kunoichi-DPO-v2-7B</td><td>mistral</td><td>7B</td><td>2024-01-13</td><td>1216</td><td>-586.90</td><td>72.40</td></tr>
  <tr><td>757</td><td>SanjiWatsuki/Kunoichi-7B</td><td>mistral</td><td>7B</td><td>2024-01-04</td><td>1230</td><td>-579.35</td><td>72.13</td></tr>
  <tr><td>758</td><td>SanjiWatsuki/Loyal-Macaroni-Maid-7B</td><td>mistral</td><td>7B</td><td>2023-12-24</td><td>1297</td><td>-575.31</td><td>71.68</td></tr>
  <tr><td>759</td><td>SanjiWatsuki/Silicon-Maid-7B</td><td>mistral</td><td>7B</td><td>2023-12-27</td><td>1578</td><td>-580.90</td><td>70.31</td></tr>
  <tr><td>760</td><td>SanjiWatsuki/Sonya-7B</td><td>mistral</td><td>7B</td><td>2023-12-31</td><td>5399</td><td>-594.55</td><td>68.48</td></tr>
  <tr><td>761</td><td>sarvamai/OpenBnT-Hi-v1.0-Base</td><td>llama-2</td><td>6B</td><td>2023-12-13</td><td>1766</td><td>-613.73</td><td>46.64</td></tr>
  <tr><td>762</td><td>sch10v/llama-1-typoon-v1.5-8b-instruct (Pipatanakul et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-12-19</td><td>1913</td><td>-561.57</td><td>57.87</td></tr>
  <tr><td>763</td><td>sch10v/llama-1-typoon-v1.5-8b-instruct (Pipatanakul et al., 2023)</td><td>llama-3</td><td>8B</td><td>2023-05-06</td><td>6088</td><td>-590.46</td><td>65.62</td></tr>
  <tr><td>764</td><td>sch10v/yphoos-7b (Pipatanakul et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-12-20</td><td>1908</td><td>-617.60</td><td>58.05</td></tr>
  <tr><td>765</td><td>SciPhi/SciPhi-Self-RAG-Mistral-7B-32k (Mukherjee et al., 2023; Longpre et al., 2023; Asai et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-10-27</td><td>1214</td><td>-664.82</td><td>56.46</td></tr>
  <tr><td>766</td><td>SealLMS/SealLMS-TB-7v2 (Zhang et al., 2023; Zheng et al., 2023; Kojima et al., 2023; Nguyen et al., 2024)</td><td>mistral</td><td>7B</td><td>2024-01-29</td><td>6294</td><td>-548.56</td><td>67.57</td></tr>
  <tr><td>767</td><td>SealLMS/SealLMS-TB-2.5 (Zhang et al., 2023; Nguyen et al., 2024)</td><td>gemma</td><td>8B</td><td>2024-04-03</td><td>13928</td><td>-565.35</td><td>69.07</td></tr>
  <tr><td>768</td><td>selfrag/selfrag_llama-2_7B (Asai et al., 2023)</td><td>llama-2</td><td>7B</td><td>2023-10-18</td><td>4388</td><td>-605.61</td><td>51.30</td></tr>
  <tr><td>769</td><td>senseable/West-Lake-TB-v2</td><td>mistral</td><td>7B</td><td>2024-01-22</td><td>1189</td><td>-594.85</td><td>74.68</td></tr>
  <tr><td>770</td><td>sethuyer/Medchat-Llama3-8B</td><td>llama-3</td><td>8B</td><td>2024-04-22</td><td>4542</td><td>-535.77</td><td>66.03</td></tr>
  <tr><td>771</td><td>Severian/ANIMA-Phi-Neptune-Mistral-7B</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1191</td><td>-613.73</td><td>55.54</td></tr>
  <tr><td>772</td><td>shadow1/BeguSake-7B</td><td>mistral</td><td>7B</td><td>2024-01-31</td><td>11842</td><td>-569.27</td><td>75.38</td></tr>
  <tr><td>773</td><td>shanchen/llama3-8B-super-biomed-chinese</td><td>llama-3</td><td>8B</td><td>2024-04-09</td><td>2790</td><td>-591.60</td><td>63.90</td></tr>
  <tr><td>774</td><td>shanchen/llama3-8B-super-med-chinese</td><td>llama-3</td><td>8B</td><td>2024-04-30</td><td>8028</td><td>-593.26</td><td>63.99</td></tr>
  <tr><td>775</td><td>shanchen/llama3-8B-super-med-262k</td><td>llama-3</td><td>8B</td><td>2024-04-30</td><td>2697</td><td>-599.35</td><td>53.65</td></tr>
  <tr><td>776</td><td>shenzi-wang/Llama3-8B-Chinese-Chat (Wang et al., 2024b)</td><td>llama-3</td><td>8B</td><td>2024-04-21</td><td>55718</td><td>-550.92</td><td>67.10</td></tr>
  <tr><td>777</td><td>shibing024/chinese-alpaca-plus-7b-hf (Ming, 2023)</td><td>llama-1</td><td>1B</td><td>2023-05-01</td><td>1527</td><td>-741.70</td><td>44.77</td></tr>
  <tr><td>778</td><td>shitsuhou123/nyllama-20000</td><td>llama</td><td>1B</td><td>2024-01-09</td><td>1195</td><td>-1213.36</td><td>27.95</td></tr>
  <tr><td>779</td><td>SJ-Donald/llama3-passthrough-chat</td><td>llama-3</td><td>11B</td><td>2024-05-17</td><td>2241</td><td>-607.16</td><td>60.15</td></tr>
  <tr><td>780</td><td>SJ-Donald/SJ-SOLAR-10-7B-DPO</td><td>llama</td><td>10B</td><td>2024-01-25</td><td>2306</td><td>-533.83</td><td>72.67</td></tr>
  <tr><td>781</td><td>SJ-Donald/SJ-SOLAR-10-7B-slep</td><td>llama</td><td>10B</td><td>2024-01-12</td><td>2296</td><td>-533.06</td><td>53.58</td></tr>
  <tr><td>782</td><td>SJ-Donald/SJ-BioMedAlpha-7B</td><td>mistral</td><td>7B</td><td>2024-04-01</td><td>4041</td><td>-604.14</td><td>57.44</td></tr>
  <tr><td>783</td><td>specklelab/BlekH-7B-Instruct-v1. (Levine et al., 2020; Granzioi et al., 2021; Ociepa et al., 2024c; Wang et al., 2024a; Ociepa et al., 2024b)</td><td>mistral</td><td>7B</td><td>2024-03-30</td><td>3573</td><td>-871.06</td><td>51.24</td></tr>
  <tr><td>784</td><td>specklelab/BlekH-7B-v1.0 (Ociepa et al., 2024c)</td><td>mistral</td><td>7B</td><td>2024-03-30</td><td>2822</td><td>-721.18</td><td>50.01</td></tr>
  <tr><td>785</td><td>stabilityai/japanese-stablelm-base-gamma-7b (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-10-16</td><td>2072</td><td>-581.58</td><td>52.59</td></tr>
  <tr><td>786</td><td>stabilityai/japanese-stablelm-instruct-gamma-7b (Jiang et al., 2023)</td><td>mistral</td><td>7B</td><td>2023-10-16</td><td>1412</td><td>-584.50</td><td>52.82</td></tr>
  <tr><td>787</td><td>stabilityai/teRehelia-7B (Wuherie et al., 2023; Touvron et al., 2023b)</td><td>llama</td><td>1B</td><td>2023-07-27</td><td>6171</td><td>-561.95</td><td>63.07</td></tr>
  <tr><td>788</td><td>stabilityai/teRehelia-7B (Wuherie et al., 2023; Touvron et al., 2023b)</td><td>llama</td><td>1B</td><td>2023-07-27</td><td>6691</td><td>-565.19</td><td>53.56</td></tr>
  <tr><td>789</td><td>stabilityai/stablelm-base-alpha-7b (Andonian et al., 2021)</td><td>gpt_neox</td><td>3B</td><td>2023-04-17</td><td>1728</td><td>-677.41</td><td>31.50</td></tr>
  <tr><td>790</td><td>stabilityai/stablelm-base-alpha-7b-v2 (Gao et al., 2020; Shazeeer, 2020; Rajbhandari et al., 2020; Su et al., 2023a; Li et al., 2023a; Tow, 2023)</td><td>stablelm_alpha</td><td>6B</td><td>2023-08-04</td><td>2209</td><td>-505.46</td><td>46.18</td></tr>
  <tr><td>791</td><td>stabilityai/stablelm-tuned-alpha-3b (Tao et al., 2023; Chiang et al., 2023; Anand et al., 2023)</td><td>gpt_neox</td><td>3B</td><td>2023-04-19</td><td>2145</td><td>-736.44</td><td>32.14</td></tr>
  <tr><td>793</td><td>stabilityai/stablelm-tuned-alpha-7b (Tao et al., 2023; Chiang et al., 2023; Anand et al., 2023)</td><td>gpt_neox</td><td>7B</td><td>2023-04-19</td><td>3765</td><td>-694.37</td><td>34.04</td></tr>
  <tr><td>794</td><td>stabilityai/stablelm-2p7h-1b (Zheng et al., 2023; Rafailov et al., 2024)</td><td>stablelm</td><td>2B</td><td>2023-11-21</td><td>8261</td><td>-777.78</td><td>53.43</td></tr>
  <tr><td>795</td><td>stabilityai/stablelm-2p7h-1.6b (Stability AI Language Team, 2024; Rafailov et al., 2024)</td><td>stablelm</td><td>12B</td><td>2024-04-04</td><td>4843</td><td>-588.55</td><td>68.38</td></tr>
  <tr><td>797</td><td>stabilityai/stablelm-2-1b-6h-chat (Stability AI Language Team, 2024; Rafailov et al., 2024)</td><td>stablelm</td><td>1B</td><td>2024-04-08</td><td>4156</td><td>-683.91</td><td>50.71</td></tr>
  <tr><td>798</td><td>stabilityai/stablelm-3b-6l1t (Bu et al., 2016; Zhang and Seunrich, 2019; Gao et al., 2020; Rajbhandari et al., 2020; Black et al., 2022; Su et al., 2023a; Tow et al., 2023; Li et al., 2023a; Touvron et al., 2023b)</td><td>stablelm</td><td>2B</td><td>2023-09-29</td><td>11112</td><td>-510.56</td><td>46.58</td></tr>
  <tr><td>799</td><td>stabilityai/table-code-3b (Rajbhandari et al., 2020; Black et al., 2022; Su et al., 2023a; Li et al., 2023a; Touvron et al., 2023b; Yu et al., 2024b; Azerbayev et al., 2024; Pinnaparaju et al., 2024)</td><td>stablelm</td><td>2B</td><td>2024-01-09</td><td>5670</td><td>-616.25</td><td>41.53</td></tr>
  <tr><td>800</td><td>starmpc/Aesclipsus-Llama2-13B (Kweon et al., 2024)</td><td>llama-2</td><td>13B</td><td>2023-09-19</td><td>1175</td><td>-645.45</td><td>50.25</td></tr>
  <tr><td>801</td><td>starmpc/Aesclipsus-Llama2-7B (Kweon et al., 2024)</td><td>llama-2</td><td>7B</td><td>2023-09-19</td><td>1231</td><td>-688.58</td><td>47.15</td></tr>
  <tr><td>802</td><td>starklm/zyr-7b-full-rop</td><td>mistral</td><td>7B</td><td>2024-05-18</td><td>2278</td><td>-548.89</td><td>53.16</td></tr>
  <tr><td>803</td><td>StudentLLM/Ipagasms-2-15b-QLoRA-merged (Chen et al., 2024a)</td><td>llama</td><td>13B</td><td>2023-09-02</td><td>1303</td><td>-533.28</td><td>54.31</td></tr>
  <tr><td>805</td><td>SuperAGI/SAM</td><td>mistral</td><td>7B</td><td>2023-12-22</td><td>1235</td><td>-550.64</td><td>59.30</td></tr>
  <tr><td>805</td><td>swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA (Basile et al., 2023; Al@Meta, 2024; Polignano et al., 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-29</td><td>5995</td><td>-635.92</td><td>75.12</td></tr>
  <tr><td>806</td><td>S4chz/zeprhyr-neural-chat-frankennergel1b</td><td>mistral</td><td>11B</td><td>2023-11-28</td><td>1181</td><td>-618.27</td><td>58.57</td></tr>
  <tr><td>807</td><td>TaylorAI/Flash-Llama-3B</td><td>llama-2</td><td>13B</td><td>2023-08-19</td><td>1181</td><td>-530.82</td><td>53.67</td></tr>
  <tr><td>808</td><td>TaylorAI/Flash-Llama-7B</td><td>llama</td><td>7B</td><td>2023-08-13</td><td>1177</td><td>-578.13</td><td>40.13</td></tr>
  <tr><td>809</td><td>TaylorAI/Flash-Llama-7B</td><td>llama</td><td>7B</td><td>2023-08-19</td><td>1184</td><td>-549.87</td><td>49.73</td></tr>
  <tr><td>810</td><td>TecZe/Bielic-SOLAR-LIKE-10-7B-Instruct-v0.1</td><td>mistral</td><td>10B</td><td>2024-04-10</td><td>1566</td><td>-854.79</td><td>53.50</td></tr>
  <tr><td>811</td><td>TehVenom/Dolly_Malion-6b</td><td>gptj</td><td>6B</td><td>2023-03-27</td><td>1201</td><td>-867.39</td><td>37.77</td></tr>
  <tr><td>812</td><td>TehVenom/Dolly_Shygnal-6b</td><td>gptj</td><td>6B</td><td>2023-03-29</td><td>1195</td><td>-491.76</td><td>39.89</td></tr>
  <tr><td>813</td><td>TehVenom/Dolly_Shygnal-6b-Dev_V8P2 (Wang and Komatsuzaki, 2021)</td><td>gptj</td><td>6B</td><td>2023-05-23</td><td>1197</td><td>-487.61</td><td>40.11</td></tr>
  <tr><td>814</td><td>TehVenom/GPT-J-Fry_PPO-6B-Dev-V8p4</td><td>gptj</td><td>6B</td><td>2023-03-26</td><td>1237</td><td>-495.79</td><td>46.60</td></tr>
  <tr><td>816</td><td>TehVenom/Metharain-13b-Merged</td><td>llama-1</td><td>13B</td><td>2023-05-18</td><td>1191</td><td>-561.32</td><td>54.15</td></tr>
  <tr><td>817</td><td>TehVenom/Moderator-Chat_GPT-JT-6b</td><td>gptj</td><td>6B</td><td>2023-03-19</td><td>1185</td><td>-502.43</td><td>42.17</td></tr>
  <tr><td>818</td><td>TehVenom/PPO_Pygway-V8p4_Dev-6b (Wang and Komatsuzaki, 2021)</td><td>gptj</td><td>6B</td><td>2023-03-17</td><td>1191</td><td>-489.49</td><td>39.85</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>819</td><td>TelVenom/PPO.Shymalonia-V8p4-Dev-6b</td><td>gptj</td><td>6B</td><td>2023-03-23</td><td>1188</td><td>-490.13</td><td>39.85</td></tr>
  <tr><td>820</td><td>TelVenom/PPO.Shymalonia-6b</td><td>gptj</td><td>6B</td><td>2023-03-23</td><td>1199</td><td>-489.16</td><td>39.35</td></tr>
  <tr><td>821</td><td>TelVenom/Pymalion-Vicuna-1.1-7b</td><td>llama-1</td><td>6B</td><td>2023-05-02</td><td>1246</td><td>-572.52</td><td>49.25</td></tr>
  <tr><td>822</td><td>TelVenom/Pymalion-13b-Merged</td><td>llama-1</td><td>13B</td><td>2023-05-18</td><td>1207</td><td>-588.35</td><td>48.49</td></tr>
  <tr><td>823</td><td>TelVenom/Pymalion_Alpacalora-7b</td><td>llama-1</td><td>7B</td><td>2023-04-30</td><td>1195</td><td>-605.84</td><td>46.49</td></tr>
  <tr><td>824</td><td>teilomielt/MiniMerlin-3B</td><td>llama-1</td><td>3B</td><td>2023-12-15</td><td>1168</td><td>-681.62</td><td>47.63</td></tr>
  <tr><td>825</td><td>teknuum/OpenHermes-13B</td><td>llama-2</td><td>13B</td><td>2023-09-06</td><td>1594</td><td>-543.02</td><td>55.24</td></tr>
  <tr><td>826</td><td>teknuum/OpenHermes-2.5-Mistral-7B</td><td>mistral</td><td>7B</td><td>2023-10-22</td><td>10099</td><td>-506.53</td><td>61.45</td></tr>
  <tr><td>827</td><td>Telege-LLM-13B-Chat-gemina-7b-finetuned-sft-Navarasa-2.0</td><td>gemina</td><td>8B</td><td>2023-01-17</td><td>2065</td><td>-659.29</td><td>57.74</td></tr>
  <tr><td>828</td><td>TencentARC/Llama-Pro-PreB</td><td>llama-2</td><td>8B</td><td>2024-01-05</td><td>1319</td><td>-557.58</td><td>51.67</td></tr>
  <tr><td>829</td><td>TencentARC/Llama-Pro-PreB-Instruct</td><td>llama-2</td><td>8B</td><td>2024-01-06</td><td>1423</td><td>-634.71</td><td>58.06</td></tr>
  <tr><td>830</td><td>TheBlock/airboros-13B-HF</td><td>llama-1</td><td>13B</td><td>2023-05-23</td><td>1214</td><td>-581.17</td><td>54.05</td></tr>
  <tr><td>831</td><td>TheBlock/airboros-7b-gpt4-fp16</td><td>llama-1</td><td>7B</td><td>2023-06-04</td><td>1208</td><td>-603.39</td><td>47.70</td></tr>
  <tr><td>832</td><td>TheBlock/CodeLlama-13B-Instruct-fp16</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>2193</td><td>-579.86</td><td>45.82</td></tr>
  <tr><td>833</td><td>TheBlock/CodeLlama-13B-Python-fp16</td><td>llama-2</td><td>13B</td><td>2023-08-24</td><td>2114</td><td>-587.86</td><td>37.52</td></tr>
  <tr><td>834</td><td>TheBlock/gpt4-alpaca-lora-13B-HF</td><td>llama-1</td><td>13B</td><td>2023-04-11</td><td>1181</td><td>-547.42</td><td>53.98</td></tr>
  <tr><td>835</td><td>TheBlock/guanaco-7B-HF</td><td>llama-1</td><td>7B</td><td>2023-05-25</td><td>1237</td><td>-581.59</td><td>51.54</td></tr>
  <tr><td>836</td><td>TheBlock/guanaco-13B-HF</td><td>llama-1</td><td>13B</td><td>2023-05-25</td><td>1267</td><td>-586.15</td><td>47.34</td></tr>
  <tr><td>837</td><td>TheBlock/koula-13B-HF</td><td>llama-1</td><td>13B</td><td>2023-04-07</td><td>2501</td><td>-594.23</td><td>51.16</td></tr>
  <tr><td>838</td><td>TheBlock/koula-7B-HF</td><td>llama-1</td><td>7B</td><td>2023-04-07</td><td>1238</td><td>-617.95</td><td>44.29</td></tr>
  <tr><td>839</td><td>TheBlock/Llama-2-13B-fp16</td><td>llama-2</td><td>13B</td><td>2023-07-18</td><td>6470</td><td>-530.82</td><td>53.67</td></tr>
  <tr><td>840</td><td>TheBlock/Nous-Hermes-13B-SuperHOT-8K-fp16</td><td>llama-1</td><td>13B</td><td>2023-06-26</td><td>1225</td><td>-633.76</td><td>52.18</td></tr>
  <tr><td>841</td><td>TheBlock/Planter-7B-fp16</td><td>llama-1</td><td>7B</td><td>2023-06-05</td><td>1219</td><td>-562.04</td><td>45.65</td></tr>
  <tr><td>842</td><td>TheBlock/Stable-vicuna-13B-HF (von Werra et al., 2023; Touvron et al., 2023a; Wang et al., 2023c; Taori et al., 2023; Chiang et al., 2023)</td><td>llama-1</td><td>13B</td><td>2024-02-28</td><td>1337</td><td>-590.21</td><td>51.64</td></tr>
  <tr><td>843</td><td>TheBlock/tulu-13B-fp16 (Chaudhary, 2023; Conover et al., 2023; Touvron et al., 2023a; Wang et al., 2023c; Köpf et al., 2023; Longpre et al., 2023; Peng et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-06-10</td><td>1231</td><td>-583.06</td><td>53.58</td></tr>
  <tr><td>844</td><td>TheBlock/tulu-7B-fp16 (Chaudhary, 2023; Conover et al., 2023; Touvron et al., 2023a; Wang et al., 2023c; Köpf et al., 2023; Longpre et al., 2023; Peng et al., 2023a)</td><td>llama-1</td><td>7B</td><td>2023-06-10</td><td>4079</td><td>-616.51</td><td>50.24</td></tr>
  <tr><td>845</td><td>TheBlock/UltraLM-13B-fp16 (Ding et al., 2023)</td><td>llama-1</td><td>13B</td><td>2023-06-29</td><td>1211</td><td>-567.38</td><td>54.62</td></tr>
  <tr><td>846</td><td>TheBlock/Vicuna-13B-CotFp16 (Lacoste et al., 2019)</td><td>llama-1</td><td>13B</td><td>2023-06-08</td><td>1219</td><td>-646.61</td><td>53.28</td></tr>
  <tr><td>847</td><td>TheBlock/WizardLM-13B-V1-1-SuperHOT-8K-fp16 (Xu et al., 2023a)</td><td>llama-1</td><td>13B</td><td>2023-07-07</td><td>1225</td><td>-626.40</td><td>53.16</td></tr>
  <tr><td>848</td><td>TheBlock/wizard-vicuna-13B-HF</td><td>llama-1</td><td>13B</td><td>2023-05-04</td><td>1219</td><td>-610.98</td><td>52.75</td></tr>
  <tr><td>849</td><td>TheBlock/Wizard-Vicuna-13B-Uncensored-HF</td><td>llama-1</td><td>13B</td><td>2023-05-13</td><td>1648</td><td>-579.62</td><td>54.14</td></tr>
  <tr><td>850</td><td>TheBlock/Wizard-Vicuna-7B-Uncensored-HF</td><td>llama-1</td><td>7B</td><td>2023-08-18</td><td>1976</td><td>-607.46</td><td>52.27</td></tr>
  <tr><td>851</td><td>TheTravelingEngineer/lama2-7b-chat-hf-guanaco</td><td>llama-2</td><td>7B</td><td>2023-08-01</td><td>1201</td><td>-559.39</td><td>50.38</td></tr>
  <tr><td>852</td><td>TheTravelingEngineer/lama2-7b-chat-hf-v2</td><td>llama-2</td><td>7B</td><td>2023-08-02</td><td>1209</td><td>-597.96</td><td>50.02</td></tr>
  <tr><td>853</td><td>TheTravelingEngineer/lama2-7b-chat-hf-v2</td><td>llama-2</td><td>7B</td><td>2023-08-08</td><td>1208</td><td>-549.87</td><td>49.73</td></tr>
  <tr><td>854</td><td>TheTravelingEngineer/lama2-7b-chat-hf-v3</td><td>llama-2</td><td>7B</td><td>2023-08-10</td><td>1203</td><td>-557.97</td><td>48.81</td></tr>
  <tr><td>855</td><td>TheTravelingEngineer/lama2-7b-chat-hf-v4</td><td>llama-2</td><td>7B</td><td>2023-08-10</td><td>1213</td><td>-549.47</td><td>49.78</td></tr>
  <tr><td>856</td><td>TheTravelingEngineer/lama2-7b-hf-guanaco</td><td>llama-2</td><td>7B</td><td>2023-07-25</td><td>1206</td><td>-554.41</td><td>50.12</td></tr>
  <tr><td>857</td><td>TigerResearch/TigerBot-7B-base</td><td>llama</td><td>7B</td><td>2023-08-19</td><td>1228</td><td>-587.11</td><td>47.93</td></tr>
  <tr><td>858</td><td>TigerResearch/TigerBot-7B-TH2-Plus (Yue et al., 2024b)</td><td>mistral</td><td>7B</td><td>2023-05-18</td><td>1017</td><td>-601.86</td><td>47.57</td></tr>
  <tr><td>859</td><td>TIGER-Lab/TIGERBot-7B-Plus (Yue et al., 2024b)</td><td>llama</td><td>7B</td><td>2023-05-06</td><td>1239</td><td>-595.71</td><td>67.49</td></tr>
  <tr><td>860</td><td>TIGER-Lab/TIGERBot-13B (Jiang et al., 2024)</td><td>llama</td><td>13B</td><td>2023-11-26</td><td>1429</td><td>-484.56</td><td>56.79</td></tr>
  <tr><td>861</td><td>tinae/falcon-rv-1b (Brown et al., 2020; Diao et al., 2022; Press et al., 2022; Penedo et al., 2023)</td><td>falcon</td><td>1B</td><td>2023-04-26</td><td>22921</td><td>-688.36</td><td>37.07</td></tr>
  <tr><td>862</td><td>tinae/falcon-7b (Shazeer, 2019; Brown et al., 2020; Gao et al., 2020; Diao et al., 2022; Su et al., 2023a; Almazrouei et al., 2023; Penedo et al., 2023; Su et al., 2023a; Almazrouei et al., 2023; Penedo et al., 2023)</td><td>falcon</td><td>7B</td><td>2023-04-24</td><td>104010</td><td>-549.44</td><td>44.17</td></tr>
  <tr><td>863</td><td>tinae/falcon-7b (Shazeer, 2019; Brown et al., 2020; Diao et al., 2022; Su et al., 2023a; Almazrouei et al., 2023; Penedo et al., 2023; Su et al., 2023a; Almazrouei et al., 2023; Penedo et al., 2023)</td><td>falcon</td><td>7B</td><td>2023-04-25</td><td>179952</td><td>-621.07</td><td>43.16</td></tr>
  <tr><td>864</td><td>tmpaloj/Mistral-7B-v1.0-flashback-v2</td><td>mistral</td><td>7B</td><td>2023-12-04</td><td>1275</td><td>-578.05</td><td>57.53</td></tr>
  <tr><td>865</td><td>TinyLlama/TinyLlama-1.1B-Chat-v0.6</td><td>llama</td><td>1B</td><td>2023-11-20</td><td>15745</td><td>-642.54</td><td>34.94</td></tr>
  <tr><td>866</td><td>TinyLlama/TinyLlama-1.1B-Chat-v1.0</td><td>llama</td><td>1B</td><td>2023-12-30</td><td>107800</td><td>-619.77</td><td>37.17</td></tr>
  <tr><td>867</td><td>TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T</td><td>llama</td><td>1B</td><td>2023-12-11</td><td>1365</td><td>-613.87</td><td>36.26</td></tr>
  <tr><td>868</td><td>TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T</td><td>llama</td><td>1B</td><td>2023-12-11</td><td>79832</td><td>-606.46</td><td>38.12</td></tr>
  <tr><td>869</td><td>TinyLlama/TinyLlama-1.1B-intermediate-step-955k-token-2T</td><td>llama</td><td>1B</td><td>2023-11-19</td><td>1119</td><td>-646.34</td><td>34.56</td></tr>
  <tr><td>870</td><td>togethercomputer/GPT-JT-6B-v0</td><td>gptj</td><td>6B</td><td>2023-12-22</td><td>1422</td><td>-484.70</td><td>44.05</td></tr>
  <tr><td>871</td><td>togethercomputer/GPT-JT-6B-v1 (Tay et al., 2022, 2023)</td><td>gptj</td><td>6B</td><td>2023-11-24</td><td>5765</td><td>-503.02</td><td>43.13</td></tr>
  <tr><td>872</td><td>togethercomputer/Llama-2-7B-32K</td><td>llama-2</td><td>7B</td><td>2023-07-26</td><td>8884</td><td>-530.20</td><td>47.07</td></tr>
  <tr><td>873</td><td>togethercomputer/Llama-2-7B-32K-Instruct (Liu et al., 2023b)</td><td>llama-2</td><td>7B</td><td>2023-08-08</td><td>5596</td><td>-564.83</td><td>50.02</td></tr>
  <tr><td>874</td><td>togethercomputer/Python-Chat-Base-/B</td><td>gpt_neox</td><td>7B</td><td>2023-06-22</td><td>7040</td><td>-522.45</td><td>39.81</td></tr>
  <tr><td>875</td><td>togethercomputer/RedPajama-INCITE-3B-3B-v1</td><td>gpt_neox</td><td>3B</td><td>2023-05-04</td><td>2653</td><td>-584.54</td><td>39.54</td></tr>
  <tr><td>876</td><td>togethercomputer/RedPajama-INCITE-Chat-3B-v1</td><td>gpt_neox</td><td>3B</td><td>2023-05-04</td><td>1616</td><td>-401.76</td><td>39.53</td></tr>
  <tr><td>877</td><td>togethercomputer/RedPajama-INCITE-3B-v1</td><td>gpt_neox</td><td>3B</td><td>2023-05-04</td><td>2014</td><td>-552.27</td><td>39.06</td></tr>
  <tr><td>878</td><td>togethercomputer/RedPajama-INCITE-7B-Base</td><td>gpt_neox</td><td>7B</td><td>2023-05-04</td><td>1487</td><td>-560.75</td><td>41.49</td></tr>
  <tr><td>879</td><td>togethercomputer/RedPajama-INCITE-7B-Chat</td><td>gpt_neox</td><td>7B</td><td>2023-05-04</td><td>1583</td><td>-931.82</td><td>39.37</td></tr>
  <tr><td>880</td><td>togethercomputer/RedPajama-INCITE-7B-Instruct</td><td>gpt_neox</td><td>7B</td><td>2023-05-05</td><td>1274</td><td>-525.00</td><td>42.38</td></tr>
  <tr><td>881</td><td>TomGrc/FusionNet</td><td>llama</td><td>10B</td><td>2023-12-31</td><td>1166</td><td>-561.26</td><td>74.43</td></tr>
  <tr><td>882</td><td>TomGrc/FusionNet_linear</td><td>llama</td><td>10B</td><td>2023-12-31</td><td>2171</td><td>-534.13</td><td>73.32</td></tr>
  <tr><td>883</td><td>totally-not-an-InfiniEverythingLM-13b-16k</td><td>llama-2</td><td>13B</td><td>2023-09-21</td><td>1162</td><td>-633.45</td><td>54.19</td></tr>
  <tr><td>884</td><td>totally-not-an-InfiniEverythingLM-13b-16k-v2</td><td>llama-2</td><td>13B</td><td>2023-12-12</td><td>1201</td><td>-557.34</td><td>72.50</td></tr>
  <tr><td>885</td><td>Toten5/Marconi-nerual-chat-7B-v2</td><td>mistral</td><td>7B</td><td>2024-04-27</td><td>3187</td><td>-569.88</td><td>52.23</td></tr>
  <tr><td>886</td><td>TsinghuaC31/Llama-3-8B-UltraMedical (Zhang et al., 2024c)</td><td>llama-3</td><td>8B</td><td>2024-01-03</td><td>1856</td><td>-582.06</td><td>49.11</td></tr>
  <tr><td>887</td><td>Unbabel/TowerBase-7B-v0.1 (Alves et al., 2024)</td><td>llama</td><td>7B</td><td>2024-01-04</td><td>2230</td><td>-590.52</td><td>53.29</td></tr>
  <tr><td>888</td><td>Undi95/Meta-Llama-3-8B-HF (AI Meta, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-18</td><td>11376</td><td>-514.33</td><td>58.35</td></tr>
  <tr><td>889</td><td>Undi95/Mistral-11B-v1</td><td>mistral</td><td>11B</td><td>2023-10-09</td><td>1196</td><td>-558.96</td><td>58.05</td></tr>
  <tr><td>890</td><td>Undi95/Mistral-11B-v2</td><td>mistral</td><td>11B</td><td>2023-10-09</td><td>1196</td><td>-542.18</td><td>58.11</td></tr>
  <tr><td>891</td><td>Undi95/SMLewd-2.0-v2-13B</td><td>llama</td><td>13B</td><td>2023-09-26</td><td>1187</td><td>-569.88</td><td>52.23</td></tr>
  <tr><td>892</td><td>Undi95/Mistral-11B-v2-13B</td><td>llama</td><td>11B</td><td>2023-09-16</td><td>1177</td><td>-551.09</td><td>57.75</td></tr>
  <tr><td>893</td><td>Undi95/Mistral-11B-v2-13B</td><td>llama</td><td>11B</td><td>2023-09-04</td><td>1172</td><td>-666.07</td><td>53.12</td></tr>
  <tr><td>894</td><td>Undi95/Mistral-11B-v2-13B</td><td>llama</td><td>11B</td><td>2023-09-26</td><td>1280</td><td>-571.03</td><td>56.37</td></tr>
  <tr><td>895</td><td>Undi95/Nous-Hermes-13B-Code</td><td>llama</td><td>13B</td><td>2023-09-02</td><td>1207</td><td>-602.47</td><td>55.93</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>897</td><td>Und95/OpenRP-13B</td><td>llama</td><td>13B</td><td>2023-09-11</td><td>1242</td><td>-536.49</td><td>56.57</td></tr>
  <tr><td>898</td><td>Und95/ReMM-SLERP-L2-13B</td><td>llama</td><td>13B</td><td>2023-09-04</td><td>1511</td><td>-585.33</td><td>56.03</td></tr>
  <tr><td>899</td><td>Und95/ReMM-v2.1-2-13B</td><td>llama</td><td>13B</td><td>2023-09-09</td><td>1207</td><td>-565.06</td><td>56.99</td></tr>
  <tr><td>900</td><td>Und95/ReMM-v2.1-2-13B</td><td>llama</td><td>13B</td><td>2023-09-12</td><td>1217</td><td>-564.83</td><td>56.71</td></tr>
  <tr><td>901</td><td>Und95/ReMM-v2.2-1-13B</td><td>llama</td><td>13B</td><td>2023-09-21</td><td>1384</td><td>-567.22</td><td>57.10</td></tr>
  <tr><td>902</td><td>Und95/UndMix-v1-13b</td><td>llama</td><td>13B</td><td>2023-08-31</td><td>1220</td><td>-649.19</td><td>55.50</td></tr>
  <tr><td>903</td><td>Und95/UndMix-v4-13B</td><td>llama</td><td>13B</td><td>2023-09-12</td><td>1212</td><td>-569.06</td><td>56.93</td></tr>
  <tr><td>904</td><td>Und95/UndMix-v1-12L-13B</td><td>llama</td><td>13B</td><td>2023-09-10</td><td>1210</td><td>-541.57</td><td>57.47</td></tr>
  <tr><td>905</td><td>Und95/MythicChronos-13B</td><td>llama</td><td>13B</td><td>2023-09-18</td><td>1203</td><td>-560.69</td><td>58.43</td></tr>
  <tr><td>906</td><td>und95/Ph1-3-medium-4k-instruct</td><td>mistral</td><td>7B</td><td>2024-02-18</td><td>4119</td><td>-532.63</td><td>60.34</td></tr>
  <tr><td>907</td><td>und95/Ph1-3-medium-4k-instruct</td><td>mistral</td><td>13B</td><td>2024-05-23</td><td>3994</td><td>-552.43</td><td>73.57</td></tr>
  <tr><td>908</td><td>und95/Ph1-3-mini-4k-instruct</td><td>mistral</td><td>3B</td><td>2024-04-29</td><td>12224</td><td>-575.74</td><td>69.86</td></tr>
  <tr><td>909</td><td>und95/tinyllama-chat</td><td>llama</td><td>1B</td><td>2024-02-14</td><td>5959</td><td>-619.71</td><td>37.24</td></tr>
  <tr><td>910</td><td>upstage/SOLAR-10.7B-Instruct-v1.0 (Kim et al., 2024b,a)</td><td>llama</td><td>10B</td><td>2023-12-12</td><td>67725</td><td>-557.67</td><td>74.20</td></tr>
  <tr><td>911</td><td>upstage/SOLAR-10.7B-v1.0 (Kim et al., 2024b)</td><td>llama</td><td>10B</td><td>2023-12-12</td><td>24478</td><td>-525.51</td><td>66.04</td></tr>
  <tr><td>912</td><td>ukuguy/speechless-code-mistral-7b-1.0</td><td>mistral</td><td>7B</td><td>2023-10-10</td><td>4393</td><td>-540.50</td><td>58.55</td></tr>
  <tr><td>913</td><td>ukuguy/speechless-llama2-hermes-orca-platypus-wizardlm-13b (Touvron et al., 2023b)</td><td>llama-2</td><td>13B</td><td>2023-09-01</td><td>2101</td><td>-609.53</td><td>57.52</td></tr>
  <tr><td>914</td><td>ukuguy/speechless-llama2-hermes-orca-platypus-13b (Touvron et al., 2023b)</td><td>llama-2</td><td>13B</td><td>2023-09-01</td><td>1368</td><td>-587.97</td><td>57.17</td></tr>
  <tr><td>915</td><td>ukuguy/speechless-zephyr-code-functionary-7b</td><td>mistral</td><td>7B</td><td>2024-01-23</td><td>4098</td><td>-529.18</td><td>63.93</td></tr>
  <tr><td>916</td><td>ukuguy/zephyr-7b-alpha-due-0.85</td><td>mistral</td><td>7B</td><td>2023-11-23</td><td>6141</td><td>-529.44</td><td>63.35</td></tr>
  <tr><td>917</td><td>vargartha/lama-3-merged-linear</td><td>llama-3</td><td>8B</td><td>2024-05-09</td><td>12676</td><td>-570.63</td><td>73.92</td></tr>
  <tr><td>918</td><td>VAGOSolutions/Llama-3-SauerkrattLM-8h-Instruct</td><td>llama-3</td><td>8B</td><td>2024-04-19</td><td>55400</td><td>-579.40</td><td>73.74</td></tr>
  <tr><td>919</td><td>VAGOSolutions/SauerkrattLM-Gemma-7b</td><td>gemma</td><td>8B</td><td>2024-02-27</td><td>5691</td><td>-561.27</td><td>67.83</td></tr>
  <tr><td>920</td><td>VAGOSolutions/SauerkrattLM-SOLAR-Instruct</td><td>llama</td><td>10B</td><td>2023-12-20</td><td>1175</td><td>-560.76</td><td>74.21</td></tr>
  <tr><td>921</td><td>VAGOSolutions/SauerkrattLM-7b-Hero</td><td>mistral</td><td>7B</td><td>2023-11-24</td><td>1226</td><td>-553.88</td><td>64.49</td></tr>
  <tr><td>922</td><td>varox34/Bio-Saul-Dolphin-Beagle-Breadcrumbs</td><td>mistral</td><td>7B</td><td>2024-05-01</td><td>2679</td><td>-610.01</td><td>48.72</td></tr>
  <tr><td>923</td><td>vihiborg101/lama-2-13b-chat-hf-phr_mental_therapy</td><td>llama-2</td><td>13B</td><td>2023-09-17</td><td>1269</td><td>-630.34</td><td>42.50</td></tr>
  <tr><td>924</td><td>viegalle/Carbon-Beagle-1 (Wortsman et al., 2022)</td><td>mistral</td><td>10B</td><td>2024-02-10</td><td>6669</td><td>-546.40</td><td>64.64</td></tr>
  <tr><td>925</td><td>viegalle/ConfigurableBeagle-1B (Gallego, 2024)</td><td>mistral</td><td>10B</td><td>2024-02-10</td><td>13887</td><td>-554.84</td><td>76.10</td></tr>
  <tr><td>926</td><td>viegalle/ConfigurableBeagle-1B (Gallego, 2024)</td><td>mistral</td><td>10B</td><td>2024-02-17</td><td>6045</td><td>-548.90</td><td>75.40</td></tr>
  <tr><td>927</td><td>viegalle/ConfigurableHermes-7B (Gallego, 2024)</td><td>mistral</td><td>7B</td><td>2024-02-17</td><td>6085</td><td>-572.68</td><td>68.89</td></tr>
  <tr><td>928</td><td>viegalle/ConfigurableSOLAR-10.7B (Gallego, 2024)</td><td>llama</td><td>10B</td><td>2023-04-30</td><td>5135</td><td>-559.06</td><td>73.94</td></tr>
  <tr><td>929</td><td>viegalle/Configurable-Hermes-2-Pro-Llama-3-8B (Gallego, 2024)</td><td>llama-3</td><td>8B</td><td>2024-05-02</td><td>10273</td><td>-580.31</td><td>70.10</td></tr>
  <tr><td>930</td><td>viegalle/Configurable-Llama-3-8B-v0.3 (Gallego, 2024)</td><td>llama-3</td><td>8B</td><td>2024-04-20</td><td>6030</td><td>-555.07</td><td>68.79</td></tr>
  <tr><td>931</td><td>viegalle/Configurable-Yi-1.5-9B-Chat (Gallego, 2024)</td><td>llama</td><td>8B</td><td>2024-05-12</td><td>6537</td><td>-633.77</td><td>70.50</td></tr>
  <tr><td>932</td><td>viegalle/Secommons-7B-v2-DPO-Chat</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>1191</td><td>-542.34</td><td>57.21</td></tr>
  <tr><td>933</td><td>vihang/dopeylept-1.1b-2v1</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>1181</td><td>-579.23</td><td>68.36</td></tr>
  <tr><td>934</td><td>vihang/dopeyleptsharedplats-1.3b-v1</td><td>llama-2</td><td>1B</td><td>2023-11-26</td><td>1191</td><td>-682.92</td><td>35.28</td></tr>
  <tr><td>935</td><td>vihang/dopeyleptsharedplats-2.7b-v1</td><td>llama-2</td><td>2B</td><td>2023-12-16</td><td>1173</td><td>-709.46</td><td>42.90</td></tr>
  <tr><td>936</td><td>vihang/neuralfalcon-1b-v1</td><td>falcon</td><td>1B</td><td>2023-12-17</td><td>1185</td><td>-895.18</td><td>29.72</td></tr>
  <tr><td>937</td><td>vihang/shearedplats-1.3b-v1</td><td>llama-2</td><td>1B</td><td>2023-11-16</td><td>1184</td><td>-706.16</td><td>35.97</td></tr>
  <tr><td>938</td><td>vihang/shearedplats-2.7b-v2</td><td>llama-2</td><td>2B</td><td>2023-11-18</td><td>2081</td><td>-641.47</td><td>41.61</td></tr>
  <tr><td>939</td><td>vihang/xmarrayplats-1.3b-v1</td><td>llama</td><td>3B</td><td>2023-09-14</td><td>1190</td><td>-600.04</td><td>40.00</td></tr>
  <tr><td>940</td><td>vihang/xmarrayplats-3.2b-v2</td><td>llama</td><td>3B</td><td>2023-09-14</td><td>1179</td><td>-597.07</td><td>40.29</td></tr>
  <tr><td>941</td><td>vikash06/llama-2.7b-small-model-new</td><td>llama-2</td><td>2B</td><td>2023-12-22</td><td>1174</td><td>-925.37</td><td>46.62</td></tr>
  <tr><td>942</td><td>vmajor/Orcas-1.3B-selfmerge-26B</td><td>llama</td><td>13B</td><td>2023-12-01</td><td>2041</td><td>-653.10</td><td>62.24</td></tr>
  <tr><td>943</td><td>vmajor/Orcas-1.3B-selfmerge-39B</td><td>llama</td><td>13B</td><td>2023-12-01</td><td>1208</td><td>-653.10</td><td>62.24</td></tr>
  <tr><td>944</td><td>VMware/open-llama-0.7T-7B-open-instruct-v1.1</td><td>llama-1</td><td>7B</td><td>2023-05-31</td><td>1161</td><td>-652.46</td><td>41.11</td></tr>
  <tr><td>945</td><td>VMware/open-llama-7b-open-instruct</td><td>llama-1</td><td>7B</td><td>2023-06-08</td><td>6470</td><td>-642.10</td><td>42.59</td></tr>
  <tr><td>946</td><td>Voicelab/truth-2-13B-academic</td><td>llama-2</td><td>13B</td><td>2023-09-18</td><td>2774</td><td>-568.46</td><td>57.94</td></tr>
  <tr><td>947</td><td>Voicelab/truth-2-13B-academic</td><td>llama-2</td><td>13B</td><td>2023-09-18</td><td>2774</td><td>-568.46</td><td>57.94</td></tr>
  <tr><td>948</td><td>vojack/Qwen-LLaMAfied-HFTok-7B-Chat</td><td>llama-2</td><td>7B</td><td>2023-08-09</td><td>1189</td><td>-839.96</td><td>50.64</td></tr>
  <tr><td>949</td><td>violet/llama-merged_dpo_7B</td><td>mistral</td><td>7B</td><td>2023-12-11</td><td>1225</td><td>-559.92</td><td>72.81</td></tr>
  <tr><td>950</td><td>violet/violet_merged_dpo_7B</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>1210</td><td>-592.71</td><td>70.26</td></tr>
  <tr><td>951</td><td>wang7776/Llama-2-7b-chat-hf-10-sparsity (Sun et al., 2024)</td><td>llama-2</td><td>6B</td><td>2023-12-11</td><td>1178</td><td>-660.87</td><td>52.48</td></tr>
  <tr><td>952</td><td>wang7776/Llama-2-7b-chat-hf-20-sparsity (Sun et al., 2024)</td><td>llama-2</td><td>6B</td><td>2023-12-13</td><td>1175</td><td>-668.34</td><td>52.01</td></tr>
  <tr><td>953</td><td>wang7776/Llama-2-7b-chat-hf-30-sparsity (Sun et al., 2024)</td><td>llama-2</td><td>6B</td><td>2023-12-14</td><td>1179</td><td>-676.50</td><td>51.80</td></tr>
  <tr><td>954</td><td>wenhuan/ALMA-7B-1.5-neural-xu (Xu et al., 2024a)</td><td>llama-2</td><td>7B</td><td>2023-12-11</td><td>1177</td><td>-597.45</td><td>48.85</td></tr>
  <tr><td>955</td><td>wenhuan/Faro-1.3B-13B-NEAT-2048-TEST</td><td>llama-2</td><td>13B</td><td>2023-09-18</td><td>1174</td><td>-558.02</td><td>53.62</td></tr>
  <tr><td>956</td><td>wenhuan/Faro/Yi-9B-DPO (OpenAI et al., 2024)</td><td>llama</td><td>8B</td><td>2023-04-27</td><td>6127</td><td>-594.07</td><td>66.37</td></tr>
  <tr><td>957</td><td>wenge-research/yi-7b</td><td>bloom</td><td>7B</td><td>2023-06-02</td><td>1192</td><td>-653.40</td><td>41.88</td></tr>
  <tr><td>958</td><td>wenge-research/yi-7b-llama2</td><td>llama-2</td><td>7B</td><td>2023-07-21</td><td>1195</td><td>-562.60</td><td>49.88</td></tr>
  <tr><td>959</td><td>Weyaxi/ChatAYT-Lora-Assamble-Marcoroni</td><td>llama</td><td>13B</td><td>2023-09-14</td><td>1203</td><td>-569.45</td><td>57.76</td></tr>
  <tr><td>960</td><td>Weyaxi/Dolphin2.1-OpenOrca-7B</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1221</td><td>-557.64</td><td>60.47</td></tr>
  <tr><td>961</td><td>Weyaxi/MetaMath-2-Seraph-7B</td><td>mistral</td><td>7B</td><td>2023-12-12</td><td>1217</td><td>-574.46</td><td>68.48</td></tr>
  <tr><td>962</td><td>Weyaxi/Luban-Marcoroni-13B-v3</td><td>llama</td><td>13B</td><td>2023-09-13</td><td>1214</td><td>-566.44</td><td>57.94</td></tr>
  <tr><td>963</td><td>Weyaxi/MetaMath-Chupacabra-7B-v2.01-Slerp</td><td>mistral</td><td>7B</td><td>2023-12-08</td><td>1210</td><td>-546.54</td><td>70.26</td></tr>
  <tr><td>964</td><td>Weyaxi/MetaMath-NeuralHermes-2.5-Mistral-7B-Ties</td><td>mistral</td><td>7B</td><td>2023-12-05</td><td>1205</td><td>-560.95</td><td>67.60</td></tr>
  <tr><td>965</td><td>Weyaxi/MetaMath-neural-chat-7b-v3-2-Slerp</td><td>mistral</td><td>7B</td><td>2023-12-05</td><td>1212</td><td>-564.36</td><td>67.03</td></tr>
  <tr><td>966</td><td>Weyaxi/MetaMath-neural-chat-7b-v3-2-Ties</td><td>mistral</td><td>7B</td><td>2023-12-08</td><td>1212</td><td>-544.26</td><td>69.79</td></tr>
  <tr><td>967</td><td>Weyaxi/MetaMath-neural-chat-7b-v3-3-Slerp</td><td>mistral</td><td>7B</td><td>2023-12-04</td><td>1212</td><td>-590.97</td><td>65.54</td></tr>
  <tr><td>968</td><td>Weyaxi/MetaMath-neural-chat-7b-v3-3-Ties</td><td>mistral</td><td>7B</td><td>2023-12-08</td><td>1217</td><td>-545.15</td><td>69.92</td></tr>
  <tr><td>969</td><td>Weyaxi/MetaMath-Tulpar-7b-Slerp</td><td>mistral</td><td>7B</td><td>2023-12-08</td><td>1217</td><td>-555.63</td><td>70.07</td></tr>
  <tr><td>970</td><td>Weyaxi/MetaMath-una-cybertron-v2-bf10-Ties</td><td>mistral</td><td>7B</td><td>2023-12-06</td><td>1218</td><td>-602.74</td><td>68.88</td></tr>
  <tr><td>971</td><td>Weyaxi/neural-chat-7b-v3-1-OpenHermes-2.5-TB</td><td>mistral</td><td>7B</td><td>2023-12-01</td><td>1205</td><td>-570.29</td><td>67.19</td></tr>
  <tr><td>972</td><td>Weyaxi/OpenHermes-2.5-neural-chat-7b-v3-1-TB</td><td>mistral</td><td>7B</td><td>2023-12-01</td><td>1230</td><td>-567.44</td><td>67.84</td></tr>
  <tr><td>973</td><td>Weyaxi/OpenHermes-2.5-neural-chat-7b-v3-2-TB</td><td>mistral</td><td>7B</td><td>2023-12-03</td><td>1217</td><td>-576.02</td><td>68.71</td></tr>
  <tr><td>974</td><td>Weyaxi/OpenHermes-2.5-neural-chat-7b-v3-2-TB</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1208</td><td>-564.12</td><td>64.97</td></tr>
  <tr><td>975</td><td>Weyaxi/Samantha-Nebula-7B</td><td>mistral</td><td>7B</td><td>2023-10-05</td><td>1166</td><td>-544.51</td><td>54.58</td></tr>
  <tr><td>976</td><td>Weyaxi/SauerkrattLM-UNA-SOLAR-Instruct</td><td>llama</td><td>10B</td><td>2023-12-21</td><td>1225</td><td>-561.98</td><td>74.26</td></tr>
</table>
<table>
  <tr>
    <th>ID</th>
    <th>Model Name</th>
    <th>Model Type</th>
    <th>Size</th>
    <th>Date</th>
    <th>DLS</th>
    <th>\( \hat{\ell}_i \)</th>
    <th>Task</th>
  </tr>
  <tr><td>981</td><td>Weyaxi/Seraph-openchat-3.5-1210-Slerp</td><td>mistral</td><td>7B</td><td>2023-12-27</td><td>1210</td><td>-567.06</td><td>70.89</td></tr>
  <tr><td>982</td><td>Weyaxi/Seraph-7B</td><td>mistral</td><td>7B</td><td>2023-12-11</td><td>1214</td><td>-546.73</td><td>71.86</td></tr>
  <tr><td>983</td><td>Weyaxi/SlimOpenOrca-Mistral-7B</td><td>mistral</td><td>7B</td><td>2023-10-11</td><td>1220</td><td>-587.26</td><td>60.84</td></tr>
  <tr><td>984</td><td>WhiteRabbitNeo/WhiteRabbitNeo-13B-v1</td><td>llama-2</td><td>13B</td><td>2023-12-17</td><td>2090</td><td>-615.73</td><td>49.11</td></tr>
  <tr><td>985</td><td>winglian/Llama-2-3b-hf</td><td>llama-2</td><td>3B</td><td>2023-09-19</td><td>1565</td><td>-1376.16</td><td>29.53</td></tr>
  <tr><td>986</td><td>winglian/Llama-2-4b</td><td>llama-2</td><td>4B</td><td>2023-09-19</td><td>1193</td><td>-676.31</td><td>34.23</td></tr>
  <tr><td>987</td><td>w01ss/blade-1b</td><td>gpt_neox</td><td>1B</td><td>2023-07-17</td><td>1204</td><td>-929.29</td><td>32.59</td></tr>
  <tr><td>988</td><td>XDAN-AIR/DAN-LI-Chat-RL-v1</td><td>mistral</td><td>7B</td><td>2023-12-21</td><td>1178</td><td>-574.97</td><td>68.38</td></tr>
  <tr><td>989</td><td>Xwin-LM/xwin-LM-13B-v1 (Xwin-LM Team, 2023)</td><td>llama-2</td><td>13B</td><td>2023-10-15</td><td>1401</td><td>-559.13</td><td>32.29</td></tr>
  <tr><td>990</td><td>Xwin-LM/xwin-LM-7B-v1 (Xwin-LM Team, 2023)</td><td>llama-2</td><td>7B</td><td>2023-09-15</td><td>1474</td><td>-592.93</td><td>52.08</td></tr>
  <tr><td>991</td><td>yam-peleg/Hebrew-Mistral-7B</td><td>mistral</td><td>7B</td><td>2024-04-26</td><td>5993</td><td>-625.28</td><td>58.76</td></tr>
  <tr><td>992</td><td>yanolja/Bookworm-10.7B-v0.4-DPO (Mukherjee et al., 2023; Lian et al., 2023; Cui et al., 2024)</td><td>llama</td><td>10B</td><td>2024-01-18</td><td>2239</td><td>-586.77</td><td>66.59</td></tr>
  <tr><td>993</td><td>yanolja/EEVE-Korean-Instruct-10.8B-v1.0 (Mukherjee et al., 2023; Lian et al., 2023; Kim et al., 2024c; Cui et al., 2024)</td><td>llama</td><td>10B</td><td>2024-02-22</td><td>13241</td><td>-555.03</td><td>66.48</td></tr>
  <tr><td>994</td><td>ycen214/test_Llama2_7b_small_tuning_v1</td><td>llama-2</td><td>7B</td><td>2023-10-02</td><td>3289</td><td>-1402.04</td><td>28.56</td></tr>
  <tr><td>995</td><td>ycen214/test_Llama2_7b_7b</td><td>llama-2</td><td>7B</td><td>2023-10-02</td><td>3285</td><td>-1405.43</td><td>29.99</td></tr>
  <tr><td>996</td><td>ycen214/test_Llama2_7b_7b</td><td>llama-2</td><td>7B</td><td>2023-09-30</td><td>3289</td><td>-549.87</td><td>49.73</td></tr>
  <tr><td>997</td><td>yphy3/open_Llama_7b_v2_med_instruct</td><td>llama-1</td><td>7B</td><td>2023-07-09</td><td>1197</td><td>-561.62</td><td>46.24</td></tr>
  <tr><td>998</td><td>yhyu13/chimera-inst-chat-13b-hf</td><td>llama-1</td><td>13B</td><td>2023-05-11</td><td>1295</td><td>-582.39</td><td>52.86</td></tr>
  <tr><td>999</td><td>yhyu13/LMCocktail-10.7B-v1 (Xiao et al., 2023)</td><td>llama-2</td><td>10B</td><td>2023-12-20</td><td>3332</td><td>-556.38</td><td>74.06</td></tr>
  <tr><td>1000</td><td>Yukang/Llama-2-7b-longlora-32k-f (Chen et al., 2024b)</td><td>llama-2</td><td>7B</td><td>2023-09-12</td><td>2423</td><td>-1387.05</td><td>29.20</td></tr>
  <tr><td>1001</td><td>Yukang/LongApaca-13B (Chen et al., 2023b, 2024b)</td><td>llama</td><td>13B</td><td>2023-10-07</td><td>1880</td><td>-873.14</td><td>71.74</td></tr>
  <tr><td>1002</td><td>Yukang/LongApaca-13B (Chen et al., 2023b, 2024b)</td><td>llama</td><td>6B</td><td>2023-10-07</td><td>2773</td><td>-794.46</td><td>39.26</td></tr>
  <tr><td>1003</td><td>yulan-team/Yulan-Chat-2-13b-fp16 (Yulan-Team, 2023)</td><td>llama</td><td>13B</td><td>2023-08-04</td><td>1165</td><td>-645.56</td><td>57.01</td></tr>
  <tr><td>1004</td><td>yunconglong/DARE_TIES_13B (Yadav et al., 2023a; Yu et al., 2024a)</td><td>mixtral</td><td>12B</td><td>2024-01-30</td><td>7029</td><td>-611.23</td><td>77.10</td></tr>
  <tr><td>1005</td><td>yunconglong/MoE_13B_DPO</td><td>mixtral</td><td>12B</td><td>2024-01-28</td><td>3939</td><td>-603.88</td><td>77.05</td></tr>
  <tr><td>1006</td><td>yunconglong/Truthful_DPO_TomGrc_FusionNet_7Bx2_MoE_13B</td><td>mixtral</td><td>12B</td><td>2024-01-21</td><td>7965</td><td>-598.84</td><td>77.44</td></tr>
  <tr><td>1007</td><td>01-ai/Yi-1.5-6B (01. Al et al., 2025)</td><td>llama</td><td>6B</td><td>2024-05-11</td><td>5102</td><td>-525.57</td><td>61.57</td></tr>
  <tr><td>1008</td><td>01-ai/Yi-1.5-6B-Chat (01. Al et al., 2025)</td><td>llama</td><td>6B</td><td>2024-05-11</td><td>1944</td><td>-644.86</td><td>66.17</td></tr>
  <tr><td>1009</td><td>01-ai/Yi-1.5-9B-Chat (01. Al et al., 2025)</td><td>llama</td><td>8B</td><td>2024-05-11</td><td>20252</td><td>-513.39</td><td>67.73</td></tr>
  <tr><td>1010</td><td>01-ai/Yi-1.5-9B-Chat (01. Al et al., 2025)</td><td>llama</td><td>8B</td><td>2024-05-10</td><td>20108</td><td>-626.90</td><td>69.56</td></tr>
  <tr><td>1011</td><td>01-ai/Yi-1.5-9B-Chat-16K (01. Al et al., 2025)</td><td>llama</td><td>8B</td><td>2024-05-15</td><td>14262</td><td>-601.39</td><td>66.98</td></tr>
  <tr><td>1012</td><td>01-ai/Yi-1.5-9B-32K (01. Al et al., 2025)</td><td>llama</td><td>8B</td><td>2024-05-15</td><td>9287</td><td>-510.38</td><td>55.22</td></tr>
  <tr><td>1013</td><td>01-ai/Yi-6B (Zhang et al., 2024b; Yue et al., 2024a; 01. Al et al., 2025)</td><td>llama</td><td>6B</td><td>2023-11-01</td><td>6997</td><td>-515.62</td><td>54.02</td></tr>
  <tr><td>1014</td><td>01-ai/Yi-6B-200K (Zhang et al., 2024b; Yue et al., 2024a; 01. Al et al., 2025)</td><td>llama</td><td>6B</td><td>2023-11-06</td><td>8370</td><td>-529.12</td><td>56.76</td></tr>
  <tr><td>1015</td><td>01-ai/Yi-9B-200K (Zhang et al., 2024b; Yue et al., 2024a; 01. Al et al., 2025)</td><td>llama</td><td>8B</td><td>2024-03-15</td><td>8915</td><td>-517.82</td><td>61.94</td></tr>
  <tr><td>1016</td><td>42dot42dot_LLM-PLM-1.3B (42dot Inc., 2023)</td><td>llama</td><td>1B</td><td>2023-09-04</td><td>1268</td><td>-600.03</td><td>35.70</td></tr>
  <tr><td>1017</td><td>42dot42dot_LLM-SFT-1.3B (42dot Inc., 2023)</td><td>llama</td><td>1B</td><td>2023-09-04</td><td>1453</td><td>-605.31</td><td>36.61</td></tr>
  <tr><td>1018</td><td>922-CA/moshika-ddlc-7b-v1</td><td>llama-2</td><td>7B</td><td>2023-10-13</td><td>1191</td><td>-633.19</td><td>50.49</td></tr>
</table>

Reviewer paper summaries: The paper takes on a task of embedding actual language models in an n-dimensional space, n being the number of prepared texts and values being their log likelihood according to the model. This gives us a representation of models mutual distances and differences, as well as model clusters, and indications of their performance trough said metrics.

1. In this work, the authors attempt to map out and compare autoregressive language models at scale by using log-likelihood vectors on pre-defined text set model coordinates.
2. The paper also considers the log-likelihood vector produced above from the pre-trained models as feature vectors and tries to predict model performance from model coordinates.
3. Through a set of well defined experiments on over 1000 models, the paper confirms the effectiveness of log likelihood vectors in structuring model relationships and predicting benchmark performance.

This paper introduces a method for characterizing language models by computing log-likelihood vectors over a predefined and diverse test set. These vectors serve as representations for each model, enabling pairwise comparison and similarity analysis. By applying this technique to a large collection of more than 1,000 language models, the authors construct a meaningful representation space, which they analyze through a variety of empirical experiments.

The approach is grounded in a straightforward idea: using model behavior (likelihood) on set of texts to produce compact, comparable descriptors of model behavior. The resulting representations are intuitive, computationally tractable, and visually interpretable. One of the highlights of the paper is a compelling 2D map showing how these 1,000 models distribute in the derived similarity space, revealing patterns aligned with model origin and performance characteristics.

The authors conduct multiple analyses using this representation. They predict model performance on benchmark tasks, interpret clusters of models with shared provenance, and evaluate how their distance metric corresponds to performance-based similarities. These analyses reinforce the value of the proposed method and showcase its utility in making sense of a fast-growing model ecosystem.

While the method is relatively simple, the execution is thorough, and the paper does well to acknowledge and discuss its limitations. The work provides a helpful lens for understanding LLM diversity and evolution and would be even more impactful if accompanied by code and tooling for ongoing use by the broader community.
Summary of strengths: The paper is very well written and detailed. It deals with a rather popular subject among language model creators and users community. The visualization is quite lovely and revealing.

1. Compared to competing research which constructs phylogenetic trees based on model generated text, the paper described method does not actually generate text (incurring the additional cost of pairwise distance comparisons) and instead computes generation probabilities on a predefined text corpus, thus making it more computationally efficient at scale.
2. The authors present a well formed list of limitations, which helps put the research in context, highlighting the strengths and limitations of the work, and also motivates areas of research where this paper is applicable.

- Simple, clear methodology: The core idea—representing models via log-likelihoods on a predefined test—is straightforward and easy to understand.
- Scalable and computationally feasible: The method is lightweight enough to be applied to a large number of models without prohibitive cost.
- Effective visualization
- Solid analyses. The authors explore various applications of the representation, such as predicting model behavior and identifying related models. 
applicability and its current constraints.
Summary of weaknesses: The paper is very lengthy including the appendices.

1. As the authors mention in the limitations section, similarity between models in the training and test sets might make it easier to predict task performance. Perhaps the authors can instead use representative models from each family of models (maybe using hierarchical clustering they did in the paper) and do the same correlation analysis. Also, it would be good to understand what kind of error margins users might expect if they use the method to estimate downstream task performance.

These are relatively minor weaknesses.
- Assumes availability of model log-likelihoods: Running each model isn't a huge cost, but it is a cost. Behavior needs to be measured across different datasets.
- Data selection. A key aspect of this measure is the selection of the data to use for measuring log likelihood. The effects of this selection, especially for domain specialized models, isn't explored. However, the results still seem compelling.
Comments, suggestions and typos: /

1. Although the predictive behaviour of log likelihood vectors in predicting benchmark scores is interesting theoretically, it is not clear how this might be used, because the prediction model essentially requires us to have access to a fully trained model, and at that point we might as well run evaluations. Would it be possible for the authors to better motivate the importance of these findings?

None
Overall assessment scores: 4

3.5

4
Metareview score: 4.5
Metareview: This paper proposed a simple yet effective way to represent each language model. Concretely, on a predefined dataset of N pieces of text, they get the sequence level log likelihood $l_i$ of each text from the model, and use $[l_1, ..., l_N]$ to represent the model.

They theoretically proved, when using log-likelihood vector, the Euclidean distance between any two model vectors is an approximation of computing the KL divergence between these two models.

They show that the vector can be used to predict individual model's downstream performance or predicted merged model's performance.
Summary of reasons to publish: 1. Analyzing and understanding the relationship among LLMs of different families is an interesting and under-studied topic.

2. The proposed method is intuitive, effective, and generalizable.

3. The paper is nicely written and have a comprehensive amount of details.

4. Visuals nicely show the cluster of models and the regression analysis also demonstrates it's promising to use these feature vectors to predict model performance.