<figure><img src="image_1.png" /></figure>

HAL
open science

# PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri

► To cite this version:

Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri. PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks. ICLR 2025, 2025, Singapore, Singapore. hal-04880495v2

HAL Id: hal-04880495

https://inria.hal.science/hal-04880495v2

Submitted on 8 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PHYLOLM: INFERRING THE PHYLOGENY OF LARGE LANGUAGE MODELS AND PREDICTING THEIR PERFORMANCES IN BENCHMARKS

**Nicolas Yax**
LNC2, INSERM, Paris, France
DEC, ENS, PSL, Paris, France
Flowers AI & CogSci Lab
Centre Inria de L'Université de Bordeaux
nicolas.yax@ens.psl.eu

**Stefano Palminteri***
LNC2, INSERM, Paris, France
DEC, ENS, PSL, Paris, France

* equal contribution

**Pierre-Yves Oudeyer***
Flowers AI & CogSci Lab
Centre Inria de L'Université de Bordeaux

## ABSTRACT

This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information.

## 1 INTRODUCTION

The Large Language Models (LLMs) landscape is vast and rapidly expanding, comprising both private and open-access models. Each day a few hundreds of new language models are created on the huggingface hub among which most will not be benchmarked, and a small minority are transparent about the training details. Evaluating these models presents challenges due to the sheer volume and the complexity of assessing their true capabilities. The evaluation methods used today mostly rely on a multitude of benchmarks, each focused on specific domains like reasoning or question-answering (Chollet, 2019; Hendrycks et al., 2021; Srivastava et al., 2023). However, tracking LLMs evolution and progress using benchmarks presents inherent limitations, including the fact that they are rather domain-specific, meaning that to get a full picture of a model's capabilities one has to run multiple costly tests that are prone to contamination (Chang et al., 2023; Deng et al., 2023; Liang et al., 2023). Moreover, the opacity of algorithmic and training data specifications in many models, adds further complexity and constraints to monitor progress in LLMs (Liao & Vaughan, 2023).

Our approach stems from the observation that most of the newly released models are not created ex-nihilo (from scratch). In fact, they rather inherit features from existing ones, such as training data or initial weights. We reasoned that we could therefore think about LLMs development as

code: https://github.com/hrl-team/PhyloLM
notebook: https://colab.research.google.com/drive/1agNE52eUevgdJ3KL3ytv5Y9JB
bfJRYqd?usp=copy
an "evolutionary" process and therefore study their relationships and functional properties with conceptual and quantitative tools borrowed from genetics.

In the field of Phylogeny, algorithms have been developed that reconstruct evolutionary trees to understand evolutionary relationships among species (Takezaki & Nei, 1996). The idea of applying these methods, initially developed for biology, to cultural artefacts is not new. Previous studies yielded particularly useful insights into the evolution of popular tales, languages, or craft assemblages (Atkinson et al., 2008; Dawkins, 1976; d'Huy, 2013; Gray et al., 2010; Tehrani & d'Huy, 2017; Tehrani & Collard, 2009). We hypothesize here that LLMs, which are a new kind of cultural artefact (in the sense that they are productions of humans that convey information about the culture of their creators and users), may also be studied using similar tools.

Thus, we here apply a conceptually similar approach to LLMs and, by doing so, we make several contributions. In a first contribution, we introduce an algorithm, PhyloLM, inspired by a simplified phylogenetic model, but specifically tailored for Large Language Models (LLMs), which core idea is to consider that generated tokens are to contexts what alleles are to genes in genetics. This analogy makes it possible to apply algorithms from the genetics framework to LLMs and to generate distance matrices and dendrograms. In addition to presenting the underlying theory, we also explore the hyperparameters of our algorithm to strike a balance between precision and computational efficiency.

In our second contribution, we analyze the resulting phylogenetic trees ("dendrograms") and confirm that PhyloLM is capable of correctly retrieving known relationships between LLMs and overall correctly capturing models families and sub-families. Our analysis primarily focuses on open-access model families (Llama (Touvron et al., 2023a;b), Mistral (Jiang et al., 2023), Bloom (BigScienceWorkshop et al., 2023), Pythia (Biderman et al., 2023), Falcon (Almazrouei et al., 2023), OPT (Zhang et al., 2022), Qwen (Bai et al., 2023) and Gemma (Team et al., 2024b) families), where ground truth information is available, but also provides insights into fine-tuning relationships for proprietary models (GPT-3 (Brown et al., 2020), 3.5 (Ouyang et al., 2022), 4 (OpenAI et al., 2023), Claude, Palm (Chowdhery et al., 2022) and Gemini models (Team et al., 2024a)). Finally, in our third contribution, we examine whether phylogenetic distance can also be used to predict performance in several benchmarks, thus showing that the utility of PhyloLM extends to the assessment of functional properties of LLMs.

To sum up, our study illustrates the potential of leveraging methods from genetics to understand how models evolve, shedding light on their relationships and functional capabilities in a relatively cost-efficient manner, even in the absence of transparent training information and also without direct access to the model.

$$S(P_1, P_2) = \frac{\sum_{g \in G} \sum_{a \in A_g} P_1(a|g) P_2(a|g)}{\sqrt{(\sum_{g \in G} \sum_{a \in A_g} P_1(a|g)^2)(\sum_{g \in G} \sum_{a \in A_g} P_2(a|g)^2)}} \quad (1)$$

Equation 1: Similarity computation with $P_1$ and $P_2$ two populations seen as probability distribution of alleles a given a gene g estimated empirically in the selected populations. G is the set of genes considered and $A_G$ the set of possible alleles for this gene and matrix S is the similarity matrix (bounded in [0,1]). In genetics people tend to use a distance matrix D to plot dendrograms derived from the similarity matrix with this formula $D(P_1, P_2) = -\log(S(P_1, P_2))$ (Takezaki & Nei, 1996). Seen from the autoregressive LLM framework, 'populations' are LLMs, 'genes' are contexts and 'alleles' are the different tokens in the vocabulary : $P(a|g) = LLM(t|c)$

# 2 METHODS

## 2.1 TRANSLATING PHYLOGENETIC ALGORITHMS TO LLMs

In the current landscape, LLMs predominantly operate on an autoregressive basis, wherein they learn the conditional probability denoted as $LLM(t|c)$. Here, LLM represents the probability learned by the language model, t signifies a token, and c denotes the context in which to sample token t. Transposing genetic methods to LLMs involves establishing analogies for the elements of the phylogenetic analysis, namely genes, alleles, and populations. Drawing a parallel with the notation for
<figure><img src="image_3.png" /><figcaption>**Figure 1: Analogy between running human genetic studies and LLMs genetic studies.** The first stage consists in selecting genes (for both humans and LLMs). Then alleles are collected for each individual in the population and will be used to compare the populations (either populations of humans or LLMs seen as populations). Finally these data go through the Nei distance computation (Takezaki & Nei, 1996) that returns a distance matrix that can then be turned into dendrograms using the NJ algorithm (Saitou & Nei, 1987) in the same way for both humans and LLMs.</figcaption></figure>

populations in the Nei genetic distance (see Equation 1) (Takezaki & Nei, 1996), $P_1(a|g)$ with a an allele and $g$ a gene, we propose that LLMs play the role of populations (i.e., the set of the individuals belonging to a given population); contexts (or "prompts") are aligned to genes (i.e., portions of DNA); finally, tokens align with alleles (i.e., variants in the DNA sequence).

To substantiate this analogy, consider that, in the realm of genetics, populations are conceptualized as probability distributions of DNA, represented by $P(a|g)$, where *a* stands for specific alleles at gene locations. Gene-specific alleles are then considered to be probabilistically drawn from the abstract statistical construct that is the population, akin to context-specific tokens are probabilistically generated from Large Language Models, expressed as $LLM(t|c)$ ($t$ being a token likely to follow text $c$). The generated text can therefore be seen as a thread of DNA, comprised of tokens (alleles) sampled in contexts (genes) according to a probability distribution defined by the LLM.

To elucidate this crucial point, consider a tokenized text sequence: 'I' '_like' '_choco' 'late'. This sequence can be analogous to a DNA thread represented as 'I_like_chocolate'. Breaking it down, the allele I corresponds to the gene $\epsilon$ (empty text), _like aligns with the gene I, _choco associates with the gene I_like, and late is linked to the gene I_like_choco. Now, consider another individual represented by 'I' '_prefer' '_ice' '_cream'. These two individuals share exactly two genes: $\epsilon$, for which they possess the same allele I, and the gene I, for which they have distinct alleles (_like and _prefer). They do not share any further genes, as their prefixes diverge beyond this point.

The algorithm is illustrated in Figure 1. The initial step involves collecting model outputs to contexts (genes). Given a set of LLMs, a set of 'genes', and the specified number of individuals in each population (i.e., the number of times the model is queried on each gene refered to as the number of probes) as *N*, the models are queried for a single token *N* times. This process generates the matrix *P*, which serves as an approximation of $P(a|g)$, the proportion of the population with allele *a* to gene *g*. Subsequently, based on this approximation, the similarity matrix *S* is computed using the Nei genetic distance formula (Takezaki & Nei, 1996) depicted in Equation 1. The pseudo code of PhyloLM can be found in Algorithm 1 in Appendix C.

## 2.2 CHOICE OF THE SET OF GENES

The implementation of phylogenetic algorithms requires selecting specific genes that show enough evolutionary changes among the species studied to differentiate them, while still retaining enough
similarity to trace relationships between closely related species (Grünwald et al., 2017). If these
genes mutate too quickly and are completely altered between similar species, they will not provide
useful information about their evolution. Conversely, if they are too stable and show no changes
across the species being considered, they are also not informative. These genes must strike a balance
between stability and variation among the species studied.

That is why we need to carefully select genes (i.e., prompt contexts) that could show a moderate
variance between LLMs. Recent LLM development focused a lot on instruction tuning, reasoning and
coding (Brown et al., 2020; Chiang et al., 2023; OpenAI et al., 2023; Taori et al., 2023). Selecting
contexts on these topics might show a relevant variance between generations of language models as
well as finetuning refinements that improved these models on these specific topics.

Furthermore, contexts ('genes') which are very likely to belong to the training data of these LLMs
can suffer from contamination issues and generate very low variance<fnref n="1" />. To obviate this issue, we used
contexts (or a 'gene' set) taken from recent test benchmarks because, in principle, LLMs shouldn't
be trained on this data. To further assess the robustness of our approach and study the impact of the
choice of the set of 'genes', we took our contexts from two different test sets: open-web-math (Paster
et al., 2023) and MBXP (Athiwaratkun et al., 2023). They address different capabilities of LLMs:
reasoning and coding, respectively, which are very relevant in recent LLM-related research and are
therefore likely to deliver useful results.

The exact selection of contexts from the benchmarks consisted of randomly and uniformly selecting
lines from the solution column in the datasets and truncating the text to leave it open for LLMs
to complete the sentence. To decide the length of the contexts we need once more to think about
making 'genes' show a moderate completion variance. If the context is only a few tokens long it may
not be informative enough for LLMs to understand the topic of the context (that is relevant for the
recent evolution of language models as discussed above) but also to follow the logics of the text that
would constrain the generation. On the other hand, making it hundreds of tokens long will induce
additional costs without necessarily improving the variability balance. That is why we decided to
truncate randomly and uniformly between the 20th and 100th characters in each text (5 to 30 tokens
approximately). 'Gene' examples are shown in Appendix A. More details about the impact of the
gene length can be found in appendix K.1.

## 2.3 SELECTION OF THE HYPER-PARAMETERS OF THE DISTANCE MATRICES

We devised two complementary analyses to estimate the right hyperparameters to run PhyloLM. The
hyperparameters are the 'gene' set, the number of probes and sampling parameters from the LLM
(see Appendix B). Testing the gene set is more difficult as testing thousands of different combinations
of genes would come at a very expensive cost. Thus we limited ourselves at 2 parameters of the
gene set : the topic (math and code in this paper) and the size of the gene set G. In this section we
will investigate the impact of G and N in the math gene set, the results for the code gene set are in
Appendix D.

First we investigate how *G* and *N* affect the variability of the distance matrix, namely how much the
similarity matrix changes between different estimations. We focus on similarity matrices (the matrix
*S* in Equation 1) instead of distance matrices at this point as they are bounded in [0,1] making them a
lot easier to plot and compare. Then, once the variance is controlled, what combination of *G* and *N*
approximate reasonably well a very high *G*' and *N*' distance matrix.

To assess the impact of the number of contexts ('genes') *G* and the number of probes/individuals *N* for each dataset, the algorithm was executed across a range of gene set sizes *G* (varying between 16 and 256 genes per run) and individuals *N* (ranging from 8 to 128) building similarity matrices. This optimization process, aimed at testing the best values for the algorithm hyperparameters, is particularly computationally expensive. Therefore it was only run on the 5 smallest OPENAI models (ada,babbage,chicken,news-001,news-002 and babbage-002), in order to minimize the costs. Thus similarity matrices in this section are 5 × 5 making it an estimate of what could be a larger distance matrix at a very low cost.

<footnote n="1">¹To understand this point, imagine using "May the force be with you" as context. All models will complete this sentence with "you", thus making impossible establishing distance matrices between them</footnote>
To investigate the variability of PhyloLM for different combination of hyperparameters, we composed 8 sets of genes of size *G*, each with different genes. Each set of gene is probed *N* times to build a similarity matrix *S*<sub>*G*,*N*,*i*</sub>, *i* ∈ [0, 7] representing the independent set of genes of size *G* used to generate the matrix with *N* probes. A variance computation over this set of matrices is finally performed yielding a matrix *V* containing the variance of each distance between 2 models : *V*<sub>*G*,*N*</sub><sup>2</sup> = <sup>1</sup>&frasl;<sub>8</sub> &sum;<sub>*i*</sub> (<i>S</i><sub>*G*,*N*,*i*</sub> - (<sup>1</sup>&frasl;<sub>8</sub> &sum;<sub>*j*</sub> <i>S</i><sub>*G*,*N*,*j*</sub>))<sup>2</sup>. The square operator is applied coefficient by coefficient. The final variability score is the mean value of the coefficients in the matrix *v*<sub>*G*,*N*</sub> = &mu; (<sqrt><i>V</i><sub>*G*,*N*</sub><sup>2</sup></sqrt>).

Then we investigated the impact of these hyperparameters when trying to approximate a high precision
matrix. For this purpose, we compute the variance around a very expensive distance matrix $S_{G',N'}$
with $G' = 2048$ and $N' = 128$. The gene set for the high precision matrix is independent from the
lower size set of genes used to estimate it. The formula to compute this variance around the high
precision matrix is $V'_{G,N'} = \frac{1}{8} \sum_i (S_{G,N,i} - S_{G',N'})^2$. The final metric is the mean value in the
matrix $v'_{G,N} = \mu(\sqrt{V'_{G,N}^2})$.

2.4 ALIGNMENT OF THE RESULTS ACROSS DIFFERENT TOKENIZATION

In situations where models do not share the same tokenizer, comparing only the first alleles generated
can pose challenges. For instance, if the context is "*The president of the US is Joe*," and one model
could complete with "*Biden*" in one token while another could complete with "*Bi*" "*den*" in two
tokens they would be considered as different alleles while both LLM meant the same completion.

To mitigate this issue of tokenizer alignment, a proxy approach was employed by only using the first 4 characters of the generated text instead of the first token. Practically, each model was instructed to generate at least 4 tokens (tokens are at least 1 character long) and the comparison focused on the first 4 characters in the concatenation of these tokens. In the previous example, the word "Biden" generated in one token or in two ("Bi" and "den") would have been considered as the same response, because the first 4 characters ("Bide") constitute the same 'allele', despite having being tokenized differently. In other words, we are retokenizing the text using a tokenizer with a vocabulary of words that are 4 characters long, and then comparing the first token of the generated text with this new tokenization scheme. An example of the results of such a proxy approach is presented in Appendix A. Further details about why 4 characters is efficient are discussed in Appendix K.2.

2.5 VISUALIZATION OF THE RESULTS

From a distance matrix obtained by the phylogenetic algorithm it is usual to plot dendrograms
representing a possible evolution between the entities in the distance matrix. For this purpose many
different algorithms exist and we chose the Neighbour Joining (NJ) technique (Saitou & Nei, 1987)
for its simplicity, efficiency and being a common choice in genetics. We plotted unrooted trees as
they are easier to make figures that fit in a paper and are more adapted to LLM evolution than rooted
ones. The analysis of the resulting dendrograms also allowed us to validate the capability of our
algorithm to predict actual relationship between LLMs in cases where the ground truth is known.

## 2.6 PREDICT BENCHMARK SCORES FROM GENETIC DISTANCE

We explored whether genetic distance can predict model performance by using logistic regression to estimate benchmark scores of large language models based on their similarity to other models. Due to the high dimensionality of the similarity matrix, we reduced the input dimensions to 15 using Independent Component Analysis (ICA), resulting in 15 parameters to learn from approximately 100 data points per fit. We then applied a sigmoid function to the output to scale the predictions between 0 and 1, corresponding to benchmark scores ranging from 0% to 100%. Since benchmark scores can be highly correlated within a family of models, we employed a leave-one-family-out method (see Figure 5 (a)). This involved training the regressor on all families but one and testing it on the excluded family. A Mean Squared Error loss was used with an Adam optimizer (learning rate of 10⁻³).
<figure><img src="image_4.png" /><figcaption>Figure 2: **Hyperparameters impact on distance matrices in the math set of genes** (a) shows the variability of distance matrices for different number of genes *G* and number of probes *N* in the math benchmark. Each set of genes of specified size contains different and independent genes from the other matrices for a total of 8 distance matrix for each data point in the figure. (b) shows the distance to the high precision matrix made of 2048 genes and *N*=128 in the math benchmark. Errorbars represent the standard error of the mean.</figcaption></figure>

We tested the benchmarks available on the hugging face open llm leaderboard which includes MMLU, ARC, Hellaswag, TruthfulQA, Winogrande and GSM8k (HuggingFaceH4) and only included open access models for which the scores are available on the leaderboard. Thus we didn't include proprietary models in this study as, as explained in later sections, distance computation is slightly biased for these models and benchmark scores are not obtained in the same conditions as in the leaderboard (number of shots, COT, ...). The benchmarks used for the 'gene' set were distinct from these benchmarks to avoid any type of contamination between the 'alleles' used to generate genetic distances and the performance of the models in the considered benchmark tasks.

# 3 EXPERIMENTS AND RESULTS

## 3.1 WHAT IS THE IMPACT OF HYPERPARAMETERS ON THE DISTANCE MATRIX?

We first ran the hyperparameters' optimization process explained in Methods2.3 and plotted the results in figure 2a left side. This graph shows a clear decrease in the variability as the number of 'genes', *G* grows with almost no effect from *N*. This is interesting: it seems that having different sets of 'genes' doesn't appear to change the similarity matrix as long as there are enough of them (at least in the open-web-math and mbxp dataset - see Appendix D for the results on the code set of 'genes').

However this method doesn't make it possible to find a good *N*, indeed, the probability for two models to generate the same token in the same context in only one try is quite low. Therefore, a very low *N* will make all models appear particularly different making the similarity matrix look like the identity matrix yielding unsatisfactory results despite having a low variance. Thus having a *N* high enough is required to get a useful similarity matrix and we need to find a better metric but how to choose it ?

We have just seen that *G* monitors the variability of the matrix (variability parameter), thus a similarity matrix with a very high *G* should be particularly stable across different sets of genes. We then compared modestly parametrized similarity matrices to study how hyperparameters *G* and *N* influence the difference between a lower precision matrices to a high precision matrix on average (see Methods 2.3 for the computational details). This new metric should penalize having a low *N* leading to similarity matrices close to the identity matrix and may yield more satisfying results.

As shown in Figure 2b, while increasing the number of genes still seems to approximate better high precision matrix, this time, the number of probes is also very important. Indeed, for each value of *N*, the performance saturates from some *G* value making less and less improvement when *G* increases. Thus, this figure gives an optimal *G* for a given *N* in order to approximate the high precision matrix efficiently with a low cost. The total cost of the algorithm in tokens being proportional to *G* × *N*, we found a good tradeoff between variance and precision around *G* = 128 and *N* = 32.
The estimated cost to run the algorithm per model is therefore 128 genes × 32 probes = 4096 queries of ≈ 20 tokens. As a point of reference, conducting the MMLU benchmark requires around 14,000 queries on significantly longer prompts (≈ 70 tokens each), making PhyloLM approximately 10 times less expensive in terms of the number of tokens required.

<figure><img src="image_5.png" /><figcaption>Figure 3: **Phylogenetic tree reconstruction.** On the left it is shown the ground truth concerning the relation of some LLMs of the Mistral family. Right is the reconstruction from the phylogenetic algorithm on the 'math' set of genes for the five latest models of this family ("leaves" of the phylogenetic tree) on which we run PhyloLM. On the right, it is shown the reconstructed phylogenetic tree PhyloLM on the 5 "leafs" models. The numerical labels (0:3) map the true common ancestors (on the right, "ground truth") to the inferred ones (on the left, "reconstructed"). It can be seen that the true and the reconstructed trees are topologically equivalent</figcaption></figure>

## 3.2 CAN WE TRACE BACK THE GENEALOGY OF LLMs USING TOOLS FROM GENETICS?

We first examine the results of PhyloLM by analyzing the resulting phylogenic trees (materialized as dendrograms). However, before dwelling into the results, an important point to understand is that, in genetics, branches in the tree show probable speciation events that occured in the past, when from an extinct common ancestor, two (or more) current species (leaves of the tree) emerged. When looking at LLMs, 'common ancestors' are not extinct, but rather among the studied 'populations'. Take for instance Mistral 7B that is the common ancestor of OpenChat3.5 and Zephyr 7B Alpha, but still included in our analysis. Oblivious of this difference, the dendrogram plotting method will put all models at the 'leaves' of the tree, while, in fact, some of them (such as Mistral 7B) should be at a speciation node. As such, without additional information about which model is at a node, it is difficult to interpret them in the same way as in genetics. Without this important phylogenetic assumption, one has to bear in mind that what matters (and should be compared with the ground truth) is their relative distance and position when evaluating the dendrograms resulting from the phylogenetic analysis of LLMs. Indeed the distance between two models is represented by the distance from their respective leaves in the dendrogram.

To investigate the capabilities of PhyloLM, let's first start by respecting this assumption by looking at a set of 9 models from the Mistral family whose relationships are known because transparently disclosed by their creators. Out of these 9 models, 5 are leaves in the ground truth dendrogram (Arc53, 2023; mlabonne, 2023; Tenyx, 2024; Ullah, 2024; Vallego, 2024). Running PhyloLM on these 5 models getting the distance matrix between them and finally plotting the NJTree we perfectly get back the ground truth phylogenetic tree (see Fig 3) validating the method. These rooted trees are not necessarily very stable as the NJ algorithm makes an unrooted tree of the evolution but then has to choose the root. In Appendix D we show that, on the code genome, the root has been mistakenly attributed to model 3 while the structure of the tree is right. That is why we prefer to plot unrooted trees in the rest of this paper.

### 3.2.1 GLOBAL DENDROGRAM

**LLMs: open-source vs private, completion vs chat** Now let's drop the assumption of not having 'common ancestors' in the set of LLMs. The LLMs we are investigating here include 111 open
<figure><img src="image_6.png" /><figcaption>Figure 4: **Inferred phylogenetic tree of LLMs on the 'math' set of genes.** (a) completion models include all open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Llama models have been split by version of the pretrained model and the number of parameters.</figcaption></figure>

access models spanning from 70M to 176B parameters and 45 closed LLMs. Most modern LLMs are only accessible through a chat API which naturally adds new tokens to the prompt such as chat messages markers biasing the completion of the given 'gene'. This can strongly influence PhyloLM as the algorithm will compare 'alleles' that do not correspond to the same 'gene'. As such we call completion models LLMs that were accessed in a way that can generate a completion to a very specific sequence of tokens without adding more tokens. All the 111 open access models we included in this study were accessed in this completion setting but among the 45 propriety models we only considered 14 of them to be completion models (see Appendix B for more details). That is why we split the LLMs and investigated them in 2 groups: completion models (to show the capabilities of PhyloLM when run in good conditions) and the others on which we suspect additional prompting manipulation. In both classes of models we found that our algorithm was largely capable of clustering LLMs into their original families, with only a few specificities discussed below. Dendrograms for both model classes are in Figure 4.

In the completion group of models we notice very clear Llama clusters separating the family from other families but also on a more fine grained level, subfamilies of llama linked to the version of the models and their respective sizes. Similarly clear cluster appear for Mistral, Qwen and Bloom. The other families such as Falcon, OPT, Pythia and GPT 3 are more mixed with each other and indeed we know that OPT, Pythia and Falcon-RW-1B (the one the closest to OPT in the tree) were trained each on their own version of the Common Crawl dataset and thus share a similar training set. Lastly, some GPT-3 models (ada, cabbage and curie) appear to be close to this OPT, Pythia and Falcon-RW cluster showing they may have been trained on a version of the Common Crawl as well. On the other hand, GPT-3.5 completion models including text-davinci-002 and text-davinci-003 seem to share more with Falcon than other models while davinci-002, cabbage-002 and gpt-3.5-turbo-instruct look more related to Qwen and more precisely its CausalLM finetuning. It is important to understand that dendrograms in LLMs are just a visualisation tool, much more details can be found in the similarity matrix shown in Figure 9 Appendix I shows dendrograms with model names of the models (see Figure 23).

In the chat models group, we also find a lot of structure : Palm and Gemini models are on the same branch, Gemini seems to be a further improvement on Palm as it is further on the branch (and indeed they are both from Google showing maybe a sharing of their training data) while claude has its own branch and Mistral / GPT-3.5 and GPT-4 models show some similarities. Dendrograms with model names are provided in Figure 23 in Appendix I.

Additional figure are available and discussed in Appendix: similarity matrices are in Figure 9 (Appendix E). Code results are in Appendix D, with the dendrogram in Figure 8 and the similarity matrix in Figure 10. Additional mixed class figures are in Appendix G: Figure 18 (math), Figure 19 (code), and global similarity matrices in Figures 16 and 17.
<figure><img src="image_7.png" /><figcaption>Figure 5: Predictions from the logistic regression compared to ground truth for every model (leave one family out method) on ARC benchmark. (a) Scatter plot showing the fitting of the logistic regression on all models but the OPT family (in grey) and the prediction of OPT performance by the regression (in red). (b) Predictions from the logistic regression for each family. To predict a family, the regressor fits on all the other families to finally predict the score of the models from the remaining family (leave one family out method - see (a)).</figcaption></figure>

## 3.3 CAN WE INFER MODEL CAPABILITIES FROM THE GENETIC DISTANCE?

We then investigated whether the genetic distance metric can be used to predict the abilities of language models. As such we used the benchmark scores from the Huggingface open LLM leaderboard. The results indicate that the prediction correlates with the true score of the models (Figures 5 (b) and 15 (a)). Indeed, we found that the Pearson's correlation coefficients (r) of the correlation between the true scores and the predicted ones was positive and significant for all benchmarks and regardless of the set of “genes” used to make the prediction (mean±sem: 0.68±0.04; Student's t-test again zero: t(11)=16.0, p<0.001; see Figure 15 (b) in Appendix F). In other terms, within benchmarks and across families, the phylogenetic distance metric allowed us to predict on average 48.2±0.03% of the variance of the between-model benchmark performance. In a control analysis, we also verified that significant correlation was also achieved within families, thus eliminating the possibility that significant prediction in the previous analysis was driven by our metric simply capturing the fact that different families have different levels of performance on average. To do so, we calculated the Pearson correlation between the true and the predicted scores per benchmark and within each family separately. The results indicate that, even though for some combinations of families and benchmarks, we obtained small or negative correlation coefficients (which is unsurprising, since these correlations were sometimes calculated across very few data points), also in this case, the results were in average positive and significant difference from zero (0.64±0.05; t(107)= 20.7, p<0.001; see Figure 15 (c) in Appendix F). Within families, the variance explained by our method amounted to 52.2±0.03% on average, thus indicating that our metric achieved good predictive power even when drastically increasing the level of granularity. Individual plots for each benchmark are shown in Appendix F

## 4 DISCUSSION

Here we show that an algorithm, inspired by those used in phylogeny, is successful in reconstructing important aspects of the genesis of LLMs, based solely on their outputs to diverse short queries. By leveraging the genetic distance matrix, it becomes feasible to robustly trace the relationships and evolution of models over time. This is particularly evident in the constructed dendrograms, where clear clusters align with distinct families of LLMs, offering a visual representation of their evolutionary trajectories or at least their training similarity. It is important to also emphasize the
applicability of these methods to proprietary models. Understanding the fine-tuning relationships and performance characteristics of private models is often challenging due to limited access to training details and data. PhyloLM offers a valuable tool for gaining insights into these aspects, by providing to the research community a more transparent image of how proprietary models evolve.

We also show that the utility of the "genetic" distance, derived from our algorithm, was not limited to capturing the training relationships, but could be used to infer the performances of models on various benchmarks. The observation that a logistic regression trained on the genetic distance matrix can accurately predict benchmark accuracy has the potential to accelerate the evaluation of new LLMs capabilities in a very computationally efficient manner. Overall, our method provides a robust and insightful analysis of the history, relationships, and performance of Large Language Models, even in cases where detailed training information is not publicly available.

Despite these promising results, it is important to acknowledge the inherent limitations of applying the genetic metaphor to LLMs. Phylogenetic algorithms, traditionally designed for biological analysis where common ancestors are not included among the tested species, face challenges when applied to LLMs, where common ancestors are present among the studied models. Furthermore, chat interfaces complicate the acquisition of reliable genetic material. Nonetheless, this work lays the foundation for further studies aimed at refining these algorithms to better fit the LLM framework and chat models. Our study did not explore the effect of temperature, and while our results were consistent across two sets of genes (and more in Appendix J), examining an even broader range of genes could provide additional insights. Additionally, while the predictive results for benchmark scores are promising (roughly 50% of the variance explained) and could be practically applied to estimate the capabilities of new models, it remains room for improvement (a possible venue being using multiple sets of genes in the evaluation).

Lastly, similarity matrices serve as versatile tools with numerous applications in the study and opti-
mization of large language models (LLMs). For instance, in our investigation of model quantization, we discovered that as the size of the model increases, the quantized version more closely approximates the original model (see Appendix H). Additional fields in which PhyloLM could provide very good insights could also include model merging (Goddard et al., 2024) and scaling laws but we leave it for further research.

5 ACKNOWLEDGMENTS

This work was granted access to the HPC/IA resources of [IDRIS HPE Jean Zay A100] under the allocation 2024- [AD011013693R2] and [IDRIS HPE Jean Zay H100] under the allocation 2024- [A0171011996] and made by GENCI. SP is supported by the European Research Council under the European Union's Horizon 2020 research and innovation program (ERC) (RaReMem: 101043804), the Agence National de la Recherche (CogFinAgent: ANR-21-CE23-0002-02; RELATIVE: ANR-21-CE37-0008-01; RANGE: ANR-21-CE28-0024-01; ANR-17-EURE-0017), the Alexander Von Humboldt foundation, the EUR Frontiers in Cognition, the Idex PSL and a Google unrestricted gift. PYO is supported by ANR AI individual chair ANR-19-CHIA-0004.

REFERENCES

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co-
jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic,
Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series
of open language models, 2023.

Arc53. Arc53/docsgpt-7b-mistral, December 2023. URL https://huggingface.co/Arc53/docsgpt-7b-mistral.

Ben Athiwaratkun et al. Multi-lingual evaluation of code generation models, 2023.

Quentin D Atkinson, Andrew Meade, Chris Venditti, Simon J Greenhill, and Mark Pagel. Languages evolve in punctuational bursts. *Science*, 319(5863):588–588, 2008.

Jinze Bai et al. Qwen technical report, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

BigScienceWorkshop et al. Bloom: A 176b-parameter open-access multilingual language model, 2023.

Tom B. Brown et al. Language models are few-shot learners, 2020.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.

François Chollet. On the measure of intelligence, 2019.

Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways, 2022.

R Dawkins. *The Selfish Gene*. Oxford University Press, Oxford, UK, 1976.

Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models, 2023.

Julien d'Huy. Polyphemus (aa. th. 1137): A phylogenetic reconstruction of a prehistoric tale. *Nouvelle Mythologie Comparée/New Comparative Mythology*, 1(1):http-nouvellemythologiecomparee, 2013.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.

Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's mergekit: A toolkit for merging large language models. *arXiv preprint arXiv:2403.13257*, 2024.

Russell D Gray, David Bryant, and Simon J Greenhill. On the shape and fabric of human history. *Philosophical Transactions of the Royal Society B: Biological Sciences*, 365(1559):3923-3933, 2010.

Niklaus Grünwald, Sydney Everhart, Brian Knaus, and Zhian Kamvar. Best practices for population genetic analyses. *Phytopathology*, 107, 05 2017. doi: 10.1094/PHYTO-12-16-0425-RVW.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.

HuggingFaceH4. open llm leaderboard. URL https://huggingface.co/spaces/HuggingFaceH4/open.llmLeaderboard.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Percy Liang et al. Holistic evaluation of language models, 2023.

Q. Vera Liao and Jennifer Wortman Vaughan. Ai transparency in the age of llms: A human-centered research roadmap, 2023.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024.
mlabonne. mlabonne/neuralhermes-2.5-mistral-7b, November 2023. URL https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B.

OpenAI et al. Gpt-4 technical report, 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.

N Saitou and M Nei. The neighbor-joining method: a new method for reconstructing phylogenetic trees. *Molecular Biology and Evolution*, 4(4):406-425, 07 1987. ISSN 0737-4038. doi: 10.1093/oxfordjournals.molbev.a040454. URL https://doi.org/10.1093/oxfordjournals.molbev.a040454.

Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.

Naoko Takezaki and Masatoshi Nei. Genetic Distances and Reconstruction of Phylogenetic Trees From Microsatellite DNA. *Genetics*, 144(1):389-399, 09 1996. ISSN 1943-2631. doi: 10.1093/ge netics/144.1.389. URL https://doi.org/10.1093/genetics/144.1.389.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.

Gemini Team et al. Gemini: A family of highly capable multimodal models, 2024a.

Gemma Team et al. Gemma: Open models based on gemini research and technology, 2024b.

Jamie Tehrani and Julien d'Huy. *2017. Phylogenetics Meets Folklore: Bioinformatics Approaches to the Study of International Folktales*, pp. 91-114. 01 2017. ISBN 978-3-319-39443-5. doi: 10.1007/978-3-319-39445-9_6.

Jamshid J Tehrani and Mark Collard. On the relationship between interindividual cultural transmission and population-level cultural diversity: a case study of weaving in iranian tribal populations. *Evolution and Human Behavior*, 30(4):286-300, 2009.

Tenyx. Tenyxchat: Language model alignment using tenyx fine-tuning, January 2024. URL https://huggingface.co/tenyx/TenyxChat-7B-v1.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.

Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023b.

Imran Ullah. Imran1/medchat3.5, January 2024. URL https://huggingface.co/Imran1/MedChat3.5.

Jorge Vallego. neovalle/h4rmoniousanthea, January 2024. URL https://huggingface.co/neovalle/H4rmoniousAnthea.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.

Shen Zheng, Yuyu Zhang, Yijie Zhu, Chenguang Xi, Pengyang Gao, Xun Zhou, and Kevin Chen-Chuan Chang. GPT-Fathom: Benchmarking large language models to decipher the evolutionary path towards GPT-4 and beyond, 2023. URL https://arxiv.org/abs/2309.16583.
# APPENDIX

This Appendix provides additional materials for PhyloLM:

* Section A presents examples of 'genes' and 'alleles' for 3 LLMs.

* Section B shows the list of models included in this study with finetuning relationships, sizes and benchmark scores.

* Section C gives PhyloLM pseudo code.

* Section D outlines the results on the code set of genes.

* Section E discusses more in depth similarity matrices results, differences between models and proposes potential explanations for such observations.

* Section F provides results about benchmark prediction on each benchmark for each family of models.

* Section G represents dendrograms including both sets of genes together.

* Section H exhibits results on PhyloLM and model quantization.

* Section I gives an overview of dendrograms with the names of the models.
A EXAMPLES OF GENES AND ALLELES

Genes were extracted from the Open-Web-Math (Paster et al., 2023) and MBXP (Athiwaratkun et al., 2023) datasets. The Open-Web-Math dataset comprises 31,577 rows, from which we selected the first 20 to 100 characters from the 'text' column (refer to Section 2.2 for detailed extraction methods). The MBXP dataset contains 6,814 rows, from which genes were extracted from the 'canonical solution' column. Twenty of the shortest genes (to fit on the page) in the extracted gene sets are presented in Table 1.

<table><caption>Table 1: Examples of 20 short genes extracted from both sets of genes and one allele sampled from 3 LLMs: Llama 1 7B, Llama 2 7B and GPT4 turbo 03/14. ␣ stands for a newline and □ represents a space (except in genes where spaces are packed to only one space for simplicity). Only the shortest genes have been included to fit in the table. Longer genes are also present in the gene set.</caption><thead><tr><th colspan="4">(a) from open-web-math set of genes</th></tr><tr><th>Genes</th><th>Llama1 7B</th><th>Llama2 7B</th><th>GPT-4 (0314)</th></tr></thead><tbody><tr><td># Ignatius and the P</td><td>igeo</td><td>ig ↕ ↕</td><td>arli</td></tr><tr><td>1 $\begingroup$ Close</td><td>vote</td><td>vote</td><td>I'm □</td></tr><tr><td># Propositional Logic</td><td>Prop</td><td>/Mis</td><td>Prop</td></tr><tr><td>[texhax] environment</td><td>vari</td><td>2 ↕ De</td><td>An □ e</td></tr><tr><td>### Homes ↕ ↕ There are</td><td>23 □ h</td><td>219 □</td><td>seve</td></tr><tr><td># Annual income of A a</td><td>pers</td><td>B □ an</td><td>I □ am</td></tr><tr><td># Image Mosaicking ¶ ↕ ↕ #</td><td>Impl</td><td>Writ</td><td>Imag</td></tr><tr><td># Physics (Version 8.4)</td><td>) ↕ ↕ P</td><td>) ↕ ↕ P</td><td>"Phy</td></tr><tr><td># Solve the linear equatio</td><td>... ↕</td><td>## □ S</td><td>To □ s</td></tr><tr><td>[texhax] \mid Description ↕ ↕</td><td>[tex</td><td>I □ am</td><td>The □</td></tr><tr><td># string.replace regex ↕ ↕ Synt</td><td>akti</td><td>acti</td><td>ax: ↕</td></tr><tr><td># Math Help - 2-norm of a ma</td><td>xtri</td><td>## □ P</td><td>To □ f</td></tr><tr><td># Math Help - matlab code hel</td><td>pe ↕ ↕</td><td>plin</td><td>I'd □</td></tr><tr><td>Thank you for visiting nature</td><td>gift</td><td>tour</td><td>You'</td></tr><tr><td>size - Maple Help ↕ ↕ MTM ↕ ↕ size ↕</td><td>The □</td><td>\ □ beg</td><td>In □ M</td></tr><tr><td># In observing a Tetrahedron...</td><td>In □ o</td><td>In □ o</td><td>A □ te</td></tr><tr><td>Previous issue · Next issue ·</td><td>Volu</td><td>Arch</td><td>Arch</td></tr><tr><td># All Questions ↕ ↕ 1,524 questions ↕</td><td>All □</td><td>## □ A</td><td>Unfo</td></tr><tr><td>## [POJ2411] Mondriaan's Dream ↕ ↕ 成</td><td>功 ↕ ↕ □</td><td>绩排名:</td><td>本题考察</td></tr><tr><td># How to prove that $C=\{x: Ax \le 0\}$</td><td>0 □ $</td><td>0 □ $</td><td>b \ □ $</td></tr></tbody></table><table><thead><tr><th colspan="4">(b) from MBXP set of genes</th></tr></thead><tbody><tr><td>return 4 * a; ↕ }</td><td>temp</td><td>func</td><td>func</td></tr><tr><td> ↕ double s</td><td>= □ 1;</td><td>1 (do</td><td>I'm □</td></tr><tr><td> ↕ res = [] ↕ for e</td><td>in □ i</td><td>in □ r</td><td>ach □</td></tr><tr><td> ↕ // TODO:</td><td>□ □ □</td><td>15.0</td><td>Ther</td></tr><tr><td>return n > 0 ? n % 10</td><td>0 □ ; □</td><td>: □ 0;</td><td>+ □ Ma</td></tr><tr><td>return a + b + c; ↕ }</td><td>\ end</td><td>&lt;/s&gt;</td><td>func</td></tr><tr><td>// Function to</td><td>hand</td><td>□ □ //</td><td>dete</td></tr><tr><td>if (monthnum3 == 6)</td><td>{ ↕ □</td><td>{ ↕ □ □</td><td>I □ m</td></tr><tr><td> ↕ str = ".join (tuple1)</td><td>) ↕ □ □</td><td>if □ t</td><td>I □ y</td></tr><tr><td> ↕ res = tuple (list (te))</td><td>.fie</td><td>.out</td><td>I □ s</td></tr><tr><td> ↕ r = n% m ↕ return (r)</td><td>prin</td><td>end ↕</td><td>def □</td></tr><tr><td> ↕ seq_num s = [seq_num s[</td><td>star</td><td>1:]]</td><td>i □ f</td></tr><tr><td>double result = 0; ↕</td><td>if □ (</td><td>doub</td><td>In □ o</td></tr><tr><td> ↕ string result =</td><td>null</td><td>null</td><td>I □ ca</td></tr><tr><td>$isSame = true; ↕ fore</td><td>: ↕ □ □</td><td>ach □</td><td>ach □</td></tr><tr><td>return n * (2 * n - 1) ↕ }</td><td>// □ T</td><td>/** ↕</td><td>func</td></tr><tr><td>let sortedArr = arr.sort</td><td>((a,</td><td>{ □ a,</td><td>((a,</td></tr><tr><td>$list1 = $list1 || []; ↕</td><td>$lis</td><td>$lis</td><td>if □ (</td></tr><tr><td>return !a[0] == !a[n-1]; ↕ }</td><td>bool</td><td>int □</td><td>This</td></tr><tr><td>string result = ""; ↕ fo</td><td>obar</td><td>.ope</td><td>r □ (i</td></tr></tbody></table>
B MODELS INCLUDED IN THIS STUDY

Open access models were run on A100 80Gb GPUs (approximately one per 20B parameters) with default floating precision except for Falcon180B that was downsized from float32 to bfloat16 as it was too big to fit on 8 GPUs. OpenAI models were accessed through the Openai API. Claude models were accessed through the Anthropic AI API. Proprietary mistral models were accessed through the Mistral API. Proprietary Google models were accessed through the VertexAI API. Tokens were sampled with a temperature of 0.7 with no sampling restriction (topp=1).

The following table 2 includes all models tested in this study with their benchmark scores. Only the completion models have been included in the benchmark score prediction study. After this table is the table of chat models that were separated in dendrogram plots and not included in benchmark score prediction. They can be found in Table 3.

<footnote n="1">¹From HELM (Liang et al., 2023) - no information about prompting - https://crfm.stanford.edu/helm/classic/latest/#/models

¹From GPT-Fathom (Zheng et al., 2023) - ARC(1 shot) - HellaSwag (1 shot) - MMLU (5 shot) - TruthfulQA (1 shot) - WinoGrande (1 shot) - GSM8K (8 shot CoT) - https://crfm.stanford.edu/helm/classic/latest/#/models

¹From HuggingFace (HuggingFaceH4) - ARC(25 shot) - HellaSwag (10 shot) - MMLU (5 shot) - TruthfulQA (0 shot) - WinoGrande (5 shot) - GSM8K (5 shot) - https://huggingface.co/spaces/HuggingFaceH4/open_llm leaderboard</footnote>
<table><caption>Table 2: List of completion LLMs included in the study. They appear in the similarity matrices (see Fig 9 10) in the same order as presented in this table. For proprietary LLMs, names are taken from their respective API. For all the other models, they were downloaded from the huggingface hub with the same name as given here. B and M stand for billions and millions, Ø for trained from scratch and ? means unknown or not officially communicated.</caption><thead><tr><th>Name</th><th>Family</th><th>Size</th><th>Parent</th><th>AR</th><th>HS</th><th>ML</th><th>TQ</th><th>WG</th><th>GS</th></tr></thead><tbody><tr><td>llama-7b</td><td>llama 1-7</td><td>7B</td><td>Ø</td><td>50.9</td><td>77.8</td><td>35.6</td><td>34.3</td><td>71.4</td><td>8.03</td></tr><tr><td>alpaca-7b</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>52.0</td><td>76.9</td><td>41.4</td><td>37.5</td><td>69.4</td><td>1.44</td></tr><tr><td>wizard-7b</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>vicuna-7b-v1.1</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>vicuna-7b-v1.3</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>50.4</td><td>76.9</td><td>48.1</td><td>47.0</td><td>70.4</td><td>5.68</td></tr><tr><td>baize-7b</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>48.9</td><td>75.0</td><td>39.6</td><td>41.3</td><td>71.1</td><td>4.16</td></tr><tr><td>chimera-inst-chat-7b</td><td>llama 1-7</td><td>7B</td><td>llama-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>llama-13b</td><td>llama 1-13</td><td>13B</td><td>Ø</td><td>56.1</td><td>80.9</td><td>47.6</td><td>39.4</td><td>76.2</td><td>7.58</td></tr><tr><td>vicuna-13b-v1.1</td><td>llama 1-13</td><td>13B</td><td>llama-13b</td><td>52.7</td><td>80.1</td><td>51.9</td><td>52.0</td><td>74.1</td><td>8.64</td></tr><tr><td>vicuna-13b-v1.3</td><td>llama 1-13</td><td>13B</td><td>llama-13b</td><td>54.6</td><td>80.4</td><td>52.8</td><td>52.1</td><td>74.8</td><td>10.7</td></tr><tr><td>openchat_v2</td><td>llama 1-13</td><td>13B</td><td>llama-13b</td><td>57.1</td><td>81.1</td><td>50.5</td><td>49.5</td><td>76.2</td><td>9.09</td></tr><tr><td>openchat_v2_w</td><td>llama 1-13</td><td>13B</td><td>llama-13b</td><td>57.3</td><td>81.2</td><td>50.1</td><td>50.6</td><td>75.9</td><td>8.41</td></tr><tr><td>chimera-inst-chat-13b</td><td>llama 1-13</td><td>13B</td><td>llama-13b</td><td>55.3</td><td>78.9</td><td>50.5</td><td>50.1</td><td>73.9</td><td>8.18</td></tr><tr><td>llama-2-7b-hf</td><td>llama 2-7</td><td>7B</td><td>Ø</td><td>53.0</td><td>77.7</td><td>43.7</td><td>38.9</td><td>74.0</td><td>14.4</td></tr><tr><td>Orca-2-7b</td><td>llama 2-7</td><td>7B</td><td>llama-2-7b</td><td>54.0</td><td>76.1</td><td>56.3</td><td>52.4</td><td>73.4</td><td>14.7</td></tr><tr><td>tigerbot-7b-base</td><td>llama 2-7</td><td>7B</td><td>llama-2-7b</td><td>47.6</td><td>72.0</td><td>45.1</td><td>42.2</td><td>69.6</td><td>10.8</td></tr><tr><td>tigerbot-7b-chat</td><td>llama 2-7</td><td>7B</td><td>tigerbot-7b-base</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>OpenHermes-7B</td><td>llama 2-7</td><td>7B</td><td>llama-2-7b</td><td>56.1</td><td>78.3</td><td>48.6</td><td>44.9</td><td>74.5</td><td>5.00</td></tr><tr><td>vicuna-7b-v1.5</td><td>llama 2-7</td><td>7B</td><td>llama-2-7b</td><td>53.2</td><td>77.3</td><td>50.8</td><td>50.3</td><td>72.1</td><td>8.18</td></tr><tr><td>llama-2-13b-hf</td><td>llama 2-13</td><td>13B</td><td>Ø</td><td>58.1</td><td>80.9</td><td>54.3</td><td>34.1</td><td>76.6</td><td>22.8</td></tr><tr><td>openchat_v3.1</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>59.8</td><td>82.8</td><td>56.7</td><td>44.4</td><td>76.2</td><td>13.7</td></tr><tr><td>openchat_v3.2</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>59.6</td><td>82.6</td><td>56.6</td><td>44.4</td><td>76.9</td><td>13.6</td></tr><tr><td>OpenHermes-13B</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>60.1</td><td>82.1</td><td>56.1</td><td>45.9</td><td>75.4</td><td>11.5</td></tr><tr><td>vicuna-13b-v1.5</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>57.0</td><td>81.2</td><td>56.6</td><td>51.5</td><td>74.6</td><td>11.2</td></tr><tr><td>openchat_v3.2.super</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>59.8</td><td>82.5</td><td>55.8</td><td>42.2</td><td>75.9</td><td>13.4</td></tr><tr><td>tigerbot-13b-base-v1</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-13b-base-v2</td><td>llama 2-13</td><td>13B</td><td>llama-2-13b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-13b-chat-v1</td><td>llama 2-13</td><td>13B</td><td>tigerbot-13b-base-v1</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-13b-chat-v2</td><td>llama 2-13</td><td>13B</td><td>tigerbot-13b-base-v2</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-13b-chat-v3</td><td>llama 2-13</td><td>13B</td><td>tigerbot-13b-base-v2</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-13b-chat-v4</td><td>llama 2-13</td><td>13B</td><td>tigerbot-13b-base-v2</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>bloom-3b</td><td>bloom</td><td>3B</td><td>Ø</td><td>35.7</td><td>54.3</td><td>26.5</td><td>40.5</td><td>57.6</td><td>1.51</td></tr><tr><td>bloom-7b</td><td>bloom</td><td>7B</td><td>Ø</td><td>41.1</td><td>61.9</td><td>26.2</td><td>38.8</td><td>65.4</td><td>1.36</td></tr><tr><td>bloomz-3b</td><td>bloom</td><td>3B</td><td>bloom-3b</td><td>36.8</td><td>54.9</td><td>32.9</td><td>40.3</td><td>57.1</td><td>0.0</td></tr><tr><td>bloomz-7b</td><td>bloom</td><td>7B</td><td>bloom-7b</td><td>42.4</td><td>63.0</td><td>37.8</td><td>45.2</td><td>64.6</td><td>0.07</td></tr><tr><td>tigerbot-7b-base-v1</td><td>bloom</td><td>7B</td><td>bloom-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-7b-base-v2</td><td>bloom</td><td>7B</td><td>bloom-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-7b-sft-v1</td><td>bloom</td><td>7B</td><td>tigerbot-7b-base-v1</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>tigerbot-7b-sft-v2</td><td>bloom</td><td>7B</td><td>tigerbot-7b-base-v2</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>phoenix-inst-chat-7b</td><td>bloom</td><td>7B</td><td>bloom-7b</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>bloom-176b</td><td>bloom</td><td>176B</td><td>Ø</td><td>50.4</td><td>76.4</td><td>30.8</td><td>39.7</td><td>72.0</td><td>6.89</td></tr></tbody></table>
<table><thead><tr><th>Name</th><th>Family</th><th>Size</th><th>Parent</th><th>AR</th><th>HS</th><th>ML</th><th>TQ</th><th>WG</th><th>GS</th></tr></thead><tbody><tr><td>pythia-70m</td><td>pythia</td><td>70M</td><td>Ø</td><td>21.5</td><td>27.2</td><td>25.9</td><td>47.0</td><td>51.4</td><td>0.30</td></tr><tr><td>pythia-160m</td><td>pythia</td><td>160M</td><td>Ø</td><td>22.7</td><td>30.3</td><td>24.9</td><td>44.2</td><td>51.5</td><td>0.22</td></tr><tr><td>pythia-410m</td><td>pythia</td><td>410M</td><td>Ø</td><td>26.1</td><td>40.8</td><td>27.2</td><td>41.2</td><td>53.1</td><td>0.68</td></tr><tr><td>pythia-1.4b</td><td>pythia</td><td>1.4B</td><td>Ø</td><td>31.4</td><td>52.8</td><td>25.8</td><td>38.8</td><td>58.0</td><td>1.51</td></tr><tr><td>pythia-2.8b</td><td>pythia</td><td>2.8B</td><td>Ø</td><td>36.2</td><td>60.6</td><td>26.7</td><td>35.5</td><td>60.2</td><td>0.83</td></tr><tr><td>pythia-6.9b</td><td>pythia</td><td>6.9B</td><td>Ø</td><td>41.2</td><td>67.0</td><td>26.4</td><td>35.1</td><td>64.0</td><td>1.66</td></tr><tr><td>pythia-12b</td><td>pythia</td><td>12B</td><td>Ø</td><td>39.5</td><td>68.8</td><td>26.7</td><td>31.8</td><td>64.1</td><td>1.74</td></tr><tr><td>dolly-v2-3b</td><td>pythia</td><td>3B</td><td>pythia-2.8b</td><td>25.2</td><td>26.5</td><td>24.6</td><td>?</td><td>59.4</td><td>1.06</td></tr><tr><td>dolly-v2-7b</td><td>pythia</td><td>7B</td><td>pythia-6.9b</td><td>44.5</td><td>69.6</td><td>25.1</td><td>34.8</td><td>60.0</td><td>1.13</td></tr><tr><td>dolly-v2-12b</td><td>pythia</td><td>12B</td><td>pythia-12b</td><td>42.4</td><td>72.5</td><td>25.9</td><td>33.8</td><td>60.8</td><td>1.21</td></tr><tr><td>oasst-sft-4-pythia-12b-epoch-3.5</td><td>pythia</td><td>12B</td><td>pythia-12b</td><td>45.7</td><td>68.5</td><td>26.8</td><td>37.8</td><td>65.9</td><td>3.03</td></tr><tr><td>opt-125m</td><td>opt</td><td>125M</td><td>Ø</td><td>22.8</td><td>31.4</td><td>26.0</td><td>42.8</td><td>51.6</td><td>0.07</td></tr><tr><td>opt-350m</td><td>opt</td><td>250M</td><td>Ø</td><td>23.5</td><td>36.7</td><td>26.0</td><td>40.8</td><td>52.6</td><td>0.30</td></tr><tr><td>opt-1.3b</td><td>opt</td><td>1B</td><td>Ø</td><td>29.5</td><td>54.5</td><td>24.9</td><td>38.7</td><td>59.7</td><td>0.15</td></tr><tr><td>opt-2.7b</td><td>opt</td><td>3B</td><td>Ø</td><td>33.9</td><td>61.4</td><td>25.4</td><td>37.4</td><td>61.9</td><td>0.22</td></tr><tr><td>opt-6.7b</td><td>opt</td><td>7B</td><td>Ø</td><td>39.1</td><td>68.6</td><td>24.5</td><td>35.1</td><td>65.9</td><td>0.98</td></tr><tr><td>opt-13b</td><td>opt</td><td>13B</td><td>Ø</td><td>39.9</td><td>71.2</td><td>24.8</td><td>34.0</td><td>68.5</td><td>1.74</td></tr><tr><td>opt-30b</td><td>opt</td><td>30B</td><td>Ø</td><td>43.2</td><td>74.0</td><td>26.6</td><td>35.1</td><td>70.6</td><td>2.19</td></tr><tr><td>opt-66b</td><td>opt</td><td>66B</td><td>Ø</td><td>46.3</td><td>76.2</td><td>26.9</td><td>35.4</td><td>70.0</td><td>1.66</td></tr><tr><td>Mistral-7B-v0.1</td><td>mistral</td><td>7B</td><td>Ø</td><td>59.9</td><td>83.3</td><td>64.1</td><td>42.1</td><td>78.6</td><td>37.0</td></tr><tr><td>Mistral-7B-Instruct-v0.1</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>54.5</td><td>75.6</td><td>55.3</td><td>56.2</td><td>73.7</td><td>14.2</td></tr><tr><td>Mistral-7B-Instruct-v0.2</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>63.1</td><td>84.8</td><td>60.7</td><td>68.2</td><td>77.1</td><td>40.0</td></tr><tr><td>zephyr-7b-alpha</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>61.0</td><td>84.0</td><td>61.3</td><td>57.9</td><td>78.6</td><td>14.0</td></tr><tr><td>zephyr-7b-beta</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>62.4</td><td>84.3</td><td>60.7</td><td>57.8</td><td>77.1</td><td>29.0</td></tr><tr><td>docsgpt-7b-mistral</td><td>mistral</td><td>7B</td><td>zephyr-7b-beta</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>openchat_3.5</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>62.4</td><td>83.9</td><td>62.8</td><td>45.4</td><td>81.0</td><td>25.7</td></tr><tr><td>TenyxChat-7B-v1</td><td>mistral</td><td>7B</td><td>openchat_3.5</td><td>65.6</td><td>85.5</td><td>64.8</td><td>51.2</td><td>80.5</td><td>63.0</td></tr><tr><td>MedChat3.5</td><td>mistral</td><td>7B</td><td>openchat_3.5</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>neural-chat-7b-v3</td><td>mistral</td><td>7B</td><td>openchat_3.5</td><td>67.1</td><td>83.2</td><td>62.2</td><td>58.7</td><td>78.0</td><td>1.21</td></tr><tr><td>neural-chat-7b-v3-1</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>65.6</td><td>83.5</td><td>62.1</td><td>59.4</td><td>78.6</td><td>20.0</td></tr><tr><td>OpenHermes-2-Mistral-7B</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>63.0</td><td>83.8</td><td>63.4</td><td>50.2</td><td>?</td><td>?</td></tr><tr><td>OpenHermes-2.5-Mistral-7B</td><td>mistral</td><td>7B</td><td>Mistral-7B-v0.1</td><td>64.9</td><td>84.1</td><td>63.6</td><td>52.2</td><td>78.0</td><td>26.0</td></tr><tr><td>H4rmoniousAnthea</td><td>mistral</td><td>7B</td><td>OpenHermes-2.5-Mistral-7B</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>NeuralHermes-2.5-Mistral-7B</td><td>mistral</td><td>7B</td><td>OpenHermes-2.5-Mistral-7B</td><td>66.5</td><td>84.9</td><td>63.3</td><td>54.9</td><td>78.2</td><td>61.3</td></tr><tr><td>Mixtral-8x7B-v0.1</td><td>mistral</td><td>8x7B</td><td>Ø</td><td>66.3</td><td>86.4</td><td>71.8</td><td>46.8</td><td>81.6</td><td>57.6</td></tr><tr><td>Mixtral-8x7B-Instruct-v0.1</td><td>mistral</td><td>8x7B</td><td>Ø</td><td>70.1</td><td>87.5</td><td>71.3</td><td>64.9</td><td>81.0</td><td>61.1</td></tr><tr><td>Qwen-1_8B</td><td>qwen</td><td>2B</td><td>Ø</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>Qwen-7B</td><td>qwen</td><td>7B</td><td>Ø</td><td>51.3</td><td>78.4</td><td>59.8</td><td>47.7</td><td>72.6</td><td>44.9</td></tr><tr><td>Qwen-14B</td><td>qwen</td><td>14B</td><td>Ø</td><td>58.2</td><td>83.9</td><td>67.7</td><td>49.4</td><td>76.7</td><td>58.9</td></tr><tr><td>Qwen-72B</td><td>qwen</td><td>72B</td><td>Ø</td><td>65.1</td><td>85.9</td><td>77.3</td><td>60.1</td><td>82.4</td><td>70.4</td></tr><tr><td>causallm-7b</td><td>qwen</td><td>7B</td><td>Qwen-7B</td><td>50.0</td><td>74.5</td><td>61.7</td><td>50.1</td><td>69.6</td><td>22.9</td></tr><tr><td>causallm-14b</td><td>qwen</td><td>14B</td><td>Qwen-14B</td><td>56.6</td><td>79.0</td><td>65.8</td><td>47.7</td><td>74.9</td><td>58.6</td></tr><tr><td>Qwen1.5-0.5B</td><td>qwen</td><td>0.5B</td><td>Ø</td><td>31.4</td><td>49.0</td><td>39.3</td><td>38.2</td><td>57.2</td><td>16.3</td></tr><tr><td>Qwen1.5-1.8B</td><td>qwen</td><td>1.8B</td><td>Ø</td><td>37.8</td><td>61.4</td><td>46.7</td><td>39.4</td><td>60.2</td><td>33.5</td></tr><tr><td>Qwen1.5-4B</td><td>qwen</td><td>4B</td><td>Ø</td><td>48.4</td><td>71.5</td><td>56.5</td><td>47.2</td><td>66.2</td><td>52.2</td></tr><tr><td>Qwen1.5-7B</td><td>qwen</td><td>7B</td><td>Ø</td><td>54.1</td><td>78.5</td><td>61.9</td><td>51.0</td><td>71.2</td><td>53.5</td></tr><tr><td>Qwen1.5-14B</td><td>qwen</td><td>14B</td><td>Ø</td><td>56.5</td><td>81.0</td><td>69.3</td><td>52.0</td><td>73.4</td><td>67.6</td></tr><tr><td>Qwen1.5-32B</td><td>qwen</td><td>32B</td><td>Ø</td><td>63.5</td><td>85.0</td><td>74.2</td><td>57.3</td><td>81.4</td><td>61.1</td></tr><tr><td>Qwen1.5-72B</td><td>qwen</td><td>72B</td><td>Ø</td><td>65.8</td><td>85.9</td><td>77.2</td><td>59.6</td><td>83.0</td><td>65.7</td></tr><tr><td>Qwen1.5-0.5B-Chat</td><td>qwen</td><td>0.5B</td><td>Qwen1.5-0.5B</td><td>30.5</td><td>44.0</td><td>33.8</td><td>42.9</td><td>54.6</td><td>7.65</td></tr><tr><td>Qwen1.5-1.8B-Chat</td><td>qwen</td><td>1.8B</td><td>Qwen1.5-1.8B</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>Qwen1.5-4B-Chat</td><td>qwen</td><td>4B</td><td>Qwen1.5-4B</td><td>43.2</td><td>69.7</td><td>55.5</td><td>44.7</td><td>64.9</td><td>2.42</td></tr><tr><td>Qwen1.5-7B-Chat</td><td>qwen</td><td>7B</td><td>Qwen1.5-7B</td><td>55.8</td><td>78.5</td><td>61.6</td><td>53.6</td><td>67.7</td><td>13.1</td></tr><tr><td>Qwen1.5-14B-Chat</td><td>qwen</td><td>14B</td><td>Qwen1.5-14B</td><td>58.7</td><td>82.3</td><td>68.5</td><td>60.3</td><td>73.3</td><td>30.8</td></tr><tr><td>Qwen1.5-32B-Chat</td><td>qwen</td><td>32B</td><td>Qwen1.5-32B</td><td>66.0</td><td>85.4</td><td>74.9</td><td>66.9</td><td>77.1</td><td>7.05</td></tr><tr><td>Qwen1.5-72B-Chat</td><td>qwen</td><td>72B</td><td>Qwen1.5-72B</td><td>68.5</td><td>86.4</td><td>77.4</td><td>63.8</td><td>79.0</td><td>20.3</td></tr></tbody></table>
<table><thead><tr><th>Name</th><th>Family</th><th>Size</th><th>Parent</th><th>AR</th><th>HS</th><th>ML</th><th>TQ</th><th>WG</th><th>GS</th></tr></thead><tbody><tr><td>falcon-rw-1b</td><td>falcon</td><td>1B</td><td>Ø</td><td>35.0</td><td>63.5</td><td>25.2</td><td>35.9</td><td>62.0</td><td>0.53</td></tr><tr><td>falcon-rw-7b</td><td>falcon</td><td>7B</td><td>Ø</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>falcon-7b</td><td>falcon</td><td>7B</td><td>Ø</td><td>47.8</td><td>78.1</td><td>27.7</td><td>34.2</td><td>72.3</td><td>4.62</td></tr><tr><td>falcon-7b-instruct</td><td>falcon</td><td>7B</td><td>falcon-7b</td><td>46.1</td><td>70.8</td><td>25.8</td><td>44.0</td><td>67.9</td><td>4.70</td></tr><tr><td>falcon-40b</td><td>falcon</td><td>40B</td><td>Ø</td><td>61.9</td><td>85.2</td><td>56.9</td><td>41.7</td><td>81.2</td><td>21.4</td></tr><tr><td>falcon-40b-instruct</td><td>falcon</td><td>40B</td><td>falcon-40b</td><td>61.6</td><td>84.3</td><td>55.4</td><td>52.5</td><td>79.7</td><td>34.3</td></tr><tr><td>falcon-180b</td><td>falcon</td><td>180B</td><td>Ø</td><td>69.1</td><td>88.8</td><td>69.5</td><td>45.1</td><td>86.8</td><td>45.9</td></tr><tr><td>gemma-2b</td><td>gemma</td><td>2B</td><td>Ø</td><td>48.3</td><td>71.7</td><td>41.7</td><td>33.0</td><td>66.2</td><td>16.9</td></tr><tr><td>gemma-2b-it</td><td>gemma</td><td>2B</td><td>gemma-2b</td><td>43.9</td><td>62.6</td><td>37.6</td><td>45.8</td><td>60.9</td><td>5.45</td></tr><tr><td>gemma-7b</td><td>gemma</td><td>7B</td><td>Ø</td><td>61.0</td><td>82.4</td><td>66.0</td><td>44.9</td><td>78.4</td><td>52.7</td></tr><tr><td>gemma-7b-it</td><td>gemma</td><td>7B</td><td>gemma-7b</td><td>51.4</td><td>71.9</td><td>53.5</td><td>47.2</td><td>67.9</td><td>29.1</td></tr><tr><td>gemma-1.1-2b-it</td><td>gemma</td><td>2B</td><td>gemma-2b-it</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>gemma-1.1-7b-it</td><td>gemma</td><td>7B</td><td>gemma-7b-it</td><td>60.0</td><td>76.1</td><td>60.9</td><td>50.7</td><td>69.6</td><td>42.9</td></tr><tr><td>codegemma-2b</td><td>gemma</td><td>2B</td><td>gemma-2b</td><td>25.6</td><td>39.6</td><td>28.3</td><td>41.2</td><td>53.6</td><td>4.62</td></tr><tr><td>codegemma-7b</td><td>gemma</td><td>7B</td><td>gemma-7b</td><td>53.9</td><td>76.7</td><td>56.5</td><td>38.0</td><td>69.6</td><td>45.4</td></tr><tr><td>codegemma-7b-it</td><td>gemma</td><td>7B</td><td>codegemma-7b</td><td>53.8</td><td>72.5</td><td>56.0</td><td>48.5</td><td>66.2</td><td>52.4</td></tr><tr><td>ada</td><td>gpt 3</td><td>?</td><td>Ø</td><td>?</td><td>43.5</td><td>24.3</td><td>21.5</td><td>?</td><td>0.6</td></tr><tr><td>babbage</td><td>gpt 3</td><td>?</td><td>Ø</td><td>?</td><td>55.5</td><td>23.3</td><td>18.8</td><td>?</td><td>0.7</td></tr><tr><td>curie</td><td>gpt 3</td><td>?</td><td>Ø</td><td>?</td><td>68.2</td><td>24.3</td><td>23.2</td><td>?</td><td>1.6</td></tr><tr><td>davinci</td><td>gpt 3</td><td>175B</td><td>Ø</td><td>35.9</td><td>22.8</td><td>34.3</td><td>21.4</td><td>48.0</td><td>12.1</td></tr><tr><td>davinci-instruct-beta</td><td>gpt 3</td><td>175B</td><td>davinci</td><td>40.9</td><td>18.9</td><td>39.9</td><td>5.4</td><td>49.6</td><td>10.8</td></tr><tr><td>text-ada-001</td><td>gpt 3</td><td>?</td><td>ada</td><td>?</td><td>42.9</td><td>23.8</td><td>23.2</td><td>?</td><td>0.4</td></tr><tr><td>text-babbage-001</td><td>gpt 3</td><td>?</td><td>babbage</td><td>?</td><td>56.1</td><td>22.9</td><td>23.3</td><td>?</td><td>0</td></tr><tr><td>text-curie-001</td><td>gpt 3</td><td>?</td><td>curie</td><td>?</td><td>67.6</td><td>23.7</td><td>25.7</td><td>?</td><td>0.6</td></tr><tr><td>text-davinci-001</td><td>gpt 3</td><td>175B</td><td>?</td><td>53.2</td><td>34.6</td><td>46.7</td><td>21.7</td><td>54.6</td><td>15.6</td></tr><tr><td>text-davinci-002</td><td>gpt 3.5</td><td>175B</td><td>code-davinci-002</td><td>75.7</td><td>64.9</td><td>62.1</td><td>47.8</td><td>65.5</td><td>47.3</td></tr><tr><td>text-davinci-003</td><td>gpt 3.5</td><td>175B</td><td>?</td><td>79.5</td><td>60.4</td><td>63.7</td><td>52.2</td><td>70.6</td><td>59.4</td></tr><tr><td>babbage-002</td><td>gpt 3.5</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>davinci-002</td><td>gpt 3.5</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td><td>?</td></tr><tr><td>gpt-3.5-turbo-instruct-0914</td><td>gpt 3.5</td><td>?</td><td>?</td><td>83.6</td><td>82.8</td><td>69.6</td><td>59.4</td><td>68.0</td><td>75.8</td></tr></tbody></table>
<table><caption>Table 3: List of chat LLMs included in the study. They appear in the similarity matrices (see Fig 9 10) in the same order as presented in this table. For proprietary LLMs, names are taken from their respective API. For all the other models, they were downloaded from the huggingface hub with the same name as given here. B and M stand for billions and millions, Ø for trained from scratch and ? means unknown or not officially communicated.</caption><thead><tr><th>Name</th><th>Family</th><th>Size</th><th>Parent</th></tr></thead><tbody><tr><td>gpt-3.5-turbo-instruct-0914</td><td>gpt 3.5</td><td>?</td><td>?</td></tr><tr><td>gpt-3.5-turbo-0301</td><td>gpt 3.5</td><td>?</td><td>?</td></tr><tr><td>gpt-3.5-turbo-0613</td><td>gpt 3.5</td><td>?</td><td>?</td></tr><tr><td>gpt-3.5-turbo-1106</td><td>gpt 3.5</td><td>?</td><td>?</td></tr><tr><td>gpt-4-0314</td><td>gpt 4</td><td>?</td><td>?</td></tr><tr><td>gpt-4-0613</td><td>gpt 4</td><td>?</td><td>?</td></tr><tr><td>gpt-4-vision preview</td><td>gpt 4</td><td>?</td><td>?</td></tr><tr><td>gpt-4-1106 preview</td><td>gpt 4</td><td>?</td><td>?</td></tr><tr><td>gpt-4-0125 preview</td><td>gpt 4</td><td>?</td><td>?</td></tr><tr><td>claude-2.0</td><td>claude</td><td>?</td><td>?</td></tr><tr><td>claude-2.1</td><td>claude</td><td>?</td><td>?</td></tr><tr><td>claude-3-haiku-20240307</td><td>claude</td><td>?</td><td>?</td></tr><tr><td>claude-3-sonnet-20240229</td><td>claude</td><td>?</td><td>?</td></tr><tr><td>claude-3-opus-20240229</td><td>claude</td><td>?</td><td>?</td></tr><tr><td>mistral-tiny-2312 (Mistral-7B-v0.1)</td><td>mistral</td><td>7B</td><td>Ø</td></tr><tr><td>mistral-small-2312 (Mistral-8x7B-v0.1)</td><td>mistral</td><td>8x7B</td><td>Ø</td></tr><tr><td>mistral-small-2402</td><td>mistral</td><td>?</td><td>?</td></tr><tr><td>mistral-medium-2312</td><td>mistral</td><td>?</td><td>?</td></tr><tr><td>mistral-large-2402</td><td>mistral</td><td>?</td><td>?</td></tr><tr><td>text-bison@001</td><td>palm</td><td>?</td><td>?</td></tr><tr><td>text-bison@002</td><td>palm</td><td>?</td><td>?</td></tr><tr><td>text-unicorn@001</td><td>palm</td><td>?</td><td>?</td></tr><tr><td>chat-bison@001</td><td>palm</td><td>?</td><td>?</td></tr><tr><td>chat-bison@002</td><td>palm</td><td>?</td><td>?</td></tr><tr><td>gemini-1.0-pro</td><td>gemini</td><td>?</td><td>?</td></tr><tr><td>gemini-1.0-pro-001</td><td>gemini</td><td>?</td><td>?</td></tr><tr><td>gemini-1.0-pro-002</td><td>gemini</td><td>?</td><td>?</td></tr><tr><td>gemini-1.0-pro-vision</td><td>gemini</td><td>?</td><td>?</td></tr><tr><td>gemini-1.0-pro-vision-001</td><td>gemini</td><td>?</td><td>?</td></tr></tbody></table>
C PHYLOLM ALGORITHM

**Data:** genes, N, LLMs

/* Estimate P
P ← array(LLMs, genes, vocab_size) #Zero filled array of given shape

for LLM in LLMs do

| for gene in genes do
|   for i ≤ N do
|     allele ← generate_4 characters (LLM, gene) #Generate 'allele'
|     P[LLM, gene, allele] ← P[LLM, gene, allele] + 1/N #Update P
|   end
| end
end

/* Compute similarity matrix S from P
S ← array(LLMs, LLMs)

for LLM1 in LLMs do

| for LLM2 in LLMs do
|   S[LLM1, LLM2] ← (Σ<sub>g∈G</sub> Σ<sub>a∈A<sub>g</sub></sub> P[LLM1, g, a]P[LLM2, g, a]) / (∑<sub>g∈G</sub> Σ<sub>a∈A<sub>g</sub></sub> P[LLM1, g, a]<sup>2</sup>) (∑<sub>g∈G</sub> Σ<sub>a∈A<sub>g</sub></sub> P[LLM2, g, a]<sup>2</sup>)
| end
end

/* Compute distance matrix D from S
D ← −log(S)

return S, D

**Algorithm 1:** PhyloLM
# D RESULTS REPLICATION ON THE CODE SET OF GENES

In the main text we discussed the results obtained on the math set of genes and analysed dendrograms (see Section 3.2.1). Here we discuss results obtained on the code set of genes about hyperparameter search for PhyloLM, ground truth tree reconstruction and dendrograms. The benchmark prediction score (Figure 5) already includes code benchmark. For more details about individual benchmarks and sets of genes, see Appendix F.

<figure><img src="image_8.png" /><figcaption>Figure 6: **Hyperparameters impact on distance matrices in the code set of genes.** (a) shows the variability of distance matrices for different number of genes *G* and number of probes *N* in the math benchmark. Each set of genes of specified size contains different and independent genes from the other matrices for a total of 8 distance matrix for each data point in the figure. (b) shows the distance to the high precision matrix made of 2048 genes and *N*=128 in the math benchmark. Errorbars represent the standard error of the mean.</figcaption></figure>

## D.1 HYPERPARAMETER INFLUENCE

We conducted the same experiments as described in Section 2.3. We first ran the variability analysis yielding the same results as in the math set of genes (see Figure 6 for code results and 2 for math set of genes): the variability decreases as *G* increases but *N* doesn't affect very much this value.

About the approximation of the high-precision matrix we get the same result as well: for each value of *N*, there is a value of *G* from which increasing *G* doesn't make the approximation very much better. One difference can be noted: with the math set of genes the approximation completely saturates while in the code set of genes it still improves a little bit. The same tradeoff of *G* = 128 and *N* = 32 can be found in this plot.

## D.2 PHYLOGENETIC TREE RECONSTRUCTION

When trying to reconstruct part of the Mistral Family we get similar results compared to the math set of genes (see Figure 7 for the code reconstruction and 3 for the math reconstruction). However, the root of the tree is not the right one. Indeed, the dendrogram algorithm, NJ (Saitou & Nei, 1987) constructs an unrooted tree that is then rooted for plotting. Here it failed to find the right root. Nonetheless, except the root choice, the structure of the tree is identical to the ground truth. That is also why we choose to plot unrooted trees in the paper as choosing a root for all the LLMs evolution is very artificial.

## D.3 DENDROGRAMS

Let's first look at the dendrogram (see Figure 8), one can notice very clear clusters for all completion families like in the math set of genes (see Figure 4). Indeed, all the llama branches are clustered together, Mistral, Gemma, Pythia, Gwen and Bloom models have very distinct branches in the phylogenetic tree. On the other hand, GPT-3, OPT and some Falcon models seem to share more than the other families like in the math similarity matrix. However, they still appear a little more different than in the math dendrogram. More details are given in the comparison of similarity matrices in Appendix E.
<figure><img src="image_9.png" /><figcaption>Figure 7: **Phylogenetic tree reconstruction on the code set of genes.** Left: ground truth concerning the relation of some LLMs of the Mistral family. Right is the reconstruction from the phylogenetic algorithm on the 'code' set of genes for the five latest models of this family ("leaves" of the phylogenetic tree) on which we run PhyloLM. Right: phylogenetic tree reconstructed by PhyloLM when given as input the 5 "leaves" models. The numerical labels (0:3) map the true common ancestors (on the right, "ground truth") to the inferred ones (on the left, "reconstructed"). It can be seen that the true and the reconstructed trees are very similar to the exception of the root that is 0 in the ground truth but 3 in the reconstruction. If we reroot the tree as having 0 as root we would get the exact ground truth like with the math set of genes.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: **Inferred phylogenetic tree of LLMs on the 'code' set of genes.** (a) completion models include 111 open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Leaves represent models and branches are colored according to model families. Llama models have been split by version of the pretrained model and the number of parameters. Length of lines from a model (leaf) to another represents the distance between the two models.</figcaption></figure>
## E SIMILARITY MATRICES COMPARISON BETWEEN SETS OF GENES

Let's now dive in the similarity matrices. It is important to understand that these matrices are a little noisy due to the value of *N* and *G* used (see Section 3.1). Estimating this noise is challenging as it may change from a LLM to another and is very difficult to represent in figures. By estimating variance on the smallest gpt models (ada, cabbage, text-ada-001, text-babbage-001, cabbage-002) we found the variance of the hyperparameters used in this study to be around 0.04 (see Figure 2a for the math set of genes and Figure 6 for the code set of genes). A difference of such a tiny value is barely visible with naked eye on a 0 to 1 scale (the scale in which similarity matrices are plotted) meaning that every visible difference of shade between models in the similarity matrix is significant. Of course, this is just an approximation, however, running such a statistical analysis on all models would be far too expensive in practice thus we stayed with this approximation in the paper.

Let's first dive into the completion models similarity matrices (see Figure 9 for the math set of genes and Figure 10 for the code set of genes). First we notice that the code similarity matrix is a lot brighter than the math similarity matrix. This is likely due to the fact that code is a very formatted language meaning less options are available to complete a context making all models appear a little bit more similar when sampling alleles. Now, let's dive into the matrix one family at a time.

In the next paragraphs, we will present observations from the different distance matrices and contrast the observations between the two sets of genes. Then, we will hypothesise what are the reasons why we observe these things and sometimes validate these hypotheses if the models are transparent enough about the training details. For proprietary models we will only hypothesise drawing a parallel with validated hypothesis on the open access models that are transparent enough.

**Llama** The same subfamily clusters can be found in both similarity matrices with, on one side, tigerbot-7b-base and tigerbot-7b-chat that appear closer to each other than the average llama models. Indeed, the second one is finetuned from the first one. Additionally, tigerbot-13b-base-v1, tigerbot-13b-base-v2, tigerbot-13b-chat-v1, tigerbot-13b-chat-v2, tigerbot-13b-chat-v3 and tigerbot-13b-chat-v4 form an additional cluster as they are based on each other (see Table 2 in Appendix B). We can also find openchat_v2 and openchat_v2_w in a same subcluster as they are trained on similar data as well as openchat_v3.1 and openchat_v3.2.

**Pythia** Very clear finetuning relationships can be found in Pythia in both sets of genes as the dolly models are based on the respective pythia model with the same size and each of the dolly models shows an above average similarity with their respective pythia model.

**OPT** This family is very interesting as in both sets of genes these models appear fairly close to each other. However, in the code similarity matrix, the family shows a lot of dissimilarity with all other models which is not the case with the math set of genes showing peculiar coding skills. While in the math matrix, OPT shares similarities with Pythia and GPT-3, in the code genome only the smaller gpt-3 family models including ada, cabbage, curie, davinci, text-ada-001, text-babbage-001, text-curie-001 and davinci-instruct-beta (but not text-davinci-001 that is finetuned from davinci-instruct-beta!) seem to share an above average similarity and all the other models look very different from OPT. The only information we have is that OPT training data include ThePile but we don't know much about GPT-3 models.

**Falcon** Falcon-RW-1 still shows similarities with the OPT family in both sets of genes but a little less in the code set of genes. Intriguingly, still in both matrices, the distance between Falcon-7b and Falcon-7b-instruct (finetuned from Falcon-7B) seems larger than the distance between Falcon-7b and Falcon-40b suggesting probably an intensive instruct finetuning on coding tasks as it is not found in the math similarity matrix. The same observation stands for Falcon-40b and Falcon-40b-instruct.

**Mistral** In the Mistral family, in both sets of genes, we find back the subfamilies used in the tree reconstruction: zephyr-alpha-7b, zephyr-beta-7b and docsgpt-7b-mistral showing a lot of similarity while on another hand we have openchat_3.5 with TenyxChat-7B-v1 and MedChat3.5 and finally OpenHermes-2.5-Mistral-7B with H4rmoniousAnthea and NeuralHermes-2.5-Mistral-7B. Additional subclusters can be found such as Mistral models and neural-chat models.
**Qwen** Qwen, in both similarity matrices shows very intricate structure but with a lower contrast. We notice a first subcluster for the original Qwen models with their finetunings in CausalLM. Each CausalLM shows a better ressemblance to the Qwen model from which they were finetuned. Then we have the 1.5 family showing a lot of similarity with their chat finetuned counterparts. The smallest model Qwen1.5-0.5B-Chat shows very little similarity to all other models in the matrix indicating maybe catastrophic forgetting to some extent during finetuning in both sets of genes. However, in the code set of genes it shows even more dissimilarity with Qwen1.5-Chat-1_8B while it is the closest model in the math matrix. We are not entirely sure as to what could be the origin of this observation.

**Gemma** In the Gemma family there is a lot of finetuning structure in both matrices. Indeed lots of white dots are far from the diagonal meaning several models show an unusual similarity with other models. Starting with gemma-2b and gemma-7b appearing very similar even more than their instruction finetuned counterparts (gemma-2b-it and gemma-7b-it). On the other hand, these finetuned versions appear to be very similar to the version 1.1 they were finetuned in. Lastly, codegemma-7b appear to be very close to gemma-2b and gemma-7b but codegemma-2b is less similar to them, probably because of its lower size that lead to forgetting to some extent.

**GPT-3** Finally in the GPT-3 family, still in all matrices, we notice a first cluster corresponding to the pretrained models ada, babbage, curie and davinci. Intestingly, davinci-instruct-beta an SFT finetuned model from davinci shows a lot of similarities to davinci in both sets of genes meaning it might have been finetuned on a low size dataset. Then their finetuned counterparts, text-ada-001, text-babbage-001, text-curie-001 and text-davinci-001 appear a lot farther from the pretrained models but we can trace back the model they were finetuned from from the similarity they share with the GPT-3 pretrained models. Then, in the math set of genes, text-davinci-001, text-davinci-002 and text-davinci-003 look vaguely similar but on the code genome, the last two appear extremely close and a little far from text-davinci-001. This might be linked to the fact that text-davinci-002 and text-davinci-003 have been finetuned from code-davinci-002 (trained on code) and not from davinci indicating different coding skills between the 1st version and the last 2 while all of them are fairly different in reasoning tasks (which was likely the object of their respective different finetuning). Going back to the similarities with OPT, we notice that, in coding tasks, text-davinci-001 doesn't show much similarities with OPT models while davinci-instruct-beta appear close to OPT. What is truely interesting here is that text-davinci-001 is either finetuned from davinci or from davinci-instruct-beta and this finetuning does not seem to share similarities with OPT on coding skills showing a potential emphasis on coding skills during this finetuning. Finally, babbage-002, davinci-002 and gpt3.5-turbo-instruct-0914 appear close to each other. While babbage-002 and davinci-002 might be pretrained, if we had to find a parent for gpt-3.5-turbo-instruct it would likely be davinci-002 as it is the closest model to it (but we don't have enough information to be sure of this).

Now that we have seen the completion models similarity matrix let's switch to the chat models (see Figure 12). These models essentially include proprietary models with very little to no transparency at all about the training details. As such we will have to completely guess the reasons for the observations we are going to make.

**GPT-3.5/GPT-4** We discern two clusters in the GPT family: before 11/06 and after and in both sets of genes. Indeed the newest versions of gpt-4 show a lot of similarity with each other but are extremely dissimilar to the previous versions that have a lot with each other. Something important must have happened at this point in the training methods at OPENAI.

**Claude** In claude the two generations of models are clearly discernible: claude 1 and 2 on one side and claude 3 on another side. Once more a big change has occurred in the training methods in between. Interestingly, the 3rd generation share a lot with bison and gemini models especially in the code context. More specifically with code-bison@001. This is not the case with the math set of genes. This may show some similarities in code training sets or data generation using another model.

**Mistral** In the Mistral API we notice a strong similarity between mistral-small-2402 and mistral-large-2402 that were probably trained on the same training set. On another hand most models appear close to the last two gpt models more specifically these two models released on 02/24. On the contrary, mistral-medium-2312 seems a lot closer to the previous version of gpt-4 and gpt-3.5 showing maybe a similar inspiration in the training set or data generation using a model.
**Palm** In the Palm family, we notice that text-bison@002 and code-bison@002 are almost identical suggesting a finetuning from one to another like it is the case in the gemma family (also from google) or that it is the same model.

**Gemini** Finally, models in the gemini family appear almost identical to each other except gemini-1.0-pro-002. Palm appear fairly close to Gemini in both context showing maybe a shared training set to some extent (both models are from google) but in the code context they appear very close to Claude 3 models (maybe a similar training set or data generation using a model).

<figure><img src="image_11.png" /><figcaption>Figure 9: Similarity matrix of completion LLMs on the math set of genes</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Figure 10: Similarity matrix of completion LLMs on the code set of genes</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Figure 11: Similarity matrix of chat LLMs on the math set of genes</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Figure 12: Similarity matrix of chat LLMs on the code set of genes</figcaption></figure>
F BENCHMARK RESULTS

In the main paper only MMLU and ARC prediction results were presented. Here we show all the 6 benchmark predictions for both gene sets (Figure 13 for the math set of genes and Figure 14 for the code set of genes). Statistics are reported in Tables (Table 4 for the math set of genes and Table 5 for the code set of genes).

<figure><img src="image_15.png" /><figcaption>Figure 13: Predictions of benchmark performances from the MLP compared to ground truth for every model(leave-one-out method) for the math set of genes. Each point is the predicted score of the model when the regressor is trained on data from all other models but the given point. For the sake of clarity, names of models have been omitted and replace by the color of models from the dendrogram plot. Lines represent a linear regression between predicted and true labels within each family.</figcaption></figure>
<table><caption>Table 4: Statistics on benchmark prediction for the math set of 'genes'. $\rho$ is the pearson correlation coefficient between the predicted labels and the true labels, p is the p-value for the student-test against 0 and N is the number of datapoint for this analysis.</caption><thead><tr><th>Benchmark</th><th>Family</th><th>ρ</th><th>p</th><th>N</th><th>Benchmark</th><th>Family</th><th>ρ</th><th>p</th><th>N</th></tr></thead><tbody><tr><td rowspan="10">arc</td><td>llama</td><td>0.78</td><td>&lt;0.001</td><td>21</td><td rowspan="10">hellaswag</td><td>llama</td><td>0.66</td><td>&lt;0.001</td><td>21</td></tr><tr><td>bloom</td><td>0.98</td><td>0.002</td><td>5</td><td>bloom</td><td>0.94</td><td>0.015</td><td>5</td></tr><tr><td>pythia</td><td>0.84</td><td>0.001</td><td>11</td><td>pythia</td><td>0.81</td><td>0.002</td><td>11</td></tr><tr><td>opt</td><td>0.94</td><td>&lt;0.001</td><td>8</td><td>opt</td><td>0.89</td><td>0.002</td><td>8</td></tr><tr><td>falcon</td><td>0.94</td><td>0.005</td><td>6</td><td>falcon</td><td>0.94</td><td>0.003</td><td>6</td></tr><tr><td>mistral</td><td>0.70</td><td>0.004</td><td>14</td><td>mistral</td><td>0.79</td><td>&lt;0.001</td><td>14</td></tr><tr><td>qwen</td><td>0.94</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.89</td><td>&lt;0.001</td><td>18</td></tr><tr><td>gemma</td><td>0.95</td><td>&lt;0.001</td><td>8</td><td>gemma</td><td>0.56</td><td>0.145</td><td>8</td></tr><tr><td>gpt</td><td>0.97</td><td>&lt;0.001</td><td>6</td><td>gpt</td><td>0.07</td><td>0.806</td><td>12</td></tr><tr><td>all</td><td>0.82</td><td>&lt;0.001</td><td>97</td><td>all</td><td>0.68</td><td>&lt;0.001</td><td>103</td></tr><tr><td rowspan="10">mmlu</td><td>llama</td><td>0.61</td><td>0.002</td><td>21</td><td rowspan="10">truthfulqa</td><td>llama</td><td>0.51</td><td>0.017</td><td>21</td></tr><tr><td>bloom</td><td>0.23</td><td>0.701</td><td>5</td><td>bloom</td><td>0.54</td><td>0.343</td><td>5</td></tr><tr><td>pythia</td><td>0.18</td><td>0.587</td><td>11</td><td>pythia</td><td>0.47</td><td>0.161</td><td>10</td></tr><tr><td>opt</td><td>0.31</td><td>0.450</td><td>8</td><td>opt</td><td>0.58</td><td>0.129</td><td>8</td></tr><tr><td>falcon</td><td>0.93</td><td>0.005</td><td>6</td><td>falcon</td><td>0.72</td><td>0.102</td><td>6</td></tr><tr><td>mistral</td><td>0.90</td><td>&lt;0.001</td><td>14</td><td>mistral</td><td>0.10</td><td>0.732</td><td>14</td></tr><tr><td>qwen</td><td>0.87</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.79</td><td>&lt;0.001</td><td>18</td></tr><tr><td>gemma</td><td>0.96</td><td>&lt;0.001</td><td>8</td><td>gemma</td><td>0.23</td><td>0.572</td><td>8</td></tr><tr><td>gpt</td><td>0.97</td><td>&lt;0.001</td><td>12</td><td>gpt</td><td>0.79</td><td>0.001</td><td>12</td></tr><tr><td>all</td><td>0.88</td><td>&lt;0.001</td><td>103</td><td>all</td><td>0.43</td><td>&lt;0.001</td><td>102</td></tr><tr><td rowspan="10">winogrande</td><td>llama</td><td>0.60</td><td>0.003</td><td>21</td><td rowspan="10">gsm8k</td><td>llama</td><td>0.31</td><td>0.169</td><td>21</td></tr><tr><td>bloom</td><td>0.91</td><td>0.029</td><td>5</td><td>bloom</td><td>0.94</td><td>0.013</td><td>5</td></tr><tr><td>pythia</td><td>0.92</td><td>&lt;0.001</td><td>11</td><td>pythia</td><td>0.58</td><td>0.058</td><td>11</td></tr><tr><td>opt</td><td>0.94</td><td>&lt;0.001</td><td>8</td><td>opt</td><td>0.76</td><td>0.026</td><td>8</td></tr><tr><td>falcon</td><td>0.93</td><td>0.005</td><td>6</td><td>falcon</td><td>0.90</td><td>0.012</td><td>6</td></tr><tr><td>mistral</td><td>0.90</td><td>&lt;0.001</td><td>13</td><td>mistral</td><td>0.63</td><td>0.019</td><td>13</td></tr><tr><td>qwen</td><td>0.87</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.30</td><td>0.215</td><td>18</td></tr><tr><td>gemma</td><td>0.55</td><td>0.149</td><td>8</td><td>gemma</td><td>0.82</td><td>0.012</td><td>8</td></tr><tr><td>gpt</td><td>0.91</td><td>0.011</td><td>6</td><td>gpt</td><td>0.77</td><td>0.003</td><td>12</td></tr><tr><td>all</td><td>0.72</td><td>&lt;0.001</td><td>96</td><td>all</td><td>0.66</td><td>&lt;0.001</td><td>102</td></tr></tbody></table>
<table><caption>Table 5: Statistics on benchmark prediction for the code set of 'genes'. $\rho$ is the pearson correlation coefficient between the predicted labels and the true labels, p is the p-value for the student-test against 0 and N is the number of datapoint for this analysis.</caption><thead><tr><th>Benchmark</th><th>Family</th><th>ρ</th><th>p</th><th>N</th><th>Benchmark</th><th>Family</th><th>ρ</th><th>p</th><th>N</th></tr></thead><tbody><tr><td rowspan="10">arc</td><td>llama</td><td>0.39</td><td>0.076</td><td>21</td><td rowspan="10">hellaswag</td><td>llama</td><td>0.70</td><td>&lt;0.001</td><td>21</td></tr><tr><td>bloom</td><td>0.95</td><td>0.012</td><td>5</td><td>bloom</td><td>0.94</td><td>0.016</td><td>5</td></tr><tr><td>pythia</td><td>0.84</td><td>0.001</td><td>11</td><td>pythia</td><td>0.52</td><td>0.095</td><td>11</td></tr><tr><td>opt</td><td>0.88</td><td>0.003</td><td>8</td><td>opt</td><td>0.93</td><td>&lt;0.001</td><td>8</td></tr><tr><td>falcon</td><td>0.97</td><td>0.001</td><td>6</td><td>falcon</td><td>0.90</td><td>0.014</td><td>6</td></tr><tr><td>mistral</td><td>0.35</td><td>0.210</td><td>14</td><td>mistral</td><td>0.09</td><td>0.739</td><td>14</td></tr><tr><td>qwen</td><td>0.87</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.83</td><td>&lt;0.001</td><td>18</td></tr><tr><td>gemma</td><td>0.18</td><td>0.655</td><td>8</td><td>gemma</td><td>0.15</td><td>0.714</td><td>8</td></tr><tr><td>gpt</td><td>0.84</td><td>0.034</td><td>6</td><td>gpt</td><td>0.15</td><td>0.637</td><td>12</td></tr><tr><td>all</td><td>0.76</td><td>&lt;0.001</td><td>97</td><td>all</td><td>0.50</td><td>&lt;0.001</td><td>103</td></tr><tr><td rowspan="10">mmlu</td><td>llama</td><td>0.67</td><td>&lt;0.001</td><td>21</td><td rowspan="10">truthfulqa</td><td>llama</td><td>0.71</td><td>&lt;0.001</td><td>21</td></tr><tr><td>bloom</td><td>0.84</td><td>0.072</td><td>5</td><td>bloom</td><td>0.65</td><td>0.230</td><td>5</td></tr><tr><td>pythia</td><td>0.08</td><td>0.794</td><td>11</td><td>pythia</td><td>0.39</td><td>0.256</td><td>10</td></tr><tr><td>opt</td><td>0.09</td><td>0.831</td><td>8</td><td>opt</td><td>0.27</td><td>0.516</td><td>8</td></tr><tr><td>falcon</td><td>0.88</td><td>0.020</td><td>6</td><td>falcon</td><td>0.71</td><td>0.106</td><td>6</td></tr><tr><td>mistral</td><td>0.49</td><td>0.068</td><td>14</td><td>mistral</td><td>0.09</td><td>0.753</td><td>14</td></tr><tr><td>qwen</td><td>0.93</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.68</td><td>0.001</td><td>18</td></tr><tr><td>gemma</td><td>0.50</td><td>0.206</td><td>8</td><td>gemma</td><td>0.77</td><td>0.024</td><td>8</td></tr><tr><td>gpt</td><td>0.95</td><td>&lt;0.001</td><td>12</td><td>gpt</td><td>0.80</td><td>0.001</td><td>12</td></tr><tr><td>all</td><td>0.79</td><td>&lt;0.001</td><td>103</td><td>all</td><td>0.58</td><td>&lt;0.001</td><td>102</td></tr><tr><td rowspan="10">winogrande</td><td>llama</td><td>0.48</td><td>0.024</td><td>21</td><td rowspan="10">gsm8k</td><td>llama</td><td>0.20</td><td>0.377</td><td>21</td></tr><tr><td>bloom</td><td>0.87</td><td>0.052</td><td>5</td><td>bloom</td><td>0.52</td><td>0.359</td><td>5</td></tr><tr><td>pythia</td><td>0.71</td><td>0.013</td><td>11</td><td>pythia</td><td>0.39</td><td>0.229</td><td>11</td></tr><tr><td>opt</td><td>0.90</td><td>0.001</td><td>8</td><td>opt</td><td>0.60</td><td>0.110</td><td>8</td></tr><tr><td>falcon</td><td>0.88</td><td>0.020</td><td>6</td><td>falcon</td><td>0.83</td><td>0.040</td><td>6</td></tr><tr><td>mistral</td><td>0.53</td><td>0.062</td><td>13</td><td>mistral</td><td>0.44</td><td>0.127</td><td>13</td></tr><tr><td>qwen</td><td>0.75</td><td>&lt;0.001</td><td>18</td><td>qwen</td><td>0.39</td><td>0.103</td><td>18</td></tr><tr><td>gemma</td><td>0.08</td><td>0.839</td><td>8</td><td>gemma</td><td>0.58</td><td>0.124</td><td>8</td></tr><tr><td>gpt</td><td>0.74</td><td>0.092</td><td>6</td><td>gpt</td><td>0.87</td><td>&lt;0.001</td><td>12</td></tr><tr><td>all</td><td>0.50</td><td>&lt;0.001</td><td>96</td><td>all</td><td>0.70</td><td>&lt;0.001</td><td>102</td></tr></tbody></table>
<figure><img src="image_16.png" /><figcaption>Figure 14: Predictions of benchmark performances from the MLP compared to ground truth for every model(leave-one-out method) for the code set of genes. Each point is the predicted score of the model when the regressor is trained on data from all other models but the given point. For the sake of clarity, names of models have been omitted and replace by the color of models from the dendrogram plot. Lines represent a linear regression between predicted and true labels within each family.</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>**Figure 15: Predictions from the logistic regression compared to ground truth for every model (leave one family out method).** (a) Scatter plot of the correlation between the true scores in six benchmarks (names are given on the left of the plot) against the predicted scores using our methods based on the genetic distance calculated on either the “code” (`*_C`) or the “math” (`*_M`) set of “genes” (the plot presents individual LLMs as points and linear regression lines; colors of the panel correspond to the colors of the benchmarks’ names). (b) Individual Pearson’s correlation coefficients of the correlation between true and predicted scores across 6 benchmarks x 2 set of genes (leading to 12 combinations). (c) Individual Pearson’s correlation coefficients of the correlation between true and predicted scores computed within each family and benchmarks (leading to 108 combinations). In (b) and (c) the horizontal line represents the mean, the dark vertical bar the standard error of the mean, the box delimitates the confidence interval and points are plotted within a probability density function.</figcaption></figure>
## G GLOBAL FIGURES

In the main paper we claimed that separating completion and chat models is important as they do not return the allele to the same gene. Indeed, many private models add additional tokens to the prompt such as message markers for user and assistant that will be concatenated to the gene such that the allele returned doesn't correspond to the same gene as completion models. We still plotted the similarity matrices (see Figures 16 and 17) and dendrograms (see Figures 18 and 19) to show that, indeed, it doesn't capture the families very well.

As explained in the main text we can see that the Mistral 7B model prompted in a completion manner doesn't appear close to the same model from the Mistral API prompted in a chat format. A possible improvement to PhyloLM would be to prompt all completion models in a chat format to see whether it bridges the gap between completion and chat models. However depending on the prompting technique, some models are finetuned to interact with a specific message token whereas others might be more familiar with another prompting technique which could bias the comparison.

Looking at the figures (see Figure 16 for the math set of genes and Figure 17 for the code set of genes), one can see that, indeed, the similarity between completion and chat models (the separation starting at the second half of the gpt-family) is very low except for text-bison@001 and the first few gpt-3.5 and gpt-4 models for both sets of genes (maybe even clearer for the code set of genes). This can be seen further in the dendrograms as the chat models are all on the same branch and, despite having a lot of structure when plotting only the chat models dendrogram (see Figure 4 for the math set of genes and Figure 8 for the code set of genes) here not much is to be seen. Indeed, one has to bear in mind that the phylogenetic tree vizualisation plots in a 2D space information coming from a 150 dimensional space and is influenced by the variance in all the models to be plotted. As such, plotting models that are too different will lead to a vague representation while plotting a very specific set of models can lead to very precise discrimination in the history of the models (see Figure 3 for phylogenetic reconstruction from the math set of gene and Figure 7 from the code set of genes). That is why we splitted both types of models as the technical limitations linked to the interaction with some proprietary LLMs doesn't make it possible to extract precise genetic information from these models biasing the completion models from which we have precise genetic information.
<figure><img src="image_18.png" /><figcaption>Figure 16: Similarity matrix with all models (completion and chat) at once on the math set of genes. All models included are shown in Appendix B. Purple lines split completion from chat models.</figcaption></figure>
<figure><img src="image_19.png" /><figcaption>Figure 17: Similarity matrix with all models (completion and chat) at once on the code set of genes. All models included are shown in Appendix B. Purple lines split completion from chat models.</figcaption></figure>
<figure><img src="image_20.png" /><figcaption>Figure 18: Dendrogram with all models (completion and chat) at once on the math set of genes. All models included are shown in Appendix B.</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Figure 19: Dendrogram with all models (completion and chat) at once on the code set of genes.
All models included are shown in Appendix B.</figcaption></figure>
## H PHYLOLM AND MODEL QUANTIZATION

Quantization of models has become very common in the LLM field as it makes it possible to run a very large model on a rather modest hardware. However this procedure relies on simplifying the inner computations of the model raising many concerns about whether the quantized version is as reliable as the original version. As such, we investigated the Qwen family of models with quantization. They provide a GPTQ 4 bit, GPTQ 8bit and AQW 4 bit version of all their chat models from the family 1.5 (Bai et al., 2023; Frantar et al., 2023; Lin et al., 2024). We computed the similarity matrix for all the models in the family (except the 32B version as all quantized models weren't online at the time of the study) and compared the similarity with respect to their 3 quantized versions. The similarity matrix between these models is shown in Figure 21 for the math genes and 22 for the code genes. The authors did not communicate on the hyperparameters used to quantize these models aside from the number of bits

Interestingly, in the Qwen 1.5 family release, the quantized version the closest to the original model is GPTQ 8bit followed closely by the 4bit and finally by the AWQ a lot farther. This observation stands for all the models in the family. However, intriguingly, the larger the model the less quantization seem to affect the distance to the original model (see Fig 20 - t(5)= 21.8 (p<0.001)).

<figure><img src="image_22.png" /><figcaption>Figure 20: Evolution of the similarity with the original model for all quantized Qwen1.5 Chat models. Each curve represent the evolution of similarity with the original model for each quantization method depending on the size of the model. All models included come from the Qwen1.5 huggingface repository including the quantized versions (.). Qwen1.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.</figcaption></figure>
<figure><img src="image_23.png" /><figcaption>Figure 21: Similarity Matrix between Qwen 1.5 Chat models and several quantized versions on the math set of genes.. For the sake of simplicity only the size of the models have been included in the legend for the original Qwen 1.5 Chat models and the quantization method as well as the size of the quantization for the quantized versions. All models included come from the Qwen1.5 huggingface repository including the quantized versions(). Qwen1.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.</figcaption></figure>
<figure><img src="image_24.png" /><figcaption>Figure 22: Similarity Matrix between Qwen 1.5 Chat models and several quantized versions on the code set of genes.. For the sake of simplicity only the size of the models have been included in the legend for the original Qwen 1.5 Chat models and the quantization method as well as the size of the quantization for the quantized versions. All models included come from the Qwen1.5 huggingface repository including the quantized versions (.). Qwen1.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.</figcaption></figure>
# I BIG DENDROGRAMS

Dendrograms contained in the main text do not include model names for the sake of clarity as most of them would overlap. For the sake of clarity, we include here large vectorial figures of these corresponding dendrograms with the names so that readers can zoom in to see better where are the individual models in the dendrograms. Figure 23 shows the math based dendrogram while Figure 24 shows the code based dendrogram.

<figure><img src="image_25.png" /><figcaption>Figure 23: **Inferred phylogenetic tree of LLMs on the 'math' set of genes with the names.** (a) completion models include all open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Llama models have been split by version of the pretrained model and the number of parameters. Names are very small but this is a vectorial figure: readers are encouraged to zoom in when reading from a computer.</figcaption></figure>
<figure><img src="image_26.png" /><figcaption>Figure 24: **Inferred phylogenetic tree of LLMs on the 'math' set of genes with the names.** (a) completion models include all open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Llama models have been split by version of the pretrained model and the number of parameters. Names are very small but this is a vectorial figure: readers are encouraged to zoom in when reading from a computer.</figcaption></figure>
# J OTHER SETS OF GENES

The main text presents an extensive evaluation of 2 sets of genes: one based on reasoning and the other on code. Choosing a set of gene is asking a question to the group of LLMs studied. In the main text we wanted to investigate the phylogeny of language models so we tested them on skills that have evolved a lot recently. Here we include results on other sets of genes by asking other questions like language spoken, chat interaction familiarity and finally another gene set about more general knowledge.

## J.1 CHINESE POETRY GENES

We wanted to test the language spoken by various LLMs so we used a set of genes extracted from huggingface - erhwenkuo/poetry-chinese-zhtw using the same pipeline as presented in the main text in section 2.2 but cutting around the 5th character instead of 20-100 as chinese characters are much more informative than latin characters. The similarity matrix show a very little contrast between models from the same family for except Qwen and Bloom, known to have been trained on Chinese. Interestingly, Mistral also shows little contrast as well as tigerbot models finetuned from llama (that do not seem to know Chinese) appear to be quite close to bloom models and its own tigerbot finetunings (see Figure 25). When visualizing the matrix with dendrogram the major axis in the dendrogram looks very linked to the ability to write Chinese with the right models in the Figure 26 being the ones that show the most contrast in the matrix. The left models show very little contrast and are not known to speak chinese.

## J.2 CHAT-STYLE GENES

We also tested genes from the reasoning gene set but with additional chat prompting to see if we can spot models that have been finetuned with a chat format. New genes have directly been obtained from the original math genes discussed in the main text by adding additional prompting around them:

> User:[Reasoning gene]

Assistant:

Results in Figure 27 show much variance within the families which appears to be a bimodal distribution. To better visualize it we plotted the dendrograms with colors of families but also from whether they are explicitly trained for chat-based interactions (see Fig 28). When looking at families we notice that are much more split in the dendrogram compared to previous results without the chat prompting (compare with Figure 4 - completion models). When looking at whether the models have been exposed to chat-based interactions during training we notice a more clear separation in the dendrogram. Indeed the bottom part of the plot only includes models not explicitly trained on chat based interactions while the top part is essentially composed of models trained for such interactions.

## J.3 WIKIPEDIA

Lastly we wanted to see if it was still possible to reconstruct the phylogeny of LLMs not using classical finetuning topics. We used a the wikimedia/wikipedia dataset from huggingface and reused the same pipeline as presented in the main text in section 2.2 to extract 'genes' from the dataset. The similarity matrix in Figure 30 shows patterns very similar to what we observed on math or code in the main text. Similarly, the dendrogram (see Figure 29) splits extremely well the different families showing that even more sets of genes can be used to study the evolution of LLMs.
<figure><img src="image_27.png" /><figcaption>Figure 25: Similarity matrix of some completion LLMs on the chinese poetry set of genes.</figcaption></figure>
<figure><img src="image_28.png" /><figcaption>Figure 26: **Inferred phylogenetic tree of LLMs on the chinese poetry set of genes.** All models included are completion models. Llama models have been split by version of the pretrained model and the number of parameters.</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>Figure 27: Similarity matrix of some completion LLMs on the chat version of the math set of genes.</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>Figure 28: **Inferred phylogenetic tree of LLMs on the chat version of the math set of genes with coloring from families and chat interaction training.** Left shows the dendrogram colored with family colors and the right plot shows the same dendrogram but with colors indicating whether they have been explicitly trained to be familiar with chat interactions. All models included are completion models. Llama models have been split by version of the pretrained model and the number of parameters.</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 29: **Inferred phylogenetic tree of LLMs on the chinese poetry set of genes.** All models included are completion models. Llama models have been split by version of the pretrained model and the number of parameters.</figcaption></figure>
<figure><img src="image_32.png" /><figcaption>Figure 30: Similarity matrix of some completion LLMs on the chinese poetry set of genes.</figcaption></figure>
# K ADDITIONAL ANALYSES OF GENE SETS' HYPERPARAMETERS

## K.1 GENE LENGTH

In section 2.2 we presented a pipeline to sample genes and explained that we truncated text coming from test benchmarks somewhere between the 20th and the 100th character. This choice is arbitrary and we provide here more precisions about the impact of the length of genes. We took a subset of LLM families included in the study and computed a gene set using the same pipeline as presented in the main text but with either only 5 characters, 200 characters or 1000 characters. We plot here the similarity matrices and phylogenetic trees (Figure 31) made from these genes. For short genes the matrix is a little darker than for longer genes. However, the similarity matrix doesn't seem to change very much with gene length from 20 to 100 characters long (main text configuration) and all 4 sets lead to good family clustering in the dendrograms. We conclude that the length of the gene doesn't impact very much the reconstruction and thus it is probably better to keep rather short genes in order to avoid unnecessary costs when running the algorithm.

## K.2 COMPLETION LENGTH

We also tested the impact of the number of characters to generate after each gene. Generating only 1 character will increase the likelihood for 2 models to generate the same thing (even if they meant something different but starting with the same character) thus the similarity matrix will be a lot brighter in general as models appear more similar. On the other hand, letting the LLM generate 20 characters is a lot more likely to produce generations that do not match leading to a similarity matrix a lot darker.

A similarity matrix with low contrast (too bright or too dark) makes all the differences between models in a very low numeric range. Knowing these matrices are a little noisy (see Figure 2b in section 2.2) it is better to have a contrast as high as possible in order to better visualize similarities between models.

Thus we computed the RMS contrast of the matrix (standard deviation of the similarity matrix including all the parameters but the diagonal) and plotted its value from 1 character long completion to 19 characters long completions. The results in Figure 32 show an optimal contrast for a completion length somewhere between 2, 3 and 4 characters.
<figure><img src="image_33.png" /><figcaption>(a)</figcaption></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /><figcaption>(b)</figcaption></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /><figcaption>(c)</figcaption></figure>
<figure><img src="image_38.png" /><figcaption>**Figure 31: Impact of the number of characters in the gene compared in each generation on similarity matrices** Similarity matrices only include Llama, Pythia, OPT, Mistral and Qwen families. (a) is the similarity matrix for only 5 character long genes (b) is for 20 to 100 characters long genes (like in the main text), (c) is for 200 characters long genes and finally (d) shows the similarity matrix for 1000 characters long genes.</figcaption></figure>
<figure><img src="image_39.png" /><figcaption>(a) Similarity matrix for 1 character long completions (b) similarity matrix for 4 character long completions</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>(c) similarity matrix for 19 character long completions between 1 and 19 characters.
(d) Evolution of the RMS contrast for all lengths be-</figcaption></figure>

<caption>**Figure 32: Impact of the number of characters compared in each generation on similarity matrices**

Similarity matrices only include Llama, Pythia, OPT, Mistral and Qwen families.</caption>