PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks
Authors: Nicolas Yax
Pierre-Yves Oudeyer
Stefano Palminteri
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
phylogeny
benchmark
Primary area: interpretability and explainable AI
TL;DR: This paper introduces PhyloLM, a method that adapts phylogenetic algorithms to Large Language Models to analyze their relationships and predict performance in benchmarks.

Abstract: This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture  known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for  a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development,  relationships and capabilities, even in the absence of transparent training information.

Published as a conference paper at ICLR 2025
PHYLOLM: INFERRING THE PHYLOGENY OF LARGE LANGUAGE MODELS AND PREDICTING THEIR PERFORMANCES IN BENCHMARKS
Nicolas Yax
Pierre-Ye Yves Oudeyer*
LNC2, INSERM, Paris, France DEC, ENS, Paris, France Inria, France
Inria and University of Bordeaux, France
nicolas.yax @ ens.esssse ens.psl.eu
Stefano Palminteri*
LNC2, INSERM, Paris, France DEC, ENS, Paris, France
equal contribution
ABSTRACT
This paper introduces PhyloLM,, method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for time and cost- effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning. we propose and validate tool to evaluate L.M development, relationships and capabilities, even in the absence of transparent training information.
INTRODUCTION
The Large Language Models (LLMs) landscape vast and rapidly expanding, comprising both private and open-access models. Each day few hundreds of new language models are created on the huggingface hub among which most will not benchmarked, and a small minority are transparent about the training details. Evaluating these models presents challenges due to the sheer volume and the complexity of assessing their true capabilities. The evaluation methods used today mostly rely on multitude of benchmarks, each focused specific domains like reasoning or question-answeriin answering (Chollet 2019: Hendrycks et al., 2021 Srivastava et al., 2023). However, tracking [.LMs evolution and progress using benchmarks presents inherent limitations, including the fact that they are rather domain-specific meaning that to get a full picture of a model' 's capabilities one has to run multiple costly tests that are prone contamination (Chang et al., 2023 Deng al. 2023 Liang et al. 2023). Moreover, the opacity of algorithmic and training data specifications in many models, adds further complexity and constraints to monitor progress in LLMs (Liao & Vaughan 2023).
Our approach stems from the observation that most of the newly released models are not created
ex-nihilo (from scratch). In fact, they rather inherit features from existing ones, such as training data or initial weights. We reasoned that we could therefore think about L.LMs development as
code: ht http:s////LLaa...ooooneeeaaaaaaa notebook: https://ccles research www..h.ooooooeoooo com http////./e////aa.eeeeerrr.....
9J8
bfJRYcd Puspccoy

---

Published as a conference paper at ICLR 2025
an "evolutionary" process and therefore study their relationships and functional properties with conceptual and quantitative tools borrowed from genetics.
In the field of Phylogeny, algorithms have been developed that reconstruct evolutionary trees to understand evolutionary relationships among species (Takezaki & Nei 1996). The idea of applying these methods, initially developed for biology. to cultural artefacts not new. Previous studies yielded particularly useful insights into the evolution of popular tales, languages, or craft assemblages
(Atkinson al., 2008; Dawkins 1976 d Huy 2013 Gray et al. 2010; Tehrani & d' d'Huy, 2017]] Tehrani & Collard 2009). We hypothesize here that LLMs, which are a new kind of cultural artefact (in the sense that they are productions of humans that convey information about the culture of their creators and users), may also be studied using similar tools.
Thus, we here apply a conceptually similar approach to LLMs and, by doing so, we make several contributions. In a first contribution, we introduce an algorithm, PhyloLM inspired by simplified phylogenetic model, but specifically tailored for Large Language Models (LLMs), which core idea to consider that generated tokens are to contexts what alleles are to genes in genetics. This analogy makes it possible to apply algorithms from the geneties framework to LLMs and to generate distance matrices and dendrograms. In addition to presenting the underlying theory, we also explore the hyperparameters of our algorithm to strike a balance between precision and computational efficiency.
In second contribution, confirm that PhyloLM is capable of correctly retrieving known relationships between LLMs and overall correctly capturing models families and sub-families, Our analysis primarily focuses
analyze the resulting phylogenetic trees ("dendrograms" ") and
open-access model families (Llama (Touvron et al.] 2023a b). Mistral (Jiang et al.) (023). Bloom SeeSScienceWokshhhe et al. 2002)), Pythia (Biderman et al.) 2023), Falcon (Almazrouei et al. 2023
OPT (Zhang et al. 2022). Qwen (Bai et al. 2023) and Gemma (Team et al. (024bb families), where ground truth information is available, but also provides insights into Linettunig relationships for proprietary models (GPT-3 (Brown et al. (020)) 3.5 (Ouyang et al. (2022) (OpenAl et al. 2023). Claude, Palm (Chowdhery et al.] 2022)) and Gemini models (Team al. 2024a)) ). Finally, in our third contribution, we examine whether phylogenetic distance can also be used to predict performance in several benchmarks, thus showing that the utility of PhyloLM extends to the assessment of functional properties of LLMs.
To sum up, our study illustrates the potential of leveraging methods from genetics to understand how models evolve, shedding light on their relationships and functional capabilities in relatively cost-efficient manner, even in the absence of transparent training information and also without direct access to the model.
E Eees, P((al)) V E EOEA, Pi(alg)))( EXEr, Pa(alg)
P.P.PP)
(I)
Equation 1: Similarity computation with P: and P3 two populations seen as probability distribution of alleles a given gene g estimated empirically in the selected populations. G is the set of genes considered and Ag the set of possible alleles for this gene and matrix S is the similarity matrix (bounded in [0. In genetics people tend to use a distance matrix D to plot dendrograms derived from the similarity matrix with this formula D(P1, P2) = log( S P2)) (Takezaki & Nei 1996). Seen from the autoregressive LLM framework, 'populations' are LLMs, 'genes' are contexts and 'alleles' are the different tokens in the vocabulary: P(alg) = LfM(
METHODS
2.1 TRANSLATING PHYLOGENETIC ALGORITIMMS TO LLMS
In the current landscape, LLMs predominantly operate autoregressive basis, wherein they learn the conditional probability denoted as (tcc). Here, LLM represents the probability learned by the language model, signifies token, and denotes the context in which to sample token Transposing genetic methods to LLMs involves establishing analogies for the elements of the phylogenetic analysis, namely genes, alleles, and populations. Drawing parallel with the notation for
2

---

Published as a conference paper at ICLR 2025
Gene /
Allele
Phylogenetic tree
Biological species
Species,
Species;
Species,
Sequencing
MHC
DQA1
BE3HAV
(s3735007
Genetk (Nei) distance
X.%
Large Language Models
Prompt/ Completion
x
(272
Phylogenetic
LLM,
M
LLM,
Probing
0
def
hune
Figure Analogy between running human genetic studies and LLMs genetic studies. The first stage consists in selecting genes (for both humans and LLMs).. Then alleles are collected for each individual in the population and will be used compare the populations (either populations of humans or LLMs seen populations). Finally these data go through the Nei distance computation (Takezaki & Nei 1996) that returns a distance matrix that can then be turned into dendrograms using the NJ algorithm (Saitou & Nei 1987) in the same way for both humans and LLMs.
populations in the Nei genetic distance (see Equation I Takezaki & Nei 1996), (a(g) with an
allele and g gene, we propose that LLMs play the role of populations (i.e., the set of the individuals belonging to given population); contexts (or "prompts") ") are aligned to genes (i.e., portions of DNA); finally, tokens align with alleles (i.e., variants in the DNA sequence).
To substantiate this analogy, consider that, in the realm of genetics, populations are conceptualized as probability distributions of DNA, represented by P(alg), where a stands for specific alleles at gene locations. Geenesppecffi alleles abstract statistical construct that is the population, akin to context-specifie tokens are probabilistically generated from Large Language Models, expressed as (LM((t)) (t being token likely to follow text c). The generated text can therefore be seen as a thread of DNA, comprised of tokens (alleles) sampled in contexts (genes) according to probability distribution defined by the LLM.
then considered to be probabilistically drawn from the
To elucidate this crucial point, consider a tokenized text sequence: 'I' '_like' "chocoo' 'late'. This sequence can be analogous to a DNA thread represented as '[_ikee_hhe chocolate', Breaking it down, the allele I corresponds to the gene (empty text), _ike aligns with the gene I, _chooo associates with the gene like, and late is linked the gene I like choco. Now, consider another individual represented by 'I' _prefer' "_ice" cream'. These two individuals share exactly two genes: for which they possess the same allele I, and the gene I, for which they have distinct alleles (_like and prefer). They do not share any further genes, as their prefixes diverge beyond this point.
The algorithm is illustrated in Figure D The initial step involves collecting model outputs to contexts (genes). Given a set of LLMs, set of geness', and the specified number of individuals in each population (i.c., the number times the model is queried on each gene refered to as the number of probes) as N, the models are queried for single token N times. This process generates the matrix P. which serves as an approximation of P(alg), the proportion of the population with allele a to gene g. Subsequently, based on this approximation, the similarity matrix S is computed using the Nei genetic distance formula (Takezaki & Nei 1996) depicted in Equation] ] The pseudo code of PhyloLM can be found in Algorithm in Appendix
2.2 CHOICE OF THE SET OF GENES
The implementation of phylogenetic algorithms requires selecting specific genes that show enough evolutionary changes among the species studied differentiate them, while still retaining enough
3

---

Published as a conference paper at ICLR 2025
similarity to trace relationships between closely related species (Grünwald al. 2017). If these genes mutate too quickly and are completely altered between similar species, they will not provide useful information about their evolution. Conversely, if they too stable and show no changes across the species being considered, they are also not informative. These genes must strike a balance between stability and variation among the species studied.
That is why we need to carefully select genes (i.e., prompt contexts) that could show a moderate variance between LLMs. Recent LLM development focused lot on instruction tuning, reasoning and coding (Brown et al. 2020 Chiang et al. 2023; OpenAl et al. 2023 Taori et al., (023). Selecting contexts on these topics might show relevant variance between generations of language models as well as finetuning refinements that improved these models on these specific topics.
Furthermore, contexts ('genes') which are very likely to belong to the training data of these LLMs can suffer from contamination issues and generate very low variance] To obviate this issue, we used contexts (or 'gene' set) taken from recent test benchmarks because, in principle, LLMs shouldn't t be trained on this data. To further assess the robustness of our approach and study the impact of the choice of the set of genes's, we took our contexts from two different test sets: open-web-math (Paster et al. 2023) and MBXP (Athiwaratkun et al. 2023). They address different capabilities of LLMs: reasoning and coding, respectively, which are very relevant in recent LLM-related research and are therefore likely to deliver useful results.
The exact selection of contexts from the benchmarks consisted of randomly and uniformly selecting lines from the solution column in the datasets and truncating the text to leave it open for LLMs to complete the sentence. To decide the length of the contexts we need once more to think about making "genes" show a moderate completion variance. If the context is only a few tokens long it may not be informative enough for LLMs to understand the topic of the context (that is relevant for the recent evolution of language models as discussed above) but also follow the logics of the text that would constrain the generation. On the other hand, making it hundreds of tokens long will induce additional costs without necessarily improving the variability balance. That is why we decided to truncate randomly and uniformly between the 20th and 100th characters in each text (5 to 30 tokens approximately). 'Gene" examples are shown in Appendix A More details about the impact of the gene length can be found in appendix K.I
2.3 SELECTION OF THE HYPER-PARE PARAMETERS OF THE DISTANCE MATRICES
We devised two complementary analyses estimate the right hyperparameters to rum PhyloLM The hyperparameters are the gene' set, the number of probes and sampling parameters from the LLM (see Appendix (B) Testing the gene set is more difficult as testing thousands of different combinations of genes would come at a very expensive cost. Thus we limited ourselves at 2 parameters of the gene set : the topic (math and code in this paper) and the size of the gene set G. In this section we will investigate the impact of G and N in the math gene set, the results for the code gene set are in Appendix
First we investigate how G and N affect the variability of the distance matrix, namely how much the similarity matrix changes between different estimations. We focus on similarity matrices (the matrix S in Equation 1 instead of distance matrices at this point as they are bounded in [0,1] making them lot easier to plot and compare. Then, once the variance is controlled, what combination of G and N approximate reasonably well very high G' and N' distance matrix.
To assess the impact of the number of contexts ('genes') ') G and the number of probes/individuals N for each dataset, the algorithm was executed across range of gene set sizes G (varying between 16 and 256 genes per run) and individuals N (ranging from 8 to 128) building similarity matrices. This optimization process, aimed testing the best values for the algorithm hyperparameters, is particularly computationally expensive. Therefore it was only run on the 5 smallest OPENAI models bada.babbage,text tewpaaaaeeet-------eeea
order to minimize the costs. Thus similarity matrices in this section are 5 x 5 making it an estimate of what could be a larger distance matrix at a very low cost.
babbaee--0- and babbage-002),,
'To understand this point, imagine using "May the force be with" as context. All models will complete this sentence with "you", thus making impossible establishing distance matrices between them
4

---

Published as a conference paper at ICLR 2025
To investigate the variability of PhyloLM for different combination of hyperparameters, we composed 8 sets of genes of size G, each with different genes. Each set of gene is probed N times to build similarity matrix Sg,N. E [[0, 7l]] representing the independant set of genes of size G used to generate the matrix with N probes. A variance computation over this set of matrices is finally performed yielding a matrix V containiig the variance of each distance between 2 models: : VG.N
E (So.N, (5 Ej So.Ns)) The square operator is applied coefficient by coefficient. The
final variability score is the mean value of the coefficients in the matrix vG,N l (V
Then we investigated the impact of these hyperparameter' when trying to approximate a high precision matrix. For this purpose, compute the variance around a very expensive distance matrix So N' with G" = 2048 and N' = 128. The gene set for the high precision matrix is independent from the lower size set of genes used estimate The formula to compute this variance around the high precision matrix is G.N"
G.N 2 matrix o' G.N = μ (VV''.n))
1 Ez (Sa.N. Se,,N: The final metric is the mean value in the
2.4 ALIGNMENT OF THE RESULTS ACROSS DIFFERENT TOKENIZATION
In situations where models do not share the same tokenizer, comparing only the first alleles generated can pose challenges. For instance, if the context is "The president of the US is Joe," and one model could complete with "Biden" in one token while another could complete with "Bi" "den" in two tokens they would be considered different alleles while both LLM meant the same completion.
To mitigate this issue of tokenizer alignment, a proxy approach was employed by only using the first 4 characters of the generated text instead of the first token. Practically, each model was instructed to generate at least 4 tokens (tokens are at least character long) and the comparison focused on the first 4 characters in the concatenation of these tokens. In the previous example, the word "Biden" generated in one token or in two ("Bi" and "den") would have been considered as the same response, because the first 4 characters ("Bide") constitute the same 'allele', despite having being tokenized differently. In other words, we are retokenizing the text using a tokenizer with a vocabulary of words that are 4 characters long, and then comparing the first token of the generated text with this new tokenization scheme. An example of the results of such a proxy approach is presented in Appendix Further details about why 4 characters efficient are discussed in Appendix K.2
2.5 VISUALIZATION OF THE RESULTS
From distance matrix obtained by the phylogenetic algorithm it is usual plot dendrograms representing a possible evolution between the entities in the distance matrix. For this purpose many different algorithms exist and we chose the Neighbour Joining (NJ) technique (Saitou & Nei 1987) for its simplicity. efficiency and being a common choice in genetics. We plotted unrooted trees as they are easier to make figures that fit in a paper and are more adapted to L.MM evolution than rooted ones. The analysis of the resulting dendrograms also allowed us to validate the capability of our algorithm to predict actual relationship between LLMs in cases where the ground truth is known.
2.6 PREDICT BENCHMARK SCORES FROM GENETIC DISTANCE
We explored whether genetic distance can predict model performance by using logistic regression to estimate benchmark scores of large language models based on their similarity to other models. Due to the high dimensionality of the similarity matrix, we reduced the input dimensions 15 using Independent Component Analysis (ICA), resulting in 15 parameters to learn from approximately 100 data points per fit. We then applied a sigmoid function to the output to scale the predictions between 0 and I, corresponding to benchmark scores ranging from 0% to 100%. Since benchmark scores can be highly correlated within a family of models, we employed leave-one ceave-one-family-ee taee-ffmllyouu method (a)). This involved training the regressor on all families but one and testing it on the
(see Figure (a)). excluded family. A Mean Squared Error loss was used with an Adam optimizer (learning rate of 10-3).
5

---

Published as a conference paper at ICLR 2025
N
0.12
8
0.12
8
16
16
Variance
32
Variance
32
0.08
64
0.08
64
128
128
0.04
0.04
0.0
16
128
16
64
128
256
(a) Variability distance matrices
(b) Distance to the high precision matrix
Figure 2: Hyperparameters impact on distance matrices in the math set of genes (a) shows the variability of distance matrices for different number of genes G and number probes N in the math benchmark. Each set of genes of specified size contains different and independent genes from the other matrices for total of distance matrix for each data point in the figure. (b) shows the distance to the high precision matrix made of 2048 genes and N=128 in the math benchmark. Errorbars represent the standard error of the mean.
We tested the benchmarks available on the hugging face open Ilm leaderboard which includes MMLU, ARC, Hellaswag, TruthfulQA, Winogrande and GSM8k (HuggingFaceH4) and only included open access models for which the scores are available on the leaderboard. Thus we didn't include proprietary models in this study as, explained in later sections, distance computation is slightly biased for these models and benchmark scores are not obtained in the same conditions as in the leaderboard (number of shots, CoT, ...). The benchmarks used for the 'gene' set were distinct from these benchmarks to avoid any type of contamination between the 'alleles' used to generate genetic distances and the performance of the models in the considered benchmark tasks.
EXPERIMENTS AND RESULTS
3.1 WHAT IS THE IMPACT OF HYPERPAR AMETERS ON THE DISTANCE MATRIX?
We first ran the hyperparameters' optimization process explained Method 2.3 and plotted the results in figure lafleft side. This graph shows a clear decrease in the variability the number of 'genes', G grows with almost no effect from N. This is interesting of 'genes'' doesn't 't appear change the similarity matrix as long as there are enough of them (at least in the open-web-math and mbxp dataset see Appendix D for the results on the code set 'genes').
seems that having different sets
However this method doesn't 't make it possible find good N, indeed, the probability for two models to generate the same token in the same context in only one try is quite low. Therefore, a very low N will make all models appear particularly different making the similarity matrix look like the identity matrix yielding unsatisfactory results despite having a low variance. Thus having N high enough is required to get a useful similarity matrix and need find better metric but how to choose it
We have just seen that G monitors the variability of the matrix (variability parameter), thus similarity matrix with a very high G should be particularly stable across different sets of genes. We then compared modestly parametrized similarity matrices study how hyperparameters G and N influence the difference between a lower precision matrices to high precision matrix on average (see Methods 2.3 for the computational details). This new metric should penalize having a low N leading to similarity matrices close the identity matrix and may yield more satisfying results.
As shown in Figure 2b] precision matrix, this time, the number of probes is also very important. Indeed, for each value of N, the performance saturates from some G value making less and less improvement when G increases. Thus, this figure gives optimal G for given N in order approximate the high precision matrix efficiently with a low cost. The total cost of the algorithm in tokens being proportional to G x N, we found good tradeoff' between variance and precision around G 128 and N 32.
while increasing the number of genes still seems to approximate better high

---

Published as a conference paper at ICLR 2025
The estimated cost to run the algorithm per model is therefore 128 genes 32 probes 4096 queries of 2 20 tokens. As point of reference, conducting the MMLU benchmark requires around 14,000 queries significandy longer prompts (< 70 tokens each), making PhyloLM approximately 10 times less expensive in terms of the number of tokens required.
Ground truth
Reconstructed with PhyloLM
pophyr-Pe 7b- beta
dacogpt-7--matrrl
docsgpn 7h-mistal
NeuralHermes 2.5 Mistral-78
Nauuraltermeu-2.- Mistral-78
2
Mistral-
Openttermes-
2.5 Mistral-78
InfmmoniousAnthee
H4mmoniousAnttic
MedChat1..5
MedChats5 5
Openchat
TesysChet-7B-vl
TenysChatt7BB-vt
"roots"
"leaves"
branch length
Figure 3: Phylogenetic tree reconstruction. On the left it is shown the ground truth concerning the relation of some LLMs the Mistral family. Right the reconstruction from the phylogenetic algorithm on the 'math' set of genes for the five latest models of this family ("leaves" of the phylogenctic tree) on which we run PhyloLM. On the right, it is shown the reconstructed phylogenetic tree PhyloLM on the 5 "leafs" models. The numerical labels (0:3) map the true common ancestors (on the right, "ground truth") )) to the inferred ones (on the left, "reconstructed") It can be seen that the true and the reconstructed trees are topologically equivalent
3.2 CAN WE TRACE BACK THE GENEALOGY OF LLMS USING TOOLS FROM GENETICS?
We first examine the results of PhyloLM by analyzing the resulting phylogenic trees (materialized as dendrograms). However, before dwelling into the results, an important point to understand is that, in genetics, branches in the tree show probable speciation events that occured in the past, when from an extinct common ancestor, two (or more) current species (leaves of the tree) emerged. When looking at LLMs, common ancestors' are not extinct, but rather among the studied 'populations', Take for instance Mistral 7B that is the common ancestor of OpenChat3.5 and Zephyr 7B Alpha, but still included in our analysis. Oblivious of this difference, the dendrogram plotting method will put all models at the 'leaves' of the tree, while, in fact, some of them (such as Mistral 7B) should be at speciation node. As such, without additional information about which model is at a node, it is difficult interpret them in the same way as in geneties. Without this important phylogenetic assumption, one has to bear in mind that what matters (and should be compared with the ground truth) is their relative distance and position when evaluating the dendrograms resulting from the phylogenetic analysis of LLMs. Indeed the distance between two models is represented by the distance from their respective leaves in the dendrogram.
To investigate the capabilities of PhyloLM, let's first start by respecting this assumption by looking set of 9 models from the Mistral family whose relationships are known because transparently disclosed by their creators. Out of these 9 models, 5 are leaves in the ground truth dendrogram (Arc53 2023 mmlabonne| 2023 Tenyx 2024 Ullah| 2024 Vallego 2024). Running PhyloLM on these models getting the distance matrix between them and finally plotting the NJTreee we perfectly get back the ground truth phylogenetic tree (see Fig B) validating the method. These rooted trees are not necessarily very stable the NJ algorithm makes
unrooted tree of the evolution but then has to choose the root. In Appendix D we show that, on the code genome, the root has been mistakenly attributed to model 3 while the structure of the tree is right. That is why prefer to plot unrooted trees in the rest of this paper.
3.2.1 GLOBAL DENDROGRAM
LLMs: open-source vs private, completion vs chat Now let's drop the assumption of not having 'common ancestors' in the set of LLMss. The LLMS we are investigating here include 111 open
7

---

Published as a conference paper at ICLR 2025
opt
pythia
Btoo
(akoon
mistrel
1:130
pahn
complecion
[b] chat
gamrin
Figure 4: Inferred phylogenetic tree of LLMS the 'math' set of genes. (a) completion models inlcude all open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Llama models have been split by version of the pretrained model and the number of parameters.
access models spanning from 70M to 176B parameters and 45 closed LLMs. Most modern LLMs are only accessible through chat API which naturally adds new tokens to the prompt such as chat messages markers biasing the completion of the given 'gen'.. This can strongly influence PhyloLM the algorithm will compare 'alleles' that do not correspond to the same 'gene'. As such we call completion models LLMs that were accessed in a way that can generate a completion to a very specific sequence of tokens without adding more tokens. All the 111 open access models we included in this study were accessed this completion setting but among the 45 proprieraty models we only considered 14 of them be completion models (see Appendix B for more details). That is why we split the LLMs and investigated them in 2 groups: completion models (to show the capabilities of PhyloLM when run in good conditions) and the others on which we suspect additional prompting manipulation. In both classes of models we found that our algorithm was largely capable of clustering LLMs into their original families, with only few specificities discussed below. Dendrograms for both model classes are in Figure 4
In the completion group of models we notice very clear Llama clusters separating the family from other families but also more fine grained level, subfamilies of llama linked to the version of the models and their respective sizes. Similarly clear cluster appear for Mistral, Qwen and Bloom. The other families such Falcon, OPT, Pythia and GPT 3 are more mixed with each other and indeed we know that OPT, Pythia and Balcon-RW-18 (the one the closest to OPT in the tree) were trained each their version of the Common Crawl dataset and thus share a similar training set. Lastly, some GPT-3 models (ada, babbage and curie) appear to be close to this OPT,,ythhia and Falcon-RW cluster showing they may have been trained version of the CommonCrawl as well. On the other hand, GPT-3.5 completion models including fext-davinci-009 and text-davinci -at-iivinn seem to share more with Falcon than other models while davinci-002 babbage-002 and spt-3.5-turboinnstrus look more related to Qwen and more precisely its CausalLM finetuning. It is important to understand that dendrograms in LLMs are just a visualisation tool, much more details can found in the similarity matrix shown in Figure Appendix shows dendrograms with model names of the models (see Figure (3).
In the chat models group, we also find lot of structure Palm and Gemini models are the same branch, Gemini seems to be further improvement on Palm is further on branch (and indeed they are both from Google showing maybe a sharing of their training data) while claude has its own branch and Mistral / GPT-3.5 and GPT-4 models show some similarities. Dendrograms with model names provided in Figure 23 in Appendix
Additional figure are available and discussed in Appendix: similarity matrices are Figure (Appendix EE) Code results are in Appendix [D] with the dendrogram in Figure and the similarity matrix in Figure IT,, Additional mixed class figures are in Appendix G Figure (math), Figure (code), and global similarity matrices in Figures T6land 17
8

---

Published as a conference paper at ICLR 2025
1.0
1.0
LLM families the training set Mistral, Liama, Gemma, Qwen, Bloom, Pythia, falcon, and GPT
opt
pythia
bloom
0.8
0.8
qwen
felcon
Predicted scores
Predicted scores
0.6
0.6
0.4
0.4
0.2
0.2
mistral
gemima
llama
gpt
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
True scores
True scores
(a) Fitting scatter plot
(b) Predictions all families
Figure 5: Predictions from the logistic regression compared to ground truth for every model (leave one family out method) on ARC benchmark. (a) Scatter plot showing the fitting of the logistic regression all models but the OPT family (in grey) and the prediction of OPT performance by the regression (in red). (b) Predictions from the logistic regression for each family. To predict a family, the regressor fits all the other families finally predict the of the models from the remaining family (leave one family out method see (a)).
3.3 CAN WE INFER MODEL CAPABILITIES FROM THE GENETIC DISTANCE?
We then investigated whether the genetic distance metric be used predict the abilities of lan- guage models. As such we used the benchmark scores from the Huggingface open L.MM leaderboard. The results indicate that the prediction correlates with the score of the models (Figures (b) and 15 (a)). Indeed, we found that the Pearson's 's correlation coefficients (r) of the correlation between the true scores and the predicted ones was positive and significant for benchmarks and regardless of the set of "genes" used to make the prediction imeantsem 0.68+0.044: Student's 's t-test again zero: t(1 1(11)=16.0, p<0.001; see Figure 15 (b) in Appendix F.. In other terms, within benchmarks and across families, the phylogenetic distance metric allowed us to predict average 0.00%% of variance the between-
benchmark performance. In a control analysis, we also verified also achieved within families, thus eliminating the possibility that
that significant correlation significant prediction in the previous analysis was driven by our metric simply capturing the fact average. To do so, calculated the
that different families have different levels of performance Pearson correlation between the true and the predicted scores per benchmark and within each family separately. The results indicate that, even though for some combinations of families and benchmarks, we obtained small or negative correlation coefficients (which is unsurprising, since these correlations were sometimes calculated across very few data points), also in this case, the results were average positive and significant difference from zero 1..64 t( ((107)=22 20.7, p<0.001; see Figure 15 (c) Appendix F). Within families, the variance explained by our method amounted to 52.2 0.03% on average, thus indicating that our metric achieved good predictive power even when drastically increasing the level of granularity. Individual plots for each benchmark are shown in Appendix
DISCUSSION
Here we show that an algorithm, inspired by those used in phylogeny, is successful in reconstructing important aspects of the genesis of LLMs, based solely their outputs to diverse short queries. By leveraging the genetic distance matrix, becomes feasible to robustly trace the relationships and evolution of models over time. This is particularly evident in the constructed dendrograms, where clear clusters align with distinct families of LLMs, offering a visual representation of their evolutionary trajectories least their training similarity. It is important to also emphasize the

---

Published as a conference paper at ICLR 2025
applicability of these methods proprictary models. Understanding the fine-tuning relationships and performance characteristics private models often challenging due to limited access to training details and data. PhyloLM offers a valuable tool for gaining insights into these aspects. by providing to the research community a more transparent image of how proprietary models evolve.
We also show that the utility of the "genetic" distance, derived from our algorithm, capturing the training relationships, but could be used to infer the performances of models on various benchmarks. The observation that a logistic regression trained on the genetic distance matrix can accurately predict benchmark accuracy has the potential to accelerate the evaluation of new LLMs capabilities in very computationally efficient manner. Overall, our method provides robust and insightful analysis of the history, relationships, and performance of Large Language Models, even in cases where detailed training information not publicly available.
not limited to
Despite these promising results, it is important to acknowledge the inherent limitations of applying the genetic metaphor LLMs. Phylogenetic algorithms, traditionally designed for biological analysis where common ancestors are not included among the tested species, face challenges when applied to LLMs, where common ancestors are present among the studied models. Furthermore, chat interfaces complicate the acquisition of reliable genetic material. Nonetheless, this work lays the foundation for further studies aimed at refining these algorithms to better fit the LLM framework and chat models. Our study did not explore the effect of temperature, and while results sets of genes (and more in Appendix []). examining an even broader range of genes could provide additional insights. Additionally, while the predictive results for benchmark scores promising (roughly 50% of the variance explained) and could be practically applied to estimate the capabilities of new models, remains room for improvement possible venue being using multiple sets of genes in the evaluation).
consistent across two
Lastly, similarity matrices serve as versatile tools with numerous applications in the study and opti- mization of large language models (LLMs). For instance, in our investigation of model quantization, we discovered that the size of the model increases, the quantized version more closely approximates the original model (see Appendix H). Additional fields in which PhyloLM could provide very good insights could also include model merging (Goddard et al., 2024) and scaling laws but we leave it for further research.
ACKNOWLEDGMENTS
This work was granted access to the HPC/IA resources of [IDRIS HPE Jean Zay A100] under the allocation 2023- (AD made by GENCI SP is supported by the European Re- search Council under the European Union's 's Horizon 2020 research and innovation program (ERC) (RaReMem: 101043804), the Agence National de Recherche (CogFinAgentt 02: RELATIVE: ANR-21-CE37-00 Humboldt foundation and a Google unrestricted gift. PYO is supported by ANR AI individual chair ANR-I9-CHIA-0004.
01; RANGE:
the Alexander Von
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffint,, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of open language models, 2023.
Arc53. Arc533/locsgpt27-mmisraa December 2023. URL http:s:://ugginggfcc...o / /docsgpt - 7b mistral
Ben Athiwaratkun et al. Multi-lingual evaluation of code generation models, 2023.
Quentin D Atkinson, Andrew Meade, Chris Venditti, Simon Greenhill, and Mark Pagel. Languages evolve in punctuational bursts. Science, 319(5863):588-55553 2008
Jinze Bai al. Qwen technical report, 2023.
10

---

Published as a conference paper at ICLR 2025
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
BigScienceWorksshp et al. Bloom: A 76bb-parameter open-access multilingual language model, 2023.
Tom Brown al. Language models are few-shot learners, 2020.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi. Cunxiang Wang. Yidong Wang. Wei Ye, Yue Zhang, Yi Chang. Philip S. Yu, Qiang Yang, and Xing Xie. survey on evaluation of large language models, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL[htte
François Chollet. On the measure of intelligence, 2019.
Aakanksha Chowdhery et al. Palm: Scaling language modeling with pathways, 2022.
R Dawkins. The Selfish Gene. Oxford University Press, Oxford, UK, 1976.
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models, 2023.
Julien d'Huy. Polyphemus (aa. th. 1137): A phylogenetic reconstruction of prehistoric tale. Nouvelle Mythologie Comparée/New Comparative Mythology, ((Dhitp-oovelleyyhooggeommmee 2013.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee's 's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403. 13257, 2024.
Russell D Gray, David Bryant, and Simon Greenhill. On the shape and fabric of human history. Philosophical Transactions of the Royal Society B: Biological Sciences, 365(1559):392333335 2010.
Niklaus Grünwald, Sydney Everhart, Brian Knaus, and Zhian Kamvar. Best practices for population genetic analyses. Phytopathology. 107, 05 2017. doi: 10.1094/PHYTO 12- 16- 0425--VW.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
HuggingFaceH4. open Ilm leaderboard. URL htt.tpps//uggingaaee...////sss/s
@FaceH4/open 1 1m l
eaderboard rd
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie- Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Percy Liang et al. Holistic evaluation of language models, 2023.
Q. Vera Liao and Jennifer Wortman Vaughan. Ai transparency in the age of Imms: A human-ccmtert research roadmap, 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for Ilm compression and acceleration, 2024.
11

---

Published as a conference paper at ICLR 2025
mlabonne.
milbonnccheerrlllmmss2222mmtttu November 2023. CRLnhtts:: httpttt/hhsciingg:ggg/ hugginggg
OpenAl et al. Gpt-4 technical report, 2023,
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Saitou and Nei. The neighbor-joining method: a new method for reconstructing phylogenetic trees. Molecular Biology and Evolution, 4(4):406 425, 07 1987. ISSN 0737-4038. doi: 10.1093/ oxfordjoumals.. molbev.a.40455 URL /tt.://pppp////....o////////////
.molbev. a44054
Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Naoko Takezaki and Masatoshi Nei. Genetic Distances and Reconstruction of Phylogenetic Trees
From Microsatellite DNA. Genetics, 144(1):389 399, 09 1996. ISSN 1943-2631. doi: 10. 10.109e netics/ 1.39..
http://..11111////// wtt.:/1111///eeeeccsseeeme /144. 1.4.1.389
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An caaruction-following llama model. https://gi
2023.
Gemini Team et al. Gemini: A family highly capable multimodal models, 2024a.
Gemma Team et al. Gemma: Open models based on gemini research and technology, 2024b.
Jamie Tehrani and Julien Huy, 2017. Phylogenetics Meets Folklore: Bioinformatics Approaches to the Study of International Folktales, pp. 91-114. 01 2017. ISBN 978-3-319-394434 doi:
10.1107//77--3311 1919999445-9-5
Jamshid J Tehrani and Mark Collard. On the relationship between interindividual cultural transmission study of weaving in iranian tribal populations.
and population- level cultural diversity: Evolution and Human Behavior, 30(():286-33)) 300666300,00000 2009.
Tenyx. Tenyxchat: Language model alignment using tenyx fine-tuning, January 2024. URL https
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Imran Ullah. Imran 1/medchat3.5, January 2024. URL /ttps://hoginggffcccc///mmm MedChat3.5
Jorge Vallego. novaall/h-mmoonoossaneee January 2024. URL URL/stps:/huuginiiacc..o Bovaallee/HLEmooooouuedtt
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
Shen Zheng, Yuyu Zhang, Yijie Zhu, Chenguang Xi. Pengyang Gao, Xun Zhou, and Kevin Chen- Chuan Chang. GPT-Fathom: Benchmarking large language models to decipher the evolutionary path towards GPT-4 and beyond, 2023. URL www.hhttpp........ //axx w.w.g/...o///////eeeeo httees////rxx..oorr/bbbb//223. 16833
12

---

Published as a conference paper at ICLR 2025
APPENDIX
This Appendix provides additional materials for PhyloLM
Section presents examples of genes' and 'alleles' for LLMs.
shows the list models included in this study with finetuning relationships, sizes and benchmark scores,
Section
Section gives PhyloLM pseudo code.
Section outlines the results on the code set of genes.
Section discusses more in depth similarity matrices results, differences between models
and proposes potential explanations for such observations.
Section provides results about benchmark prediction on each benchmark for each family of models.
Section represents dendrograms including both sets of genes together.
Section exhibits results on PhyloLM and model quantization.
Section gives an overview of dendrograms with the names of the models.
13

---

Published as a conference paper at ICLR 2025
A EXAMPLES OF GENES AND ALLELES
Genes were extracted from the Open-Web-Malt Web-Math (Paster et al. 2023 and MBXP (Athiwaratkun et al. 2023) datasets. The Open-Web-Math dataset comprises 31,577 rows, from which selected the first 20 100 characters from the 'text' column (refer to Section 2.2 for detailed extraction methods), The MBXP dataset contains 6,814 rows, from which genes were extracted from the 'canonical solution' column, Twenty of the shortest genes (to fit on the page) in the extracted gene sets are presented in Table 1
Table I: Examples of 20 short genes extracted from both sets of genes and one allele sampled from 3 LLMs: Llama 7B, Llama 2 7B and GPT4 turbo 03/14. +d stands for a newline and represents a space (except in genes where spaces are packed to only one space for simplicity). Only the shortest genes have been included to fit the table. Longer genes are also present in the gene set.
(a) from open-web-math set of genes
Genes
Llamal 7B
Llama2 7B
GPT-4 (0314)
# Ignatius and the P
igeo
ig+J
arli
SlbegingroupS Close
vote
vote
I'mo
# Propositional Logic
Prop
/Mis
Prop
[texhax] environment
vari
2,JDe
Ance
### /eThee are
23ch
219p
# Annual income of a
pers
Boan
Team
Image Mosaicking[
Impl
Writ
Imag
# Physics (Version 8.4
J
JJJJ
"Phy
# Solve the linear equatio
...ed
##oS
Toos
[texhax] \mid Description J # string.replace. e
[tex
I=am
Thea
akti
acti
ax
# Math Help 2-norm of ma Math Help matlab code hel Thank you for visiting nature size Maple HMT- size,J
xtri
##oP
Toof
pelljJ
plin
I'da
gift
tour
You'
Theo
\obeg
InoM
# In observing Tetrahedron...
Inco
Inco
Azte
Previous issue Next issue # All Q questions
Volu
Arch
Arch
Alle
##nA
Unfo
## 1002241]MModriinn DreameldJ H&
How to prove that SC=)(x: Axlle
(la)S
0la)$
blal$
(b) from MBXP set of genes
return )
temp
func
func
A double s
=01;
1(do
I'mo
J res [] for
inoi
iner
acho
2 ////OO
00o0
15.0
Ther
return > 0.? % 10
Oata
c00;
+oMa
return a a ++b+
Acend


func
Fuunction
hand
ao//
dete
if (monthnum3 ==
Ja
Item
str ""join(tupl
Jnnn
ifot
Ifny
J res tuple(list(te
ffe
.out
Itos
r n [J return (r)
prin
endæ
defo
seq_nums [seq__ums_ssssss
star
1:]]
iluf
double result = 0
ifo(
doub
Inco
J string result
null
null
I=ca
SisSame = =tree:.] fore return n (2 n 1).l) let sortedAm arr.ort
:elnn
achn
achn
//TT
***
func
((a.
(na,
((a,
$listl Slistl I1 []:]] return la[0] == [a[n-1]:=]]
$lis
$lis
ifa(
bool
into
This
string result "":eJ
obar
.ope r=(i
14

---

Published as a conference paper at ICLR 2025
B MODELS INCLUDED IN THIS STUDY
Open access models were run on A 100 80Gb GPUs (approximately one per 20B parameters) with default floating precision except for Falcon 8OB that was downsized from float32 6 was too big to fit on 8 GPUs. OpenAl models were accessed through the Openai API Claude models were acessed through the Anthropic AI API. Proprietary mistral models were accessed through the Mistral API Proprietary Google models were accessed through the VertexAl API Tokens were sampled with a temperature of 0.7 with no sampling restriction (topp=1).
The following table 2 includes all models tested in this study with their benchmark scores. Only the completion models have been included the benchmark score prediction study. After this table the table of chat models that were separated in dendrogram plots and not included in benchmark score prediction. They can be found Table 3
From HELM
Liang et al.
2023
no
information
about
prompting
https://erfm.stami hww.://ewwwemmmmmssi/ccieee
From GPT-Fathom (Zeng et al 2023 ARC(1 shot) HellaSwag (1 shot) MMLU (5 shot) TruthfulQA
shot) WinoGrande (1 shot) GSM8K (8 shot CoT)
From HuggingFace (acegin
ARC(25 shot) HellaSwag (10 shot)
MMLU (5 shot) TruthfulQA (0 shot) WinoGrande (5 shot) GSM8K (5 shot)
Jlm leaderboard
15

---

Published as a conference paper at ICLR 2025
Table 2: List of completion LLMs included in the study. They appear in the similarity matrices (see Fig ( the same order as presented in this table. For proprictary LLMs, names are taken from their respective API. For all the other models, they were downloaded from the huggingface hub with the same name as given here. and M stand for billions and millions, Ø for trained from scratch and means unknown not officially communicated.
Name
Family
Size
Parent
AR
HS
ML
TQ
WG
GS
llama-bb
Iaaa 1-7
7B
Ø
50,9
77.8
35.6
34.3
71.4
8.03
alpaca-7b
Ilama 1-7
7B
llama- 7
52.0
76.9
41.
37.5
69.4
1.44
wizard-7b
Ilama 1-7
7B
Ilama-7b
?
2
?
2
vicuma-7b-v1.11
Ilama 1-7
7B
Ilama-7b
2
?
vicuna-7b-v1.3
Ilama 1-7
7B
llama-7b
50.4
76.9
48.1
47.0
70.4
5.68
baize-7b
Ilama 1-7
7B
llama-78 7b
48.9
75.0
39.6
41.3
71.1
4.16
chimera-inst--hat-76
Ilama 1-7
7B
Ilama-7b
?
2
7
Ilama-131
llama 1-13
13B
Ø
56.1 80.9
47.6
39.4
76.2
7.58
vicuna-13b-vl.1
Ilama 1-13
13B
Ilama-133
52.7
80.1 51.9
52.0
74.1
8.64
vicuna-13b-vl.3
Ilama 1-13
13B
Ilama-13b
54.6
80.4 52.8
52.1
74.8
10.7
openchat_y2
llama 1-13
13B
Ilama-133
57.1 81.1 50.5
49.5
76.2
9.09
openchat_v2_w
Ilama 1-13
13B
llama-133
57.3 81.2 50.1
50.6
75.9
8.41
chimera-iint-chat-13b
llama 1-13
13B
Ilama-133
55.3 78.9 50.5
50.1
73.9
8.18
1lamh-2-7b-hf
Ilama 2-7
Ø
53.0 77.7 43.7
38.9
74.0
14.4
Orca-2-7b
Ilama 2-7
7B
llama-2-7b
54.0 76.1
56.3
52.4
73.4
14.7
-a-bbet
Ilama 2-7
7B
llama-2-7b
47.6 72.0
45.1
42.2
69.6
10.8
tigerbot-7b-chat
Ilama 2-7
7B
tigerbot-7b-
7b-base
?
2
7
OpenHermes-7B
Ilama 2-7
7B
llama-2-7b
56.1 78.3
48.6
44.9
74.5
5.00
vicuna-7b-v1.5
llama 2-7
7B
llama-2-7b
53.2 77.3 50.8
50.3
72.1
8.18
llama-2-13b-4
llama 2-13
13B
Ø
58.1 80.9 54.3
34.1
76.6
22.8
openchat_v3:
llama 2-13
13B
1lama-2-13b
59.8 82.8 56.7 44.4
76.2
13.7
openchat v3.2
Ilama 2-13
13B
Ilama-2-13b
59.6 82.6 56.6 44.4
76.9
13.6
OpenHlermes-13B
Ilama 2-13
13B
Ilama-2-13b
60.1 82.1 56.1 45.9
75.4
11.5
Ilama 2-13
13B
Ilama-2-13b
$7.0 81.2 56.6
51.5
74.6
11.2
openchat_v3.2__0
sttpchattv3.2_spprr
Ilama 2-13
13B
llama-2-13b
59.8 82.5 55.8
42.2 75.9
13.4
1oerbot-13bbbsc--V
llama 2 13
13B
1lama-2-13b
?
?
? ?
lggerbt--3b-basee
Ilama 2
13B
1lama-2-13b
?
?
13r-hot-lbb-
Ilama
13B
13b-base-vl
?
Igerroo--3bb-cht-v22 13bcchat-22
Ilama 2 13
13B
Sigerbot-13b-base->>
?
?
2
tigerbot- 1gerrhot-3b--caatvv33 13-ccht-v33
llaaa 2 13
13B
?
?
?
?
tigerbot-13b-ce 13-tthat-hhttt v4
laama 2 13
13B
Soerbot-13b-base..
7
7
7
7
bloom-3b
bloom
35.7
54.3
26.5
40.5
57.6
1.51
bloom-7b
bloom
7B
Ø
41.1
61.9
26.2
38.8
65.4
1.36
bloomz-3h
bloom
3B
bloom-3b
36.8
54.9
32.9
40.3
57.1
0.0
bloomz-7h
bloom
7B
bloom-7b
42,4
63.0
37.8
45.2
64.6
0.07
tigerbot-7h-base-vl
bloom
7B
bloom-7b
?
?
?
tigerbot-7b-base-22
bloom
bloom-7b
?
tigerbot-7b-st--1
bloom
7B
tigerbot-7b-base
?
?
?
?
-igerbot-7b-sl--22
bloom
7B
tigerbot-7b-base-22
?
7
?
?
phocnix-inst-chat-76
bloom
7B
bloom-7b
?
?
7
?
bloom-1760
bloom
176B
50.4
76.4
30.8
39.7
72.0
6.89
16

---

Published as a conference paper at ICLR 2025
Name
Family
Size
Parent
AR
HS
ML
10
WG GS
Pythia-70m
pythia
70M
Ø
21.5 27.2
25.9
47.0
51.4 0.30
Pythia-160m
pythia
160M Ø
22.7
30.3
24.9
44.2
51.5 0.22
Pythia-410m
pythia
410M Ø
26.1 40.8
27.2
41.2
53.1 0.68
Pythia-1.4b
pythia
1.4B
Ø
31.4 52.8 25.8
38.8
58.0 1.51
py pythia-2.8b
pythia
2.8B Ø
36.2 60.6 26.7
35.5
60.2 0.83
pythia-6.9b
pythia
6,9B
Ø
41.2 67.0 26.4
35.1
64.0 1.66
pythia-128
pythia
12B
Ø
39.5 68.8 26.7 31.8
64,1 1.74
d223y-v2-3b
pythia
3B
pythia-2.8b
25.2 26.5 24.6
59.4
1.06
dolly-v2-7b
pythia
pythia-6.9b
44.5 69.6 25.1
34.8
60.0
1.13
dolly-v2-12h
pythia
12B
pythia-12b
42.4 72.5 25.9
33.8
60.8
1.21
oasst-sft-4-pythia-222
pythia
12B
pythia-12b
45.7 68.5
26.8
37.8
65.9
3.03
epoch-3.5
opt-125m
opt
125M
Ø
22.8 31.4
26.0
42.8
51.6 0.07
opt-350m
opt
250M
Ø
23.5 36.7 26.0
40.8
52.6 0.30
opt-1.3b
opt
1B
Ø
29.5 54.5 24.9
38.7
59.7 0.15
opt-2.7b
opt
3B
33.9 61.4 25.4
37,4
61.9 0.22
opt-6.7b
opt
7B
39.1 68.6 24.5
35.1
65.9 0.98
o3t-13b
opt
13B
39.9 71.2 24.8
34.0
68.5 1.74
opt-30b
opt
30B
43.2 74.0 26.6 35.1
70.6 2.19
opt-66b
opt 66B
46.3 76.2 26.9 35.4
70.0 1.66
Mistral-7B--0.11
mistral
7B
Ø
59.9 83.3 64.1 42.1 78.6 37.0
Mistral-71B-Instruut--00
mistral
7B
Mistral-7B-vO.
54.5 75.6 55.3 56.2 73.7 14.2
Distral-BB-Instrue-v...
mistral
Mistral-7B-vol
63.1 84,8 60.7 68.2
40.0
zephyr-7b-alpha
mistral
7B
Mistral-7B-vO.
61.0 84.0 61.3 57.9
78.6 14.0
zephyr-7b-beta
mistral
7B
Mistral-7B-v0.1
62.4 84.3 60.7
57.8
77.1 29.0
docxgpt-7h-mistral
mistral
7B
zephyr-7h-beta
?
2
?
?
openchat_3.5
mistral
7B
Mistral-7B-v0.1
62.4 83.9 62.8
45.4
81.0
25.7
TenyxChat-7B-vl
mistral
7B
openchat_3x
65.6 85.5 64.8
51.2
80.5 63.0
MedChat3.5
mistral
7B
openchat 3.5
?
?
neural-chat-7b-v3
mistral
67.1 83.2 62.2
58.7
78.0 1.21
neural-chat-7b-vv-1
mistral
7B
Mistral-7B-vdl
65.6 83.5
62.1
59.4
78.6 20.0
OpenHermes-2-Minee
mistral
7B
Mistral-7B-v0.1
63.0 83.8
63.4
50.2
?
7B
OpenHermes-22 2.5-Mistral
mistral
7B
Mistral-7B-vO.
64.9 84.1 63.6 52.2 78.0 26.0
7B
A4mmoniousAnthea
mistral
7B
OpenHermes-2255
2
?
Mistral-7B
NeuralHermes-2.55
mistral
7B
OpenHermes-22.
66.5 84,9
63.3
54.9
78.2 61.3
Mistral-7B
Mistral-7B
Mixtral-8x7B-vO..
mistral mistral
8x7B
Ø
66.3
86.4
71.8
46.8
81.6
57.6
Mixtral-&x78-Innttueet
8x7B
87.5 70.1
71.3
64.9
81.0
61.1
v0.1
qwen
2B
Ø
?
Qwen-7B
qwen
7B
Ø
$1.3 78.4
59.8
47.7
72.6
44.9
Qwen-14B
qwen
14B
58.2 83.9
67.7
49.4
76.7
58.9
Qwen-
qwen
72B
Ø
65.1 85.9 77.3
60.1
82.4
70.4
causallm-7b
qwen
7B
Qwen-7B
50.0 74.5 61.7
50.1
69.6
22.9
causallm- 14b
qwen
14B
Qwen-14B
56.6 79.0 65.8
47.7
74.9
58.6
Qwenl 500.BB
qwen
0.5B
Ø
31.4 49.0 39.3 38.2
57.2 16.3
Qwen .5-1.8B
qwen
1.8B
37.8 61.4 46.7 39.4
60.2 33.5
Qwen 1 55-4B
qwen
Ø
48.4 71.5 56.5 47.2 66.2 52.2
Qwen I. .5-7B
qwen
7B
Ø
54.1 78.5 61.9 51.0 71.2 53.5
Qwenl.5-
qwen
14B Ø
56.5 81.0 69.3 $2.0 73.4 67.6
Qwenl.5-23 3-3BBB
qwen
32B
Ø
63.5 85,0 74.2 57.3 81.4 61.1
@wenl.5-722
qwen
72B
Ø
65.8 85,9 77.2 59.6
83.0 65.7
Qwen 1 55-0.5B-Chat
qwen
0.5B
Qwenl Dwen1.5-0.5B
.5-0.5B
30.5 44.0 33.8
42.9
54.6 7.65
Qwen 1 55-11.8BChat
qwen
1.8B
Qwen1.5-1.85
15-1.8B
?
2
?
?
Qwen 1.een..5-4B-4 55-BB-
qwen
4B
Qwenl.5-4p3 1-5-BB
43.2 69.7 55.5
44.7
64.9
2.42
Qwenl 5-7B-Chat
qwen
7B
Qwenl.5-771 157BB
55.8
78.5 61.6 53.6
67.7
13.1
Qwen 15-11B-Chal
qwen
14B
Dwenl.5-144
58.7
82.3 68.5 60.3
73.3
30.8
Qwenl Dweel1.5-3BB- 5--32B-hhat
qwen
32B
Dowen1.5-32D
66.0
85.4
74.9
66.9
77.1 7.05
Qwen1 Owenl..5-2BB 5--25--7BBB5
qwen
72B
Qwenl.5-725
68.5 86.4
77.4
63.8
79.0
20.3
17

---

Published as a conference paper at ICLR 2025
Name
Family
Size
Parent
AR
HS
ML
TQ
WG
GS
falcon-rw-1b
falcon
35.0
25.2
62.0
0.53
falcon-rw-7b
falcon
7B
Ø
?
?
7
falcon-7b
falcon
7B
Ø
47.8 78.1
27.7
34.2
72.3
4.62
falcon-7b-instruct
falcon
7B
falcon-7b
46.1 70.8
25.8
44.0
67.9
4.70
falcon-40b
falcon
40B
61.9 85.2 56.9
41.7
81.2
21.4
falcon-40b-instrue
falcon
40B
falcon-40b
61.6 84.3 55.4
52.5
79.7
34.3
1accon-180b
falcon
180B
Ø
69.1 88.8 69.5
45.1
86.8
45.9
gemma-2b
gemma
2B
48.3 71.7 41.7 33.0
66.2
16.9
gemma-2b-it
gemma
2B
gemmaa2h
43.9 62.6 37.6
45.8
60.9
5.45
gemma-7b
gemma
7B
Ø
61.0 82.4 66.0
78.4
52.7
gemma- 7b-it
gemma
7B
gemma-7b
51.4 71.9 53.5 ?
47.2
67.9 29.1
gemma-l. 1.1-b--
gemma
2B
gemma- gemmaa-bbit
?
7
gemma-1.1-21 1-1-bb-it
gemma
7B
cemma-7b-it
60.0 76.1 60.9
50.7
69.6 42.9
codegemma-29
gemma
2B
gemma-2b
25.6 39.6 28.3
41.2
53.6 4.62
codegemma- 7b
gemma
7B
gemma-7b
53,9 76.7 56.5
38.0 69.6 45.4
codegerma-77-ii
gemma
7B
codegemma-75
53.8 72.5 56.0 48.5 66.2
52.4
ada
gpt 3
?
Ø
43.5 24.3 21.5 ?
0.6
babbage
gpt 3
55.5 23.3 18.8 ?
0.7
curic
gpt
Ø
68.2 24.3 23.2
davinci
gpt 3
175B Ø
35.9
22.8 34.3 21.4 48.0
12.1
davinci-instruut-beee
gpt 3
175B
davinci
40.9
18.9 39.9 5,4
49.6
10.8
text-ada-001
gpt 3
?
ada
42.9 23.8
23.2
?
0.4
text-babbage-001
gpt 3
babbage
56.1 22.9
23.3
?
text-curie-001
gpt 3
curie
67.6 23.7 25.7
text-davinci-001
gpt 3
175B ?
53.2
34.6
46.7
21.7
54.6 15.6
text-davinci-00)
gpt 3,5
175B code-davinci-002
75.7
64,9
62.1
47.8
65.5 47.3
lexi-davinci-003
gpt 3.5
175B ?
79.5
60.4
63.7
52.2
70.6
59.4
babbage-002
ept 3.5
davinci-002
gpt 3.5
?
?
tpp-3.5-turboiinstuet-
gpt 3.5
?
2
83.6
82,8
69.6
59.4
68.0 75.8
0914
18

---

Published as a conference paper at ICLR 2025
List of chat LLMs included in the study. They appear in the similarity matrices (see Fig 10p the same order as presented in this table. For proprietary LLMs, names are taken from their respective API. For all the other models, they were downloaded from the huggingface hub with the same name as given here. B and M stand for billions and millions, Ø for trained from scratch and ? means unknown not officially communicated.
Table 3:
Name
Family
Size
Parent
gpt gpt 3.5
Spt-3.5-turoo-330
gpt
gpt-3.5-turbo-0613
gpt gpt
gpt-3.5-turbo-1106
gpt 3.5
gpt-4-0314
gpt 4
gpt-4-0613
gpt
gpl-4-visionpprevieew
gpt
gp1-4-1106-preview
gpt
gp1-4-0125-preview
gpt
claude-2.0
claude
claude-2.1
claude
cland-3-haiikk-22244300
claude
clande33ssommee-222402299
claude
cladde-3-opuu-202240299
claude
mistral-tiny-22312
(Mistral-7B-vO.))
mistral
7B
mnsttral-small-2312
(Mixtral-8x7B-vO..
mistral
8x7B
mistrallssaall22402
mistral
mistral-mmedium22312
mistral
mistral- 1istral-large-24002
mistral
text-bison @001
palm
text-bison @ 002
palm
lext-unicorn 001
palm
chat-bison @ 001
palm
chat-bison @00
palm
gemini-l.0-po 1.0-pro
gemini
gemini-1.O-pro-00 1.0-pro-001
gemini
gemini-1.0-pe 1.0-pro-002
gemini
gemini- 1onmiii-.--pro--iision
gemini
1ummii--.0-pro--vsioo-00000
gemini
19

---

Published as a conference paper at ICLR 2025
C PHYLOLM ALGORITHM
Data: genes, N, LLMs
/* Estimate
*/
P array( (LLMs, geness vocab_size) for LLM LLMs do
#2ero filled array of given shape
for gene in genes do for & N do
allele generate 4 chhoractersLLLMM.gng
*Generate 'allele'
PILLM, gene, allele] PILLM, gene, gene, allele
#Update
N
end
end end
/+ Compute similarity matrix S from P array( LLMs, LLMs)
for LLMI in LLMs do
for LLM2 in LLMs do
EyEG EOEA, PILLMI, P|LMM,. g.al
SILLM1, LLM2]
(Egeg Lues, g,)3), EvEs, g(LL.M2.g,a{3)
end end
/+ Compute distance matrix D from S D 1---logh ISg(S) return D
Algorithm 1: PhyloLM
20

---

Published as a conference paper at ICLR 2025
D RESULTS REPLICATION ON THE CODE SET OF GENES
In the main text we discussed the results obtained on the math set of genes and analysed dendrograms (see Section 3.2.1 Here we discuss results obtained the code set of genes about hyperparameter search for PhyloLM, ground truth tree reconstruction and dendrograms. The benchmark prediction score (Figure 5) already includes code benchmark. For more details about individual benchmarks and sets of genes, see Appendix
N
0.12
8
0.12
8
16
16
Varlance
32
Variance
32
0.08
64
0.08
64
128
128
0.04
0.04
0.0
16
128
16
128 256
(a) Variabiliity of distance matrices
(b) Distance to the high precision matrix
Figure 6: Hyperparameters impact on distance matrices in the code set of genes. (a) shows the variability of distance matrices for different number of genes G and number probes N in the math benchmark. Each set of genes of specified size contains different and independent genes from the other matrices for total of distance matrix for each data point in the figure. (b) shows the distance to the high precision matrix made of 2048 genes and N=128 in the math benchmark. Errorbars represent the standard error of the mean.
D.I HYPERPARAMETER INFLUENCE
We conducted the same experiments as described in Section 2.3 We first ran the variability analysis yielding the same results as the math set of genes (see Figure 6 for code results and for math set of genes): the variability decreases G increases but N doesn't i affect very much this value.
About the approximation of the high-precision matrix we get the same result as well for each value of N, there is value of G from which increasing G doesn't 't make the approximation very much better. One difference can be noted: with the math set of genes the approximation completely saturates while in the code set of genes still improves little bit. The same tradeoff of G 128 and 32 can be found in this plot.
D.2 PHYLOGENETIC TREE RECONSTRUCTION
When trying to reconstruct part of the Mistral Family we get similar results compared to the math set of genes (see Figure for the code reconstruction 3n the math reconstruction), However, the root of the tree is not the right one. Indeed, the dendrogram algorithm, NJ (Saitou & Nei 1987) constructs an unrooted tree that is then rooted for plotting. Here it failed to find the right root. Nonetheless, except the root choice, the structure of the is identical to the ground truth. That also why we choose to plot unrooted trees in the paper as choosing a root for all the LLMs evolution very artificial.
D.3 DENDROGRAMS
Let's first look at the dendrogram (see Figure 8), one can notice very clear clusters for all completion families like in the math set of genes (see Figure 4). Indeed, all the llama branches are clustered together, Mistral, Gemma, Pythia, Qwen and Bloom models have very distinct branches in the phylogenetic tree. On the other hand, GPT-3, OPT and some Falcon models seem to share more than the other families like in the math similarity matrix. However, they still appear a little more different than in the math dendrogram. More details are given in the comparison of similarity matrices in Appendix H
21

---

Published as a conference paper at ICLR 2025
Ground truth
Reconstructed with PhyloLM
7b
docipp-7b-mistral
NeuraiHermes-2. Mistral-P 78
Mistrall
Openticrmes
H4moniosssmmhhe
Héi kémonussktthca
MedChats5.5
MedChatt.55
Openchar 3.5
TonyaChat
"roots"
"leaves"
0.1
branch length
Figure 7: Phylogenetic tree reconstruction on the code set of genes. Left: ground truth concerning the relation of some LLMs the Mistral family. Right the reconstruction from the phylogenetic algorithm on the 'code' set of genes for the five latest models of this family ("leaves" of the phylo- genetic tree) on which we run PhyloLM. Right: phylogenetic tree reconstructed by PhyloLM when given input the "leaves" models. The numerical labels (0:3) map the true common ancestors (on the right, "ground truth") to the inferred ones (on the left, "reconstructed") and the reconstructed trees are very similar to the exception of the root that is in the ground truth but 3 in the reconstruction. we reroot the tree as having 0 as root we would get the exact ground truth like with the math set of genes.
can be seen that the true
ogt
pythia
qeen
gemma
1-7b
laaa 2-7b
gpt
gpt 3.5
palm
(a) completion
Figure 8: Inferred phylogenetic tree of LLMSs on the 'code' set of genes. (a) completion models include 111 open source models included in our study and the openai completion models (b) chat models include additional proprictary models. Completion and chat models were separated because they are not comparable due to additional prompting from the API. Leaves represent models and branches colored according model families. Llama models have been split by version of the pretrained model and the number of parameters. Length of lines from a model (leaf) to another represents the distance between the two models.
22

---

Published as a conference paper at ICLR 2025
E SIMILARITY MATRICES COMPARISON BETWEEN SETS OF GENES
Let's now dive in the similarity matrices. It is important to understand that these matrices a little noisy due to the value of N and G used (see Section 3.1) Estimating this noise is challenging as it may change from a LLM to another and is very difficult to represent in figures. By estimating variance on the smallest gpt models (ada, babbage, text-ada-001, text-babbage-001, babbage-002) we found the variance of the hyperparameters used in this study to be around 0.04 (sce Figure 2a for the math set of genes and Figure 6 for the code set of genes). A difference of such a tiny value is barely visible with naked eye on a to scale (the scale in which similarity matrices are plotted) meaning that every visible difference of shade between models in the similarity matrix is significant. Of course, this is just an approximation, however, running such a statistical analysis on all models would be far too expensive in practice thus we stayed with this approximation in the paper.
Let's first dive into the completion models similarity matrices (see Figure 2 for the math set of genes and Figure Figure for for the code set of genes). First we notice that the code similarity matrix is lot brighter than the math similarity matrix. This is likely due to the fact that code is a very formatted language meaning less options are available to complete context making all models appear a little bit more similar when sampling alleles. Now, let's dive into the matrix one family at time.
In the next paragraphs, we will present observations from the different distance matrices and contrast the observations between the two sets of genes. Then, we will hypothesise what are the reasons why we observe these things and sometimes validate these hypotheses if the models are transparent enough about the training details. For proprietary models we will only hypothesise drawing parallel with validated hypothesis on the open access models that are transparent enough.
LJama The same subfamily clusters can be found in both similarity matrices with, on one side, rigerbot-7b-base and tigerbot-7b-chat that appear closer each other than the average llama models. Indeed, the second one is finetuned from the first one. Additionally, tigerbot- Sotrbott-1bbaan 13b-chat-v3 and igerbot-13b-char v4 form an additional cluster as they are based on each other (see Table D in Appendix [B). We can also find openchat v2 and openchat v2 w in same subcluster as they are trained on similar data as well as openchat v and openchat_v33..
serrbot-13b-basee 1, tigerbot-
13b-base-v2, ggerbbo--1bbchaty
ggerbot-13b-cha-y
Pythia Very clear finetuning relationships can be found in Pythia in both sets of genes as the dolly models are based on the respective pythia model with the same size and each of the dolly models shows an above average similarity with their repective pythia model.
OPT This family is very interesting as in both sets of genes these models appear fairly close to each other. However, in the code similarity matrix, the family shows a lot of dissimilarity with all other models which is not the case with the math set of genes showing peculiar coding skills. While in the math matrix, OPT shares similarities with Pythia and GPT-3, in the code genome only the smaller gpt-3 family models including ada, babbage, curie, davinci, text-ada-001, text text-babbage-eee text-curie-00 and davinci-instruct-ber instruct-beta !) seem to share an above average similarity and all the other models look very different from OPT. The only information we have is that OPT training data include ThePile but we don't t know much about GPT-3 models.
(but not text-davinci-001 that is finetuned from davinci-
Falcon Falcon-RW-1 still shows similarities with the OPT family in both sets of genes but a little less in the code set of genes. Intriguingly, still in both matrices, the distance between Falcon-7b and Palcon-7b-instree 7b-instruct (finetuned from falcon-7B) seems larger than the distance between Falcon-7b and Falcon-40b suggesting probably an intensive instruct finetuning coding tasks as it is not found in the math similarity matrix. The same observation stands for Falcon-40b and dalcon-40b-instree
Mistral In the Mistral family, in both sets of genes, we find back the subfamilies used in the tree reconstruction : zephyr-alpha 7b, zephyr-beta-75 and docsgpt-7b-mistra showing a lot of similarity while on another hand we have openchat 3.5 with CenyxChat-7B-vv and MedChat3.5 and finally OpenHermes- 2.5-Misttall-B with H4rmoniousAmthes and 2eural -.strall 7B, Additional subclusters can be found such as Mixtral models and neural-chat models.
23

---

Published as a conference paper at ICLR 2025
Qwen Qwen, in both similarity matrices shows very intricate structure but with a lower contrast. We notice a first subcluster for the original Qwen models with their finetunings in CausalLM. Each C shows better ressemblance to the Qwen model from which they were finetuned. Then we have the 1.5 family showing a lot of similarity with their chat finetuned counterparts. The smallest model (.wen1.5-0.5B-CCC
shows very little similarity to all other models in the matrix indicating maybe catastrophic forgetting to some extent during finetuning in both sets of genes. However, in the code set of genes shows even more dissimilarity with model in the math matrix. We are not entirely sure as to what could be the origin of this observation.
1- 8B while it is the closest
1_8B
Gemma In the Gemma family there is a lot of finetuning structure in both matrices. Indeed lots of white dots are far from the diagonal meaning several models show an unusual similarity with other models. Starting with gemma-22 2b and gemma- 7b appearing very similar even more than their instruction finetuned counterparts (gemma-2b-il and remma-7b-it). On the other hand, these finetuned versions appear to be very similar to the version 1.1 they were finetuned in. Lastly, codegemma-77 appear to be very close to gemma-2b 2b and gemma-7h 7b but codegemma-22 2b is less similar to them, probably because of its lower size that lead to forgetting to some extent.
GPT-3 Finally in the GPT-3 family, still in all matrices, we notice first cluster corresponding to the pretrained models ada, babbage, curie and davinci. Intestingly, lavinci-instruct-beta an SFT finetuned model from davinci shows a lot of similarities davinci in both sets of genes meaning it might have been finetuned on a low size dataset. Then their finetuned counterparts, text-ada-001 coxt-babbage-00 001., text-curie-001 and text-davinci-00) appear& lot farther from the pretrained models but we can trace back the model they were finetuned from from the similarity they share with the GPT-3 pretrained models. Then, in the math set of genes, texi-davinci-001, text-davinci-002 and text-davinci-003 look vaguely similar but on the code genome, the last two appear extremely close and a little far from text-davinci-001. This might be linked to the fact that text-davinci-o00 text-davinci-003 have been finetuned from code-davinci-002 (trained on code) and not from davinci indicating different coding skills between the 1st version and the last 2 while all of them are fairly different in reasoning tasks (which was likely the object of their respective different finetuning). Going back to the similarities with ОPT, we notice that, in coding tasks, text-davinci-00) doesn't 't show much similarities with OPT models while davinci-instruut--eta appear close OPT. What is truely interesting here is that text-davinci-001 is either finetuned from davinci or from davinci- instruct-beta and this finetuning does not seem to share similarities with OPT on coding skills showing a potential emphasis on coding skills during this finetuning. Finally, babbage-002 davinci-002 and wew.--tturo-insttree.
and
14 appear close to each other. While babbage-002 and davinci-002 might be pretrained, if we had to find parent for ppt-3.5-turbo-instte the closest model to it (but we don't 't have enough information to be sure of this).
it would likely be davinci-002 as it is
Now that we have seen the completion models similarity matrix let's switch to the chat models (see Figure T2}. These models essentially include proprietary models with very little to no transparency at all about the training details. As such we will have to completely guess the reasons for the observations we are going to make.
GPT-3.5/GPT- We discern two clusters in the GPT family before 1 1/06 and after and in both sets of genes. Indeed the newest versions of gpt-4 show a lot of similarity with each other but are extremely dissimilar to the previous versions that have lot with each other. Something important must have happened at this point in the training methods at OPENAL.
Claude In claude the two generations of models are clearly discemnibee claude and 2 on one side and claude 3 on another side. Once more big change has occurred in the training methods in between. Interestingly, the 3rd generation share a lot with bison and gemini models especially in the code context. More specifically with code-bison 001. This is not the case with the math set of genes. This may show some similarities in code training sets or data generation using another model.
Mistral In the Mistral API we notice strong similarity between mistral-small-2400 and mistral- large-2402 that were probably trained on the same training set. On another hand most models appear close to the last two gpt models more specifically these two models released 02/24. On the contrary, mistral-medium 23 seems a lot closer to the previous version of gpt-4 and gpt-3.5 showing maybe similar inspiration in the training set data generation using model.
24

---

Published as a conference paper at ICLR 2025
Palm In the Palm family, notice that text-bison@002 and code-bison @ 002 almost identical suggesting a finetuning from one to another like it is the case in the gemma family (also from google) or that it is the same model.
Gemini Finally, models in the gemini family appear almost identical to each other except gemini- 1.0-pro-002. Palm appear fairly close Gemini in both context showing maybe a shared training set to some extent (both models are from google) but in the code context they appear very close to Claude 3 models (maybe a similar training set or data generation using a model).
Figure 9: Similarity matrix of completion LLMs on the math set of genes
25

---

Published as a conference paper at ICLR 2025
Figure 10: Similarity matrix of completion LLMs on the code set of genes
26

---

Published as a conference paper at ICLR 2025
gpt-3.5-turboe
3.5-tbubb--0301
spt-3.5-turbo-111
gpt-4-0334
gpt-4-0613
apt-e-vision-pravleew
opt-4-1106-preveew
001-4-0225pprvvlee
claude-instant-1.2
cluude22.0
claude-2.1
clsude- 20uuee--hhllu-22240007
claude: 1--sonnet-22220222
cluude-3-ppu-2224222
mistrall-tny-22312
mistral-small-2312
mistral-smail-22402
mistral-medlum-2312
mistral-large-260
text-bisong001
text-bisengo02
text
toxt-unicomg001
chat-bisong001
chat-biseng002
code bisonggoogo1
code-bisong002
gemini-1.0-po 1.0-pro
gemini-1..m 0-pro-002
gemini-1.6 1.0--0000000 1-ppooviiio
gemini-1.0 proo-vvsion-000 001
zon
too aeceoos aoea
Top woweoooooooooo
to0 dodeooee anap
Tog opos
2o0 cooduone spos
oudig comiin
Eop
Spp
Figure 11: Similarity matrix of chat LLMs on the math set of genes
27

---

Published as a conference paper at ICLR 2025
gpt-3. 3.5-tbubb--0301
9pt-3.5 3.5-ttubbb-00113
Seb-3.5-turbo-1000
gpt-4-0334
gpt-4-0613
apt-t-viston-pravleew
opt-4-1106-preveev
00t-4-0225-prvviie
claude-instant-122
cauude22.0
claude-2.1
clsude- 0hooe-33hhiluu-2220007
claude: 1--sonnet-20240222
claude-3-opaa22242223
mistral-tiny-22312
mistral-small-2312
mistral-smail-22402
mistral-medlum-2332
mistral-large-260
text-bisong001
text-bisen@o02
toxt-unicomgg00
chat-biseng001
chat-bisen@002
code bis-nnoongo01
code-bisong002
gemini-1..m 0-pro-002
gemini-1.0 1.00-1100p0 1-ppoovviioo
gemini-1.0 p-pro-visian-00
Zop centiaa commne Too
C190
zon
too Booeuooee axea
100
too Top amep
Tog opos
200 Fooduonee epos
Top coreeeooo revonnee deeee
20 seado d-opneg
Figure 12: Similarity matrix of chat LLMs on the code set of genes
28

---

Published as a conference paper at ICLR 2025
F BENCHMARK RESULTS
In the main paper only MMLU and ARC prediction results were presented. Here we show all the benchmark predictions for both gene sets (Figure 5 for the math of genes and Figure ?? for the code set of genes). Statistics are reported in Tables (Table for the math set of genes and Table 3 for the code set of genes).
Pythia
pythia
bleom
0.8
bioom
owen Falcon
owen
falcon
0.0
0.6
Fridicted
Predicted
0.2
gemma
lama
qpt
36t
0.0
0.2
0.4
0.6
0.8
1.0
0.80
0.2
0.4
0.6
0.8
10
True scones
True scores
(a) ARC
(h) Hellaswag
opt
pythia
pythie
bloom
0.8
bloo
owen
qwen
felcon
feicon
0.8
0.6
Predicted
Predicted
gemma
Iomn
upt
gpt
0.0
0.2
6.4
0.6
0.8
3.0
88.0
0.2
0.4
0.6
0.8
True scones
True scores
(c) MMLU
(d) TruthfulQA
opt
Pythis
pythia
0.8
bloom
0.8
bloom
owen
awen
predicted corres
0.6
0.6
gemma
Ioma
loma
gpt
opt
0.0
0.2
0.4
0.6
1.0
0.8
0.2
0.4
0.6
0.5
True soores
True scores
(e) Winogrande
(f) GSM8K
Figure 13: Predictions of benchmark performances from the MLP compared to ground truth for every model(leave-nne-n score the model when the regressor is trained on data from all other models but the given point. For the sake of clarity, names of models have been omitted and replace by the color of models from the dendrogram plot. Lines represent a linear regression between predicted and true labels within each family.
method) for the math set of genes. Each point is the predicted
29

---

Published as a conference paper at ICLR 2025
Table 4: Statistics benchmark prediction for the math set of "genes'. i the pearson correlation coefficient between the predicted labels and the true labels, p is the p-value for the student-test against 0 and N the number of datapoint for this analysis.
Benchmark
Family
Benchmark
Family
p
p
llama
0.78
<0.001
21
llama
0.66
<0.001
21
bloom
0.98
0.002
5
bloom
0.94
0.015
5
pythia
0.84
0.001
11
pythia
0.81
0.002
11
opt
0.94
<0.001
opt
0.89
0,002
8
are
falcon
0.94
0.005
hellaswag
falcon
0.94
0.003
mistral
0.70
0.004
14
mistral
0.79
<0.001
14
qwen
0.94
<0.001
18
qwen
0.89
<0.001
18
gemma
0.95
<0.001
8
gemma
0.56
0.145
8
gpt
0.97
<0.001
gpt
0.07
0.806
12
all
0.82
<0.001
97
all
0.68
<0.001
103
Benchmark Family
p
p
N
Benchmark
Family
p
N
Ilama
0.61
0.002
21
llama
0.51
0.017
21
bloom
0.23
0.701
bloom
0.54
0.343
pythia
0.18
0.587
11
pythia
0.47
0.161
10
opt
0.31
0.450
opt
0.58
0.129
mmlu
falcon
0.93
0.005
6
truthfulqa
falcon
0.72
0.102
6
mistral
0.90
<0.001
14
mistral
0.10
0.732
qwen
0.87
<0,001
18
qwen
0.79
<0.001
18
gemma
0.96
<0.001
gemma
0.23
0.572
gpt
0.97
<0.001
12
gpt
0.79
0.001
12
all
0.88
<0.001
103
all
0.43
<0.001
102
Benchmark Family
p
p
Benchmark
Family
p
p
N
lama
0.60
0.003
21
Ilama
0.31
0.169
21
bloom
0.91
0.029
5
bloom
0,94
0,013
5
pythia
0.92
<0.001
pythia
0.58
0.058
11
opt
0.94
<0.001
8
opt
0.76
0.026
8
winogrande
falcon
0.93
0.005
gsm8k
falcon
0.90
0.012
6
mistral
0.90
<0.001
13
mistral
0.63
0.019
13
qwen
0.87
<0.001
18
qwen
0.30
0.215
18
gemma
0.55
0.149
8
gemma
0.82
0.012
8
gpt
0.91
0.011
6
gpt
0.77
0.003
12
all
0.72
<0.001
96
all
0.66
<0.001
102
30

---

Published as a conference paper at ICLR 2025
Table 5: Statisties benchmark prediction for the code set of "ggenes.. is the pearson correlation coefficient between the predicted labels and the true labels, p is the p-value for the student-test against 0 and N the number of datapoint for this analysis.
Benchmark
Family
Benchmark
Family
p
p
llama
0.39
0.076
21
llama
0.70
<0.001
21
bloom
0.95
0.012
5
bloom
0.94
0.016
5
pythia
0.84
0.001
11
pythia
0.52
0.095
11
opt
0.88
0.003
8
opt
0.93
<0.001
8
are
falcon
0.97
0.001
hellaswag
falcon
0.90
0.014
mistral
0.35
0.210
14
mistral
0.09
0.739
14
qwen 0.87
<0.001
18
qwen
0.83
<0.001
18
gemma
0.18
0.655
8
gemma
0.15
0.714
8
gpt 0.84
0.034
gpt
0.15
0.637
12
all
0.76
<0.001
97
all
0.50
<0.001
103
Benchmark Family
p
N
Benchmark
Family
p
N
Ilama
0.67
<0.001
21
llama
0.71
<0.001
21
bloom
0.84
0.072
bloom
0.65
0.230
pythia
0.08
0.794
11
pythia
0.39
0.256
10
opt
0.09
0.831
opt
0.27
0.516
mmlu
falcon
0.88
0.020
6
truthfulqa
falcon
0.71
0.106
6
mistral
0.49
0.068
14
mistral
0.09
0.753
qwen
0.93
<0.001
18
qwen
0.68
0,001
18
gemma
0.50
0.206
gemma
0.77
0.024
gpt
0.95
<0.001
12
gpt
0.80
0.001
12
all
0.79
<0.001
103
all
0.58
<0.001
102
Benchmark Family
p
Benchmark
Family
p
Ilama
0.48
0.024
21
Iaama
0.20
0.377
bloom
0.87
0.052
bloom
0.52
0,359
pythia
0.71
0.013
pythia
0.39
0.229
opt
0.90
0.001
8
opt 0.60
0.110
winogrande
falcon
0.88
0.020
gsm8k
falcon
0.83
0.040
mistral
0.53
0.062
13
mistral
0.44
0.127
13
qwen 0.75
<0.001
qwen 0.39
0.103
gemma
0.08
0.839
gemma
0.58
0.124
gpt
0.74
0.092
gpt
0.87
<0.001
12
all
0.50
<0.001
96
all
0.70
<0.001
102
31

---

Published as a conference paper at ICLR 2025
Pythia bleom
pythi
bicom
swen
qwen
Talcon
falcon
0.4
0.6
Predicted
gemma
lama
Qpt
26t
0.0
0.2
0.4
0.6
0.8
1.0
0.80
0.2
0.4
0.6
0.8
100
True snones
True scores
(a) ARC
(h) Hellaswag
sythia
pythie
bloom
0.8
booom
owen
owen
falcon
falcon
0.8
0.6
Predicted
Predicted
gemma
Iome
upt
gpt
0.0
0.2
6.4
0.6
0.8
3.0
08.0
0.2
0.4
0.6
0.8
True scones
True scores
(c) MMLU
(d) TruthfulQA
opt
Pythis
pythia
0.8
bloom
0.8
bioom
owen
awen
Predicted corres
0.6
0.6
gemma
Ioma
lama
gpt
opt
0.0
0.2
0.4
0.6
1.0
080
0.2
0.4
0.6
0.5
True soores
True scores
(c) Winogrande
(f) GSM8k
Figure 14: Predictions of benchmark performances from the MLP compared to ground truth for every model(eave-one-out method) for the code set of genes. Each point is the predicted score of the model when the regressor is trained on data from all other models but the given point. For the sake of clarity, names of models have been omitted and replace by the color of models from the dendrogram plot. Lines represent a linear regression between predicted and true labels within each family.
32

---

Published as a conference paper at ICLR 2025
A
B
0.8
0.8
Correlation confficient (R)
Predicted scores
GSMAK C
GSMIK_M
0.4
0.4
HELLAS_M
MMLU
0.4
-02
0.2
TRA_M JM
0.2
WINO_C C
.08
-0.8
-08
-0.8
0.2
0.4
0.6
0.0
True scores
Across families
Within families
Figure 15: Predictions from the logistic regression compared ground truth for every model (leave one family out method). (a) Scatter plot of the correlation between the true scores in six benchmarks (names are given on the left of the plot) against the predicted scores using our methods based on the genetic distance calculated on either the "code" (* ($_C) the "math" (* "_M) set of "genes" (the plot presents individual LLMs as points and linear regression lines; colors the panel correspond the colors of the benchmarks' names). (b) Individual Pearson's 'o's correlation coefficients of the correlation between true and predicted scores across benchmarks x set of genes (leading to 12 combinations).
(c) Individual Pearson's 's correlation coefficients of the correlation between true and predicted scores computed within each family and benchmarks (leading to 108 combinations). In (b) and (c) the horizontal line represents the mean, the dark vertical bar the standard error of the mean, the box delimitates the confidence interval and points are plotted within a probability density function.
33

---

Published as a conference paper at ICLR 2025
G GLOBAL FIGURES
In the main paper we claimed that separating completion and chat models important as they do return the allele the same gene. Indeed, many private models add additional tokens to the prompt such as message markers for user and assistant that will be concatenated to the gene such that the allele returned doesn't t correspond to the same gene as completion models. We still plotted the similarity matrices (see Figures I6 and [7) and dendrograms (see Figures 18 and 19] to show that, indeed, doesn' 't capture the families very well.
As explained in the main text we can see that the Mistral 7B model prompted in completion manner doesn't I appear close to the same model from the Mistral API prompted in chat format. A possible improvement to PhyloLM would be to prompt all completion models in chat format to see whether bridges the gap between completion and chat models. However depending on the prompting technique, some models are finetuned to interact with a specific message token whereas others might be more familiar with another prompting technique which could bias the comparison.
Looking at the figures (see Figure [6 for the math set of genes and Figure 17 for the code set of genes), one can see that, indeed, the similarity between completion and chat models (the separation starting at the second half of the gpt-family) is very low except for text-bison @ 001 and the first few gpt-3.5 and gpt-4 models for both sets of genes (maybe even clearer for the code set of genes). This can be seen further in the dendrograms as the chat models all on the same branch and, despite having a lot of structure when plotting only the chat models dendrogram (see Figure for the math of genes and Figure 8 for the code set of genes) here not much is to be seen. Indeed, one has to bear in mind that the phylogenetic tree vizualisation plots in 2D space information coming from 150 dimensional space and is influenced by the variance in all the models be plotted. As such, plotting models that are too different will lead to vague representation while plotting a very specific set of models can lead very precise discrimination in the history of the models (see F phylogenetic reconstruction from the math set of gene and Figure 7 from the code set of genes). That is why we splitted both types of models as the technical limitations linked to the interaction with some proprietary LLMs doesn't make it possible to extract precise genetic information from these models biasing the completion models from which we have precise genetic information.
34

---

Published as a conference paper at ICLR 2025
Ilaaa
bloom
pythia
opt
falcon
mistral
qwen
gemma
gpt
claude
mistral
palm
ger
Figure 16: Similarity matrix with all models (completion and chat) at the math set of genes. All models included are shown in Appendix B Purple lines split completion from chat models.
35

---

Published as a conference paper at ICLR 2025
Ilama
bloom
pythia
opt
falcon
mistral
qwen
gemma
gpt
claude
mistral
palm
Figure 17: Similarity matrix with all models (completion and chat) at once on the code set of genes. All models included are shown in Appendix Purple lines split completion from chat models.
36

---

Published as a conference paper at ICLR 2025
opt
pythia
bloom
qwen
falcon
mistral
gemma
llama 1-7b
llama 1-13b
llama 2-7b
llama 2-13b
gpt 3
gpt 3.5
gpt 4
claude
palm
gemini
Figure 18: Dendrogram with all models (completion and chat) at once on the math set of genes.
All models included are shown in Appendix B
37

---

Published as a conference paper at ICLR 2025
opt
pythia
bloom
qwen
falcon
mistral
gemma
llama 1-7b
llama 1-13b
llama 2-7b
llama 2-13b
gpt 3
gpt 3.5
gpt 4
claude
palm
gemini
Figure 19: Dendrogram with all models (completion and chat) at once on the code set of genes. All models included are shown in Appendix B
38

---

Published as a conference paper at ICLR 2025
H PHYLOLM AND MODEL QUANTIZATION
Quantization of models has become very common the in the LLM field as it makes it possible to run very large model on a rather modest hardware. However this procedure relies on simplifying the inner computations of the model raising many concerns about whether the quantized version is as reliable as the original version. As such, we investigated the Qwen family of models with quantization. They provide GPTQ 4 bit, GPTQ 8bit and AQW 4 bit version of all their chat models from the family 1.5 (Bai et al., 2023 Frantar et al. 2023 Lin et al. 2024). We computed the similarity matrix for the models in the family (except the 32B version all quantized models weren't 't online at the time of the study) and compared the similarity with respect to their quantized versions. The similarity matrix between these models is shown in Figure 21 for the math genes and [2] for the code genes. The authors did not communicate on the hyperparameters used to quantize these models aside from the number of bits
Interestingly, in the Qwen 1.5 family release, the quantized version the closest to the original model is GPTQ 8bit followed closely by the 4bit and finally by the AWQ lot farther. This observation stands for all the models in the family. However, intriguingly, the larger the model the less quantization seem to affect the distance the original model (see Fig 20 t(5)= 21.8 (p<0.001)).
1.0
1
0.9
0.8
0.5
0.7
0.6
0
0.5
GPTQ 4bit
0.4
GPTQ 8bit
-0.5
AQW 4bit
0.3
Math
Codef
0.2
10
10a
10
10ah
-1
Number parameters
Figure 20: Evolution of the similarity with the original model for all quantized Qwen1.5 Chat models. Each curve represent the evolution of similarity with the original model for each quantization method depending on the size of the model. All models included come from the Qwen1.5 huggingface repository including the quantized versions O. Qwenl.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.
39

---

Published as a conference paper at ICLR 2025
0.58
0.5B-GPTO-Int8
0.5B-GPTQ-Int4
0.5B-AWQ-Int4
1.88
1.8B-GPTQ-Int8
1.8B-GPTO-Int4
1.8B-AWQ-Int4
48
4B-GPTQ-Int8
4B-GPTQ-Int4
4B-AWQ-Int4
78
7B-GPTQ-Int8
7B-GPTQ-Int4
7B-AWQ-Int4
148
14B-GPTQ-Int8
14B-GPTQ-Int4
14B-AWQ-Int4
728
728-GPTQ-Int8
72B-GPTQ-Int4
72B-AWQ-Int4
0.58
0.5B-GPTQ-Inte
0.5B-GPTQ-InG
0.5B-AWQ-Inte
1.88
1.88 G-TQ-IPQQnnn
1.88 1.8B-GPTQ-InG -
1.B--AWQ-Inte
4B DB-GPTQ-Int8
48 JB-GPTQ-Intd
4B-AWQ-Intd
78
78- 7B-GPTQ-IntS
7B-GPTQ-Intd
7B-AWQ-Intd
148
14B- -PT-GPT--
14B- LAB-GPTQ-Inte Intd
14B-AWQ-Inte
728
72B- GPTQ--IQ-
728 72B-GPTQ-Intd
72B-AWQ-Intd
Figure 21: Similarity Matrix between Qwen 1.5 Chat models and several quantized versions on
the math set genes.. For the sake simplicity only the size the models have been included in the legend for the original Qwen 1.5 Chat models and the quantization method well as the size of the quantization for the quantized versions. All models included come from the Qwenl.5 huggingfacc repository including the quantized versions O. Qwen1.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.
40

---

Published as a conference paper at ICLR 2025
0.58
0.58-GPTQ-Int3
0.58-GPTQ-Int4
O.S
1.88
L8B-GPTQ-IntB
L8B-GPTQ-Int4
1.8B-AWQ-Int4
48
4B-GPTQ-Int8
4B-GPTQ-Int4
IB-AWQ-Int4
78
7B-GPTQ-Int8
7B-GPTQ-Int4
7B-AWQ-Int4
148
14B-GPTQ-Int8
14B-GPTQ-Int4
14B-AWQ-Int4
728
728-GPTQ-Int8
72B-GPTQ-Int4
12B-AWQ-Int4
0.58-GPTO-IN into
0.58-GPTQ-In into
0.5B-AWQ-ine -ntt
1.88
1.88-GPTQ-int in
1.88-GPTQ-in -nta
L.B-AWQ-ind in
3B-GPTQ-Inte into
4B-G DD-GPTQ-inta
4B-AWQ-intt inta
DB-GPTQ-inte
7B-GPTQ-Ii inta
TB-AWQ-Ints
14B
140 100-GPTQ-intt -nto-
140 14B-GPTQ-Int' Int4
14B-AWQ-inns in
728
P20-GPTQ-int8
720 728-GPTQ-Ind i-t4
72B-AWQ-ints
Figure 22: Similarity Matrix between Qwen 1.5 Chat models and several quantized versions on
the code set of genes.. For the sake of simplicity only the size of the models have been included in the legend for the original Qwen 1.5 Chat models and the quantization method well as the size of the quantization for the quantized versions. All models included come from the Qwenl.5 huggingfacc repository including the quantized versions O. Qwen1.5 32B Chat models are not included here as not all quantized versions were accessible at the time of the study.
41

---

Published as a conference paper at ICLR 2025
I BIG DENDROGRAMS
Dendrograms contained the main text do not include model names for the sake clarity as most of them would overlap. For the sake of clarity, we include here large vectorial figures of these corresponding dendrograms with the names so that readers can zoom in to see better where are the individual models in the dendrograms. Figure 23 shows the math based dendrogram while Figure 24 shows the code based dendrogram.
opt
Pythi
oven
falcon
gemma
Ilama 1-1.b
Naaa 2-7h
(b) chat models
gpt 3
3.5
gpt
cioude
gemnini
(a) completion models
Figure 23: Inferred phylogenetic tree of LLMss on the 'math' set of genes with the names. (a) completion models inlcude all open source models included in our study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they are not comparable due additional prompting from the API. Llama models have been split by version of the pretrained model and the number of parameters. Names are very small but this is vectorial figure readers are encouraged to zoom in when reading from computer.
42

---

Published as a conference paper at ICLR 2025
opt
pythia
bloom
falcon
mistral
gemma
llama 1-13b
2-13b
gpt
(b) chat models
gpt
gpt
claude
palm
gemini
(a) completion models
Figure 24: Inferred phylogenetic tree of LLMs on the 'math' set of genes with the names. (a) completion models inlcude all open source models included in study and the 14 openai completion models (b) chat models include additional proprietary models. Completion and chat models were separated because they not comparable due to additional prompting from the API. Llama models have been split by version of the pretrained model and the number parameters. Names are very small but this is vectorial figure: readers are encouraged to zoom in when reading from computer.
43

---

Published as a conference paper at ICLR 2025
J OTHER SETS OF GENES
The main text presents an extensive evaluation of 2 sets of genes : one based on reasoning and the other on code. Choosing a of gene is asking question the group of LLMs studied. In the main text we wanted to investigate the phylogeny of language models so we tested them on skills that have evolved lot recently. Here we include results on other sets of genes by asking other questions like language spoken, chat interaction familiarity and finally another gene set about more general knowledge.
J.1 CHINESE POETRY GENES
We wanted to test the language spoken by various LLMs so used set of genes extracted from huggingface hettwenkuo/poetrycchneesss
using the same pipeline as presented in the main text in section 2.2 but cutting around the 5th character instead of 20- 100 as chinese characters more informative than latin characters. The similarity matrix show a very little contrast between models from the same family for except Qwen and Bloom, known to have been trained on Chinese. Interestingly, Mistral also shows little contrast as well as tigerbot models finetuned from Ilama (that do not seem to know Chinese) appear to be quite close bloom models and its own tigerbot finetunings (see Figure 25).. When visualizing the matrix with dendrogram the major axis in the dendrogram looks very linked to the ability to write Chinese with the right models the 2 being the ones that show the most contrast in the matrix. The left models show very little contrast and are not known to speak chinese.
much
J.2 CHAT-STYLE GENES
We also tested genes from the reasoning gene set but with additional chat prompting see if can spot models that have been finetuned with a chat format. New genes have directly been obtained from the original math genes discussed in the main text by adding additional prompting around them
User:[Reasoning gene]In Assistant:
Results in Figure 27 show much variance within the families which appears to bimodal distribu- tion. To better visualize it we plotted the dendrograms with colors of families but also from whether they are explicitely trained for chat-based interactions (see Fig (8)). When looking at families we notice that much more split in the dendrogram compared to previous results without the chat prompting (compare with Figure [ completion models). When looking at whether the models have been exposed chat-based interactions during training we notice dendrogram. Indeed the bottom part of the plot only includes models not explicitely trained on chat based interactions while the top part is essentially composed of models trained for such interactions.
clear separation in the
J.3 WIKIPEDIA
I Lastly we wanted to see if it was still possible to reconstruct the phylogeny of LLMs not using classical finetuning topics. We used a the vikimedia/wikipeddiaad dataset from huggingface and reused the same pipeline as presented in the main text in section 2.2 to extract "ggnes' from the dataset. The similarity matrix in Figure 3300shhm shows patterns very similar to what we observed on math or code in the main text. Similarly, the dendrogram (see Figure 29) splits extremely well the different families showing that even more sets of genes can be used to study the evolution of LLMs.
44

---

Published as a conference paper at ICLR 2025
Figure 25: Similarity matrix of some completion LLMs on the chinese poetry set of genes.
45

---

Published as a conference paper at ICLR 2025
opt
pythia
bloom
qwen
falcon
mistral
gemma
llama 1-7b
Ilama 1-13b
llama 2-7b
llama 2-13b
Figure 26: Inferred phylogenetic tree of LLMs the chinese poetry set of genes. All models included are completion models. Llama models have been split by version of the pretrained model and the number of parameters.
46

---

Published as a conference paper at ICLR 2025
Figure 27: Similarity matrix of some completion LLMs on the chat version of the math set of genes.
47

---

Published as a conference paper at ICLR 2025
1pc rytha 1tam
cut rand dhat formut interattinss LLMs not cestilley trunsed on chat tanret interaciioss
Figure 28: Inferred phylogenetic tree of LLMs on the chat version of the math set genes with coloring from families and chat interaction training. Left shows the dendrogram colored with family colors and the right plot shows the same dendrogram but with colors indicating whether they have been explicitely trained to be familiar with chat interations. All models included completion models. Llama models have been split by version of the pretrained model and the number of parameters.
opt
pythia
bloom
falcon
mistral
gemma
llama 1-7b
llama 1-13b
llama 2-7b
llama 2-13b
Figure 29: Inferred phylogenetic tree of LLMs on the chinese poetry set of genes. All models included are completion models. Llama models have been split by version of the pretrained model and the number of parameters.
48

---

Published as a conference paper at ICLR 2025
Figure 30: Similarity matrix of some completion LLMs on the chinese poetry set of genes.
49

---

Published as a conference paper at ICLR 2025
K ADDITIONAL ANALYSES OF GENE SETS' HYPERPARAMETERS
K.1 GENE LENGTH
In section 2.2 we presented a pipeline to sample genes and explained that we truncated coming from test benchmarks somewhere between the 20th and the 100th character. This choice is arbitrary and we provide here more precisions about the impact of the length of genes. We took a subset of LLM families included in the study and computed a gene set using the same pipeline as presented in main text but with either only 5 characters, 200 characters or 1000 characters. We plot here similarity matrices and phylogenetic trees (Figure 31) made from these genes. For short genes the matrix a little darker than for longer genes. However, the similarity matrix docsn' seem to change very much with gene length from 20 to 100 characters long (main text configuration) and all 4 sets lead to good family clustering in the dendrograms. We conclude that the length of the gene doesn't t impact very much the reconstruction and thus it is probably better to keep rather short genes in order to avoid unnecessary costs when running the algorithm.
K.2 COMPLETION LENGTH
We also tested the impact of the number of characters to generate after each gene. Generating only character will increase the likelihood for 2 models to generate the same thing (even if they meant something different but starting with the same character) thus the similarity matrix will be a lot brighter in general models appear more similar. On the other hand, letting the LLM generate 20 characters is a lot more likely to produce generations that do not match leading to a similarity matrix lot darker.
similarity matrix with low contrast (too bright or too dark) makes all the differences between models in very low numeric range. Knowing these matrices little noisy (see Figure 2b in section 2.2) it is better to have a contrast as high as possible in order to better visualize similarities between models.
Thus we computed the RMS contrast of the matrix (standard deviation of the similarity matrix including all the parameters but the diagonal) and plotted its value from character long completion to 19 characters long completions. The results in Figure 32 show an optimal contrast for completion length somewhere between 2, 3 and 4 characters.
50

---

Published as a conference paper at ICLR 2025
Pythia
bloom
owen
falcon
gemma
Ilama 1-13b
Ilama 2-13b
51

---

Published as a conference paper at ICLR 2025
pythis
clivere
qwen
later
falcon
gemma
llaa 1-13b
laaa 2-13b
the
(d)
Figure 31: Impact of the number of characters in the gene compared in each generation similarity matrices Similarity matrices only include Llama, Pythia, OPT, Mistral and Qwen families. (a) is the similarity matrix for only 5 character long genes (b) is for 20 to 100 characters long genes (like in the main text), (c) is for 200 characters long genes and finally (d) shows the similarity matrix for 1000 characters long genes.
52

---

Published as a conference paper at ICLR 2025
(a) Similarity matrix for character long completions similarity matrix for character long completions
0.13
0.12
0.00
Beesl
Number
(d) Evolution RMS contrast lenghts be- (c) similarity matrix for character long completions tween and 19 characters.
Figure 32: Impact of the number of characters compared in each generation on similarity matrices Similarity matrices only include Llama, Pythia, OPT, Mistral and Qwen families.
53


Reviews:
[{"review_id": "ysd2GWVrQF", "reviewer": "Reviewer_Gfne", "summary": "The paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. More concretely, the authors make an analogy between phylogeny and LLM, with token context corresponding to the genes and generated tokens corresponding to the alleles. Then by applying Nei Generic distance to the empirically sampled character distribution conditional on some context, the authors first compute the similarities between models and then use the NJ algorithm to construct the phylogenetic tree. Experimentally, the authors have evaluated the choice of hyperparameters of the phylogenetic algorithms, the correctness of the recovered trees, and the quality of prediction when using the model similarities to infer their eval performances. They found that the recovered trees correspond to the know relationships of the different model (classes) and the performance prediction correlate well with the actual performance.", "questions": "Beyond the two questions above, I’m also curious whether the authors have experimented with any prompt ($c$) distribution other than math and coding and whether there are any distributions where the recovered phylogenetic trees do not correspond to our prior knowledge of how the models relate.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The presentation of the paper is clear and easy-to-read and the methodology seems reasonable and sound.\n- The idea of applying phylogenetic algorithms to study the evolution of language models, to the best of my knowledge, is a novel and interesting idea. This could inspire new works along this direction.\n- The observation that using the conditional empirical distribution (with N=32) of the conditional generation of 4 subsequent letters could already provide a lot of information about the identify of the language model is an interesting (and to me surprising) observation which can be further explored by future works.", "weaknesses": "- **[Practical usefulness of the phylogenetic tree]** In the current paper, the phylogenetics of Language models identified by running the NJ algorithm is verified using existing knowledge of the open and closed source models. Here, despite not knowing the exact training set/algorithm for some of the close models, their high-level relationships are still known to their end users (such as what company produces these models, which generation they are from, how they relate to previous models of the same company, etc). So it’s not clear to me whether any additional information can be gained by constructing the similarities and the phylogenetic tree that is not already know to the public. I believe this work still has its value purely for exploring this academically interesting direction, but from reading it, I haven’t got a sense whether the authors believe their methods could provide new insights about model evolution/relationships that’s not already publicly known.\n- **[How to think about the benefits of performance prediction using similarities]** In terms of the contribution of using the similarity score to predict the downstream eval benchmarks’ performance, the authors have mentioned in line 312 - 315 that their estimation could use fewer queries than actually evaluating on the actual benchmarks such as MMLU. However, I think the authors could use more space to discuss the type of tradeoffs involved in actually using the similarity matrix-based performance estimators. For example, although the authors’ method use fewer inference compute for eval, they also are only correlated with the actual values but not very accurate, which makes me think some further discussion on what scenarios the authors could imagine their methods as practically useful for performance prediction/comparison.\n- Minor:\n    - There is a small amount of content redundancy between line 211 - 214 on  page 4 and line 297 - 305 on page 6 discussing the very expensive distance matrix.\n    - The *completion models* on line 384 could be italicized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mPrJp0Sngk", "reviewer": "Reviewer_DQM3", "summary": "The paper proposes a biology-inspired paradigm for categorizing LLMs - an \"evolutionary tree\" of LLMs can be constructed by sampling from each LLM using a shared prompt dataset, and then comparing all outputs. The specific term used in the paper is \"phylogenetic tree\". The paper proposes an algorithm for constructing such a tree. The paper also discusses results, and the correlation of such generated trees with the ground truth and model performance.", "questions": "Suggestion: It may be more informative to apply the same technique on more well-defined and challenging tasks, such as writing an essay, rather than compare next-few-token completions on what are essentially random string prompts.\n\nQuestion: What might be the relative contributions of model architecture vs. pre-training/instruction-tuning datasets when using this approach to classify models?\n\nQuestion: The paper suggests that this analysis can provide an estimate of model capabilities; but why would this analysis provide any more insight than simply inspecting the model size, and directly evaluating the model on well-defined tasks?\n\nQuestion: The paper appears to use a string similarity measure of sampled outputs to draw conclusions about model relationships. Could we compare sampled outputs in any \"deeper\" way, such as by comparing some notion of intent, or \"tendencies\" of particular models?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\n- Classifying LLMs into an evolutionary tree is a novel idea\n\nQuality:\n- The paper provides an algorithm, sample prompts used on LLMs, an extensive list of tested LLMs, and a re-constructed evolutionary tree on L324, vs. the \"ground truth\" evolutionary tree.\n\nClarity:\n- The paper is generally clearly written, and the authors' thought process, various analogies to biology (e.g. a prompt as an \"allele\") are well-explained.", "weaknesses": "While the connection to biology is interesting and well-described, a case was not well-made for *why* the analogies should be made as the paper describes.\n\nFor example, the paper proposes the analogy of \"gene & allele\" and \"prompt & completion\": this is a bit hand-wavy; genes & alleles suggest the idea of some inherent characteristic, but how a prompt is completed may be more a function of the training dataset, rather than some model characteristic as implied by the paper: the idea that models are \"evolved\", and thus classifiable into an evolutionary tree. The separate contributions of model architecture vs. training dataset to a model was not explored.\n\nFurther, the prompt datasets used in the paper appeared to be fairly trivial, and only a very short completion was evaluated. This would be like asking college applicants to write essays, but limited to 3 words each: why comparing such short completions would still be meaningful was not explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XX63KSp7Jn", "reviewer": "Reviewer_mtg8", "summary": "The authors transfer a well-known paradigm of phylogenetic similarity from computational biology to the study of LLMs. The authors leverage the responses of LLMs to prompts contained in common benchmarks as \"genes' to evaluate the difference of responses to them as \"alleles.\" They then apply Nei's standard genetic distance to the distance between tokens in the responses with additional character agglutination normalization on the first tokens for models with different tokenizers. The authors use their \"genetic\" LLM distance to build phylogenetic trees and predict the model performance on standardized benchmarks.", "questions": "- L242-256, Is there a reason you did not use a 3rd-party tokenizer to calculate the distance between the models with different tokenizers? It would seem to improve the methodological consistency with the rest of the work.\n- L046-L047, Is there a reason you did not cite Dawkin's \"Selfish Gene\", that is generally regarded as the source of the \"meme\" idea and hence the idea of applying population genetics to cultural artifacts?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This work is outstanding.\n- The clarity of writing is top-notch, and this paper should be readable not only by the ML and genetics community, but by the general public\n- The problem the authors are trying to address - choice of most related LLMs and inference of performance of a previously untested LLM - is timely and important, as the proliferation of LLMs and tests means that for domains of interest, it is currently impossible to select the most appropriate LLM for a given application\n- Authors have a deep understanding of bases of computational genetics and phylogenetics (reference genes, ancestral species inclusion) to accurately map the concepts from that field to LLMs\n- Authors also have an excellent grasp of the inner workings of LLMs (e.g., the tokenization step), allowing them to \n- Authors are able to choose appropriate statistical tools to perform tasks of interest rather than the most common ones (e.g., ICA instead of common but inappropriate PCA on L268).\n- Authors perform extensive benchmarking, offering insights that are in themselves interesting and shed light on some of the long-standing questions in the LLM community, e.g., the performance of quantized vs. non-quantized models (L496-498) or synthetic dataset re-use in non-opensource models (Fig 4.)\n- Supplementary data is extensive (full distance matrices), and all the necessary code to re-run the experiments is provided", "weaknesses": "- Fig 5A is overloaded and confused, and 5B is non-informative. Please consider splitting out samples to make the quality of correlation clear on a first view\n- There is no discussion of neutral drift vs. selective pressure, which is particularly relevant, given that the former in genetics acts as an \"evolutionary clock,\" generally used to estimate divergence in close populations. Authors, however, selected prompts from common benchmarks, for which models are frequently optimized to perform well, and that can be seen as genes highly selected for.\n- The usage of exact token matching is a major weakness. LLMs are known for their semantic variability and ability to provide answers identical in meaning but with drastically different wording, which the author's method would consider as different. While the author's approach is arguably more relevant to identifying the similarity of training data for LLMs, overall performance is more likely to be predicted by a semantic similarity metric", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Vmf2TLZmp", "reviewer": "Reviewer_Lm8z", "summary": "The paper utilizes a metric inspired by phylogenetic to measure similarities between LLMs, outputs dendrograms between them, and predicts performance. By putting LLMs into the evolution framework, the authors propsoes a new way to evaluate model performance in a cost effective way. For models where the training details are scarce, this new framework also helps in gaining insights into their training process.", "questions": "The main questions are listed in the \"weakness\" section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. Originality: the paper is quite original in its making connection between phylogenetics and LLMs. This might open a novel avenue for studying LLM lineage and evolution\n2. Significance: the paper has the potential to have high impact as it proposes a new framework of understanding LLMs. Moreover, the use of the framework in evaluating LLMs with low resources is quite valuable considering the emergence of LLMs and the scarcity of compute resources in some research communities.", "weaknesses": "1. While the connection between phylogenetics and LLMs is novel, a not of the details are not worked out thoroughly in the paper. For example, the paper makes a connection between tokens used by LLMs and alleles in genetics. However, when comparing different LLMs, which usually use different tokenization schemes, the analogy breaks down. The authros tries to mitigate that using the 4-character generated by the model. Then an obvious drawback is that for models that do not exactly output a token of 4-character long, what the authors are measureing are no longer p(a|g). Why is 4 used here also seems arbitrary, and could be explained or motivated better. Have the authors done any experiments on changing this number and see if the results are different? \n2. Again on the analogy between phylogenetics and LLMs, these kinds of analogies could serve as an inspiration and starting point, but relying entirely too much on it could be questionable. For example, a key component of the paper is the definition of the similarity metric between LLMs, which is adapted from a metric in phylogenetics. But why is this metric suited for the analysis of LLM's? Since what's being compared is P(a|g), i.e., distributions, why not use some more standard metrics for distributions like divergence?\n3. The writing of the paper could be improved significantly. The clarity of the paper could benefit from some proof-reading. There are many places where the use of punctuations is wrong, e.g., ’gene’,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)