Unsupervised Model Tree Heritage Recovery
Authors: Eliahu Horwitz
Asaf Shul
Yedid Hoshen
Venue: ICLR 2025 Poster (2025)
Keywords: Weight Space Learning
Deep Weight Space
Model Tree
Neural Phylogeny
Phylogeny
Finetuning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose an unsupervised method to recover the hereditary structure of model populations

Abstract: The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at a foundation model and having directed edges between a parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g., "model cards") not providing accurate information about heritage. In this paper, we introduce the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us to perform this task. By using these properties, we formulate the MoTHer Recovery task as finding a directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.

Published as a conference paper at ICLR 2025
UNSUPERVISED MODEL TREE HERITAGE RECOVERY
Eliahu Horwitz, Asaf Shul, Yedid Hoshen
School of Computer Science and Engineering The Hebrew University of Jerusalem, Israel
tel heliahu.he horwitz, asaf. shul, yedid.hoshen. 9ww.l. .1l
https://horwiie ai aim.ttheeaimmmtt /mother
ABSTRACT
The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at foundation model and having directed edges between parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g., "model cards") not providing accurate information about heritage. In this paper, introduce the task Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us perform this task. By using these properties, we formulate the MoTHer Recovery task as finding directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.
INTRODUCTION
The number and diversity of neural models shared online have been growing at an unprecedented rate. For instance, on the popular model repository Hugging Face alone there are over one million models, with thousands more added daily. Many of these models related through common ancestor (i.e., foundation model) from which they were fine-tuned. Recovering the heritage of a model is important for model and data attribution. Determining whether model originated from another (i.e.. via fine-tuning) can help resolve legal disputes over model authorship in cases where each party claims the other model is based on their proprietary model. Moreover, can help identify models that resulted from the wrongful use of proprictary training data.
metadata, architecture, and weights. While model metadata may specify model heritage explicitly, the majority of public models lack proper documentation (e.g., empty model cards). Concretely, we found that over 60% of models on Hugging Face are missing information about their heritage (see App. A for a detailed analysis). Also, as many unrelated foundation models share the same architecture (e.g., VIT-B (Dosovitskiy et al., 2020)), we cannot predict heritage by looking at the model architecture. As weights of public models are very expressive and always available, in this paper we use just the weights.
Public models provide main sources of information:
Motivated by Darwin's tree of life (Darwin, 1859), which describes the relationships between organ- isms, we analogously aim to discover the Model Tree, a structure that describes the hereditary relations between models. Reminiscent of the natural world. the structure of the Model Tree is unknown. We therefore propose the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for mapping Model Trees in the rapidly evolving neural network ecosystem.

---

Published as a conference paper at ICLR 2025
We begin our exploration of the Model Tree by studying the relationship between the weights of related models. First, we establish that the distance between the weights of a pair of models correlates with their node distance on the Model Tree. We then proceed to examine how the weights of model evolve over the course of training. We observe that the number of weight outliers changes monotonically over the course of training. Specifically, we distinguish between a generalization stage (often called pre-training) and a specialization stage (often referred to as fine-tuning), We find that during generalization, the number of weight outliers grows, while during specialization, it decreases.
With these observations, we recover a Model Tree for given set of models. This requires determining whether each pair of models is directly connected and establishing the direction of the relationship. We use weight distance to create pairwise distance matrix between models and the outlier monotonicity to create a binary edge direction matrix. Finally, we use a minimum directed spanning tree algorithm on the combined distance matrix to recover our Model Tree. We extend the Model Tree to Model Graph, allowing us to recover ecosystems with multiple Model Trees by first clustering the nodes based on their pairwise distances. To evaluate the MoTHer Recovery task, we introduce the MoTHer dataset, a Model Graph comprising of over 500 models from diverse architectures and modalities.
To summarize, our main contributions are:
1. Introducing the task of Unsupervised Model Tree Heritage Recovery for model attribution.
2. Uncovering connection between model weights and their relative location on Model Trees.
3. Proposing MoTHer, new approach for unsupervised Model Tree Heritage Recovery and demonstrating its effectiveness.
2 RELATED WORKS
The field of weight space learning treats models as data points and attempts to extract information about a model by processing its weights. Eilertsen et al. (2020) performed large empirical study of model weights, representing each model as a point in the neural weight spuce and training classifiers to predict different training hyperparameters. The seminal work of (Kong et al., 2016) proposed using a HyperNet to generate parameters of model, recent works (Erkoç et al., 2023; Zhang et al., 2024; Wang et al., 2024; Peebles et al., 2022) used diffusion models to generate the weights of small networks. A different line of works predicts model performance or classifies models based on their weights by projecting the weight into embedding space (Dar al., 2022; Gueta et al., 2023), learning model weights representations (Schürholt et al., 2021; 2022; 2024), or by incorporating permutation invariance priors while training neural network on the weights (Zhou et al., 2024; Kofinas et al., 2024; Lim et al., 2023; Navon et al., 2023b;a; Unterthiner et al., 2020). Recently, Horwitz et al. (2024) recovered the weights of pre-trained model using small number of LoRA fine-tuned models, which means that their safety is vulnerable. Most related to our work is Yu & Wang (2024) which uses model Jacobians to identify the parent model for a given model and suspected set of models. However, it can not recover the direction of an edge or compute a complex heritage hierarchy. addition, Yu & Wang (2024) uses the gradients of models, which are both very computationally expensive and require running samples (which are often unavailable) through the model. In contrast, our method is unsupervised, data-free, and can recover the structure large and complex model collections. Overall, model weight processing remains an underexplored area that is poised to grow in the coming years.
MODEL TREES AND MODEL GRAPHS
This section describes the Model Tree, a data structure for describing the origin of models stemming from base model (e.g.. a foundation model) and defines the task of recovering its structure.
Definition. Consider a set of models V, where the base model Up E > serves as the root node. Every model v E V {vaJ. is fine-tuned from another model. We refer to the model from which was fine-tuned as its parent model, and denote it by Pa(v). Conversely, we refer to v as a child of Pa(v). A parent can have multiple children (including none), while all models except the root have only one parent. The set of tree edges is denoted by E, where each directed edge between a parent and its child is represented
(Pa(v), v). Overall, we define the Model Tree T by its nodes and
2

---

Published as a conference paper at ICLR 2025
directed edges, T
E), We denote by d(u, v) the number of edges the shortest path in T+ between the nodes u and v. The tree T+ is the same as our tree T except that the directed edges are replaced by undirected ones.
collection of Model Trees Ti...., Tn forms a forest, which we call a Model Graph. This Model Graph is defined 9 = (V 11 U .U...U... U,,, 3 & U....UU E.)). In Model Graph, di u. v) is only defined if u, V E Ti when u E T; and v E TT d(u, v) is undefined. Note that all the models within a Model Tree share the same architecture'. As the architecture of a model is given by its weights, and since different architectures necessarily belong to different trees, can assume without loss of generality that all U E V are of the same architecture.
Task definition. Due to the large number and diversity of models, the structure of the Model Graph unknown and is non-trivial to estimate. We therefore introduce the task of Model Tree Heritage Recovery (MoTHer Recovery) for mapping the structure of the Model Graph.
Formally, given a set of models V. the goal is to recover the structure of the Model Graph G = (V, E) based solely on the weights of the models E V. Since Model Graph is forest of Model Trees, the task involves two main steps: (i) Cluster the nodes into different components T1, T where each component is Model Tree with unknown structure. (ii) Recover the structure of each Model Tree T. Essentially, as each graph is defined by its vertices and edges, the task to recover the directed edges& using the weights of E V.
4 MODEL GRAPH PRIORS
Full FT (p 0.990)
Despite the recent growth in public models, and al- though model weights fully characterize the behav- ior of model, understanding of model weights is limited. Here, we explore two key properties of model weights that we in Sec. 5 for Unsupervised Model Graph recovery (MoTHer).
0.95
LoRA (p 0.998)
100
0.90
80
LORA
-60
0.85
-40
4.1 ESTIMATING
0.80
NODE DISTANCE FROM MODEL WEIGIITS
Model Tree Edge Distance
In this section, we investigate the use of model weights to predict the distance between two mod- els in the Model Tree. This will help us determine whether two models are related via an edge.
Figure 1: Weight Distance Model Tree Edge Distance: For every pair of models, we plot the weight distance and the correspond- ing edge distance on the Model Tree. Our weight distances FTT and LIOR almost per- fectly correlate with the number of edges be- tween models in Model Tree. This correla- tion confirms these weight distances are good indicators for determining parent-child rela- tion, i.e., models that were fine-tuned from one another. We use a levels deep Model Tree that contains 21 models
weight distance between models. Let and be two models, and di and 1 denote the weight matrix of layer ( of models and respectively,
Definition:
L
(pr(u, w) = Σ E2(1,, m)
(1)
where L is the number of model layers.
Full fine-tuning. We first study the weight distance (Fr v) between pairs of models as a function of the edge distance d(u, v) between their respective nodes on the Model Tree. In Fig. 1, we plot the relationship between these two distances (p = 0.99). It is evident that nodes with direct parent-child connections (i.e., models fine-tuned from another) have the lowest weight distance of 1. We conclude that a low ETT distance between two models highly correlated with an edge between their nodes and vice versa.
'While it is common to practice to add one or more layers to the end of a model and perform linear probing fine-tuning, the pre-trained and fine-tuned models still share the "stump" architecture. In such cases, we simply discard the disjoint layers and consider the intersecting layers.
3

---

Published as a conference paper at ICLR 2025
ResMet
ResNet
2.92
VIT
Restett
-5.0
Recdlet
-1.1-
Step
Step
Figure 2: Directional Weight Score: We plot the change in the directional weight score throughout the pre-training (generalization) stage (left) and fine-tuning (specialization) stage (right). In all cases, the directional score is almost monotonic, indicating the increasing number of weight outlier values during generalization and the decreasing number during specialization. This confirms that our directional weight score is effective for determining the direction of an edge. For the fine-tuning, we used publicly available, pre-traine pre-trained backbones as initialization
LoRA fine-tuning. LoRA (Hu et al., 2021) has become the dominant method for parameter-efficient fine-tuning. When fine-tuning model via LoRA, the entire model is frozen, and only a subset of the layers tune a new low-rank matrix for each layer. Consequently, model fine-tuned via rank LoRA differs from its base model by a matrix of at most rank for each layer. Furthermore, two models fine-tuned from the same base model using rank rı and LoRAs differ from each other by matrix of at most rank 71 + r2 per layer. We use this property to provide better estimate of the node distance between LoRA models and define the LoRA weight distance as:
LORA (u. v) max (rank(u vz))
where L is the number of fine-tuned LoRA layers. In practice, we compute the rank using singular value decomposition (SVD), where the rank the number of singular values greater than some threshold Similar to the full fine-tuning case (see Fig. 1). a low LoRA weight distance between two models is highly correlated with an edge between their nodes.
4.2 ESTIMATING EDGE DIRECTION FROM WEIGHTS
The direction of an edge between two nodes and reflects whether model v was trained from or vice versa. The weight distance from Sec. 4.1 cannot disentangle the direction as it is symmetric. Estimating the direction of edges requires a statistic of the weights that evolve monotonically during training. To this end, we use kurtosis (i.e., fourth moment) and define the directional weight score as:
E' 8((u -4)²
k(u)=
Σ TeL E (wl -m)=-
where is a set of model layers and μ the mean of the layer weights Note that the directional score only defines an order between related nodes; unrelated nodes may have very different scores.
To study the effectiveness of this score, we study how the weights of a model evolve throughout the training process. Concretely, we calculate Eq. 3 at multiple points throughout the training process and plot the results. Interestingly, we found that the training process can be categorized into two stages: a generalization stage and a specialization stage. As seen in Fig. 2, while the score is monotonic in both stages, it increases during generalization and decreases during specialization. The generalization stage usually corresponds with model pre-training and the specialization with model fine-ture fine-tuning, however, this is not necessarily always the case. The term fine-tuning is typically used for any training performed after the initial pre- pre-training stage. However, generalization training may also take place in an already pre-trained model (we show such an example in Sec. 6 (Stable Diffusion)). We therefore use the terms generalization and specialization.
Intuition. Consider model with Gaussian- based weight initialization, such Kaiming initial- ization by He et al. (2015). During the generalization stage, to better encode large, general-purpose datasets, the weights of model will likely take an increasing number of diverse values. This may
4

---

Published as a conference paper at ICLR 2025
GPC
PC2
B
PCS
S3
(a] Input models
(b) Edge via weight distance
(c) Root via weight directional score
Figure 3: Recovering Simplified Model Graph: We enumerate all possible Model Graphs of size 3 (left). On the right, we demonstrate a Model Graph Recovery process. (a) A set of 3 models with no prior knowledge regarding their relations. (b) Place edges between the nodes with the lowest weight distance. (c) Designate the node with the highest directional weight score as the root
lead to an increase the number of outlier values in the weight matrix. In contrast, during the spe- cialization stage, the weights of model with lower value diversity will likely suffice encompass the smaller, task-specific data. This may lead to decrease the number of outlier values in the weight matrix. Kurtosis, measure of the "tailedness" of distribution, captures this property.
MODEL TREE HERITAGE RECOVERY
Given the above priors, we now describe how recover the structure of Model Graphs. As warm-up problem, in Sec. 5.1 we start by manually solving all the 3 nodes Model Graphs. Then, in Sec. 5.2 we describe MoTHer, our proposed algorithm for recovering real Model Graphs.
In both cases, we have access to the model weights and assume that we are given the training stage of each node (i.e., generalization or specialization) Each Model Graph may contain one or more Model Trees. Connected nodes within a Model Tree are derived from each other via additional training steps. Unless otherwise specified, we assume no prior knowledge regarding the model relations.
5.1 WARM-UP: SIMPLIFIED MODEL GRAPH
To recover Model Graph of size 3, place edges between the nodes with the lowest weight distance and designate the node with the highest directional weight score as the root. Next, we elaborate on each of the possible size 3 Model Graphs.
Grandparent-Pareen Peetent-Parent-hhiieee (GPC). This Model Tree exhibits 3--eeneattionll relationship. where each node is derived from the previous one (see Fig. 3). To recover the underlying Model Tree structure, we use FT to place edges between node pairs with the lowest weight distances. This is motivated by our analysis in Sec. 4.1 and Fig. 1. Next, to determine the order the nodes, we use the directional score shown in Eq. 3 and designate the node with the highest score the root. Combining these steps fully constructs the GPC Model Tree, we illustrate this process in Fig. 3. Note that the above process assumes the specialization training stages for all 3 models. To adjust the process for the generalization stage, we can simply flip the sign of the directional score. When the training stages of each node differ, we choose the sign according each node's training stage.
(PC2). This Model Tree contains one parent with two children (see Fig. 3). As in the GPC case, we start by placing the edges using the weight distance defined Eq. 1. Since both children are derived from the root, the directional score will predict the node with the highest score as the root (see Fig. 3). Different training stages are handled as in the GPC case.
Carent-Child-Straan (PCS). Unlike the previous triplets, here we have Model Graph comprised of two Model Trees (see Fig. 3). To recover this structure, we first identify the node with no edge
TThis is reasonable assumption as generally know whether model is foundation model with general capabilities fine-tuned specialized model.
5

---

Published as a conference paper at ICLR 2025
D
M
MOST Algoritia
(h)
(a) Cluster Input models
distance matrix directional matrix K
using merged prior matrix M
Figure 4: MoTHer Recovery Overview: Our proposed Model Graphs and Model Trees are new data structures for describing the heredity training relations between models. In these structures, heredity relations are represented as directed edges. We introduce the task MoTHer Recovery its goal is uncover the unknown structure of Model Graphs based the weights of set of input models. Our algorithm works as follows: (a) Cluster into different Model Trees based on the pairwise weight distances. (b) For each cluster, i) use E LORA to create a pairwise distance matrix D for placing edges, and ii) create a binary directional matrix K based on the kurtosis to determine the edge direction. (c) To recover the final Model Tree, run minimum directed spanning tree (MDST) algorithm on the merged prior matrix M. The final recovered Model Graph is the union of the recovered Model Trees
according the pairwise weight distance, Since that node belongs a different Model Tree, it will have a larger distance than set threshold, allowing us to classify it as the odd one. With the node isolated, we can follow the protocol of the previous Model Trees for the two remaining nodes.
Stranger-Stranger (S3). Finally, here have Model Graph comprising 3 Model Trees (see Fig. 3). Similar to PCS, we can cluster them into different Model Trees based on their large distances, allowing identify the structure.
5.2 MOTHER RECOVERY: SCALING MODEL GRAPH RECOVERY
We present MoTHer, U)...., Un E V set of nodes representing different models. For simplicity, assume for now that Up Up T i.e., all nodes from the same Model Tree, will later relax this assumption.
method for recovering the structure of larger Model Graphs. Let
all Ui,...,
Our goal is to recover the edges & of the Model Tree T, this is done by placing edges between two nodes, where
trained from the other. As above, recovering the structure requires combination of the estimated weight distance and the edge direction. Let D be a weight distance matrix and let K be binary directional matrix.
Dij [vvi, Uz), if 0c,
[1, if k(u) < k(vz)
Kú 10, 0, otherwise
(4)
otherwise
To allow for both generalization and specialization node relations, we define as a binary matrix
if generalization
Tij 0, otherwise
The final distance matrix for recovering the Model Tree takes into account all constraints follows,
Mij + X(K + 7)
(6)
where @ is binary XOR and À regularizes the directional score, allowing for some mistakes. Since FTT and LOOR may range in value, we define À to be proportional to with X = c. ( Σi D,y) where e is some constant. In practice, we found that the results are virtually unchanged for values of E (0, 5), and chose 0.3 arbitrarily.
We can recover the Model Tree from M using a minimum directed spanning tree (MDST) algorithm. algorithm (Chu, 1965; Edmonds et al., 1967), which
In this paper, employ the

---

Published as a conference paper at ICLR 2025
iteratively contracts eycles in the graph until forming a tree. The algorithm proceeds as follows: initially, treats each node as a temporary tree. Then, it merges the temporary trees via the incoming edge with the minimum weight. Subsequently, it identifies cycles in the remaining temporary trees and removes the edge with the highest weight. This merging process continues until all cycles are eliminated, resulting in the minimum directed spanning tree. The algorithm runs in O(EV), however, faster MDST algorithms exist (Gabow et al., 1986) with O + V Vo
Multiple components. Thus far we assumed all ....... Un are from the same Model Tree. If we were to simply run the above algorithm on models that are unrelated (i.e., belong to different Model Trees), we would end up with single, wrong Model Tree. Therefore, when dealing with general model populations, we first cluster UI...., Un into different components based on their pairwise distances (using Eq. I or Eq. 2). We then run the above MoTHer recovery algorithm on each of the clusters independently. Note that while in some cases clustering is trivial by using the model architecture, it is not sufficient for the general case there are many unrelated foundation models that share the exact same architecture. For instance, DINO (Caron et al., 2021), MAE (He et al., 2022), and CLIP (Radford et al., 2021) all use VII=B(Dosovitskiy et al., 2020) architecture despite being completely unrelated.
EXPERIMENTS
6.1 1 EXPERIMENTAL SETUP
To evaluate the performance MoTHer we construct the MoTHer dataset, a Model Graph with over 500 models organized in different Model Trees. We distinguish between 4 main disjoint sub-graphs of the dataset: i) LoRA-V: LoRA fine-tuning with varying ranks, ii) LoRA-F: LoRA fine-tuning with fixed ranks, iii) FT: full fine-tuning, and iv) Mixed: mixed LoRA and full fine-tuning.
Each category contains 105 models in 3 levels of hierarchy and is comprised of 5 Model Trees rooted by different, unrelated pre-trained ViTs (Dosovitskiy et al., 2020) found Hugging Face. The second level of each Model Tree contains 4 models fine-tuned on randomly chosen datasets from the VTAB benchmark(Zhai et al., 2019). Each second-leve model has 4 child nodes, fine-tuned with randomly sampled VTAB datasets while ensuring they are different than their parent model. We label all of the models with 121 models in levels of hierarchy as well as a ResNet500Ha He et al., 2016) Model Tree (ResNet) with 21 models in 3 levels of hierarchy. The fine-tuning exhibits some variation in hyperparameters, for more details see App. B. In addition to the MoT dataset, we evaluate our method on the Stable Diffusion Model Tree found on Huugging Face.
ppccializaiion-trannedd In addition, we also construct deeper Model Tree (Deep)
We use accuracy as the evaluation metric, a correct prediction is one where both the edge placement and direction are correct. In all our tests, MoTHer ran in seconds minutes even on a CPU (see App. F for more details). For clustering the Model Graph into different Model Trees, we use hierarchical clustering over the & pairwise distance and assume knowledge of the number of clusters. We use the "scipy" (Virtanen et al., 2020) implementation with the default hyperparameters.
6.2 MOTHER DATASET RESULTS
We now present the results of our method. As discussed in Sec. 5.2, when given set of models, we first cluster the models into smaller subsets where the models belong to disjoint Model Trees. We then run MoTHer recovery on the set of models in each cluster. Notably, the clustering succeeded with perfect accuracy, we therefore discuss the results of independent Model Trees. In Sec. 6.4 show the relation between the size of the Model Graph and the accuracy of the clustering.
LoRA fine-tuning, We start by testing MoTHer on the LoRA sub-graphs of the dataset. We set L from Eg. 2 and Eq. 3 to be all the LoRA fine-tuned layers of the model. We first study the performance of LoRA fine-tuned models with varying ranks. Meaning, that the rank of the difference between two pairs of models is likely to be different and therefore discriminative. Indeed, as can be seen in Tab. 1, we successfully reconstruct all 5 Model Trees within the sub-graph with perfect accuracy. In contrast, when all models use the same rank, the variance between different models decreases, resulting in reduced accuracy for one of the 5 Model Trees.
7

---

Published as a conference paper at ICLR 2025
ImageNet
ImageNet-22 21k
MAR
DINO
MSN
Figure 5: MoTHer Dataset Overview: Our dataset simulates a Model Graph consisting of over 20 Model Trees with a total of over 500 models fine-tuned on varying datasets with different hyperparameters. We distinguish between 4 main disjoint sub-graphs, differing in backbone and fine-tuning paradigm. We visualize the ground truth structure single sub-graph that contains 105 models across 5 Model Trees. The different colors represent the different Model Trees, each rooted in different foundation model. In practice, this structure is unknown and we are only given the set of models, without knowing their relations their origin. Note that all 105 models use the same ViT architecture, making non-trivial to recover the structure
MoTHHr Recovery achieves high accuracy both for individual Model Trees and entire Model Graphs. Each sub-graph comprises 105 models from 5 Model Trees, the Mixed sub-graph simulates a real-world repository where models from the same Model Tree use either LoRA or full fine-tuning
Table 1:
MoTHer Results:
Model Tree Root
Model Graph
Sub-graph
ImageNet
ImageNet-21k
MAE
DINO
MSN
LoRA-V LoRA-F
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.8
0.96
FT
0.85
0.85
1.0
1.0
1.0
0.94
Mixed 0.9
0.9
0.9
0.9
0.9
0.83
To ablate whether the LoRA-based LORA distance defined in Eq. 2 is necessary, we repeat the above experiment with Err defined in Eg. Not using the low-rank distance prior reduces the results on LoRA-V by 22% to 0.78, demonstrating the significance of LORA for LoRA fine-tuned models.
Full fine-tuning. We now proceed to our method in cases where all the models used full fine- tuning. As before, we use the already clustered sets and run MoTHer on each set independently. We set L. from Eq. all model layers and Eq. to be all the dense layers of the model. For out of the Model Trees in the dataset, MoTHer successfully reconstructs the tree structure with perfect accuracy. The other two Model Trees suffered from wrong directional score, which resulted in an imperfect reconstruction, show the full breakdown in Tab. 1. Our method also generalizes to other architectures, we demonstrate this by recovering the structure of the ResNet Model Tree with perfect accuracy.
Mixed LoRA and full fine-tuning, Finally, we created a sub-graph to simulate real model repositories where Model Trees contain models that use different fine-tuning paradigms. In partieular, construct a Model Graph where the fine-tuning method is randomly chosen to be either full fine-tuning or LoRA fine-tuning (with fixed rank). Since the weights of models that used full fine-tuning are full rank, we must use E'g. and set L to be all the model layers. Similar to the drop in performance when using FT with LoRA fine-tuned models, here too there is some decrease in performance, resulting in an overall accuracy of 0.9, see Tab. for the Model Tree breakdown.
6.3 IN-THE-WILD MOTHER RECOVERY
Having shown we can recover the Model Graph structure with high accuracy on the MoTHer dataset, attempt to recover the Model Tree of in-the-wild models found on Hugging Face. We note
8

---

Published as a conference paper at ICLR 2025
100
v1.3
SD
SD
X
40
VTHR FT LORA-V
GT
SD
20
Stable Diffusion
v1.4
v1.5
0.6 Percentage
Figure 7: Stable Diffusson Model Tree: We suc- cessfully reconstruct all but single edge, where the estimated weight distance was incorrect but the direction was correct
Figure 6: Clustering Robustness to Small Model Graphs: both cases, the clustering works with high accuracy even for small Model Graphs
that the example below is special in that it provides relatively detailed description of its model hierarchy, this provides a unique opportunity for us to test our method a real-world situation with ground truth metrics but also emphasizes the relevance and importance of the MoTHer recovery task.
Stable Diffusion The Hugging Face model cards for Stable Diffusion describe a level hierarchy spanning 5 of their models. These are: i) Stable Diffusion 1.1, ii) Stable Diffusion 1.2, iii) Stable Diffusion 1.3, iv) Stable Diffusion 1.4, and v) Stable Diffusion 1.5. We use the frr distance described above, however, since the different model versions better and more generalized foundation models we treat them as generalization nodes (i.e., the directional score is now flipped, as seed in Eq. 5). As seen in Fig. 7, MoTHer successfully reconstructs all but single edge, incorrectly placing "Stable Diffusion 1,4" as descendent of "Stable Diffusion 1.3", instead of sibling. Notably, the mistake occurred due to a wrong distance, as the directional score returned the correct edge direction.
6.4 ABLATIONS
Robustness to similar models. Foundation models often come with fine-tuning "recipes", as such, many publicly available models are almost identical to each other, often with just different seed. We therefore test the robustness of MoTTHer with identically fine-tuned versions of ViT that used different seeds. In all 3 cases, the distance between sibling models was greater than to the parent model, allowing us to correctly recover the Model Tree.
Deeper and larger Model Trees. We ablate whether MoTHer can scale to deeper and larger Model Trees. To this end, we train a 5-level hierarchy of ViT models, rooted at the ImageNet foundation model. Each in level model has 3 child models, resulting in a set of 121 models, all belonging the same Model Tree. Indeed, although this structure has 6x more nodes and is 2x deeper (the root to leaf path is now 4 edges instead of 2), observe minor decrease in accuracy, from 0.85 0.79 79. We note that we did not test MoT on huge, web scale Model Graphs, in Sec. we discuss scaling MoTHer recovery web scale Model Graphs.
Clustering robustness to smaller Model Graphs. We now test whether the clustering succeeds for small Model Graphs. As seen in Fig. 6, even with little 10 models (across 5 Model Trees), the clustering achieves high accuracy. indicating MoTHer recovery could be performed even for small, yet diverse Model Graphs.
Other directional scores. Our directional score (Eq. 3) uses kurtosis to estimate the distributional change in outliers weight values. However, other directional scores may favorable. We compared the performance using the variance, skewness, kurtosis, and entropy. To do so, we fine-tuned each of the root models from the MoTHer dataset and extracted intermediate weights throughout the training process. The kurtosis is the only metric that demonstrated consistent monotonicity across the different models (see Fig. 8).
Effect of layers types. Neural networks often contain multiple layer types (c.g., linear, convolutional, attention). We therefore study the change in the directional score for different layer types throughout the fine-tuning process. Despite different types of layers exhibiting similar trends on average, the dense layer remained consistent across all Model Trees (see Fig. 13).
Robustness to Pruning and Quantization. Our method is robust to pruned (see App. D.1) and quantized models (see App. D.2). For example, with 90% pruming, the accuracy decreases by only 4%. In the extreme where 99% of weights are pruned, method still achieves 68% accuracy

---

Published as a conference paper at ICLR 2025
20
(a) Variance
(b) Skewness
(c) Kurtosis
(d) Entropy
Figure 8: Other directional scores: We compute the different types of directional scores throughout the fine-tuning process. The kurtosis is the only metric that remained consistently (almost) monotonic. Each color represents fine-tuning from a different Model Tree root
(random baseline is roughly 5%). Moreover, when 50% of the models underwent quantization, the performance of our method decreases by less than 1%.
DISCUSSION AND LIMITATIONS
Training stage supervision. Throughout the paper we assumed the training stage is known in advance. While for many use cases, this a reasonable assumption, finding a method to automatically infer the stage based on the model weights would allow for greater applicability in cases where the training stage supervision missing. Alternativeyy, one could extend MoTHer with methods for identifying the direction of edge that does not rely the training stage.
MoTHer Recovery at web scale. This paper makes the first step in recovering the Model Graph of trained neural networks. However, recovering web scale Model Graphs requires significant computational resources for storing the millions of models and computing their distance matrices. One possible solution is to train neural network to learn compact representations that encode the relations between models. This may be done by using the limited number of already documented models as weak supervision. However, as recent studies demonstrate (Schürholt al., 2024; Lim et al., 2023; Eilertsen et al., 2020), while the weights encode vast information about the model, treating the weights as an input neural network is non-trivial even for small models.
Models with Mixed Heritage. In this work we focused on models with single parent. However, in some cases two or more models are merged to form new model. App. we describe preliminary experiment that explores the potential of identifying such cases where we show that it is determine forward to determine both parents of merged model. Extending our method fully handle such cases is left for future work.
CONCLUSION
As the number of neural network models in repositories like Hugging Face now crosses the one million threshold, tracing the heritage of model becomes essential for attribution and help resolve legal disputes. We analyze current model repositories and conclude that most models are poorly documented and do not contain enough information for model attribution. We propose using the Model Graph and Model Tree to organize collections of models and represent the hereditary relations between them. As the structure of the Model Graph of public repositories is unknown, we introduce the task of unsupervised MoTHer Recovery, which aims map out unknown Model Graphs. We identify two key properties of model weights that enable the recovery of Model Graphs. We validate approach by successfully reconstructing Model Graph with over 500 nodes well as a Model Graph in in-the-wild production models. Taken together, the Model Graph and MoTHer Recovery make exciting first step toward understanding the origin of models.
10

---

Published as a conference paper at ICLR 2025
SOCIAL IMPACT
The ability to recover model heritage trees has significant implications for intellectual property rights and ethical AI development. By providing method to trace the heritage of fine-tuned models, this work could help resolve disputes over model ownership and identify cases of proprietary data misuse. Additionally, promotes transparency in the Al ecosystem, potentially encouraging more responsible model sharing and development practices. However, this technique could also be misused by bad actors reverse-engineer proprictary models training datasets, compromising the competitive advantage privacy safeguards of legitimate Al developers.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our method and results, we describe the experimental setup in Sec, 6,1. We elaborate on the exact models and hyperparameters we use App. B. We include the our code in the supplementary material and will make it publicly available through github upon acceptance. We will also upload the entire MoTHer dataset to Hugging Face.
ACKNOWLEDGEMENTS
This work was supported in part by the "Israel Science Foundation" (ISF), the "Council for Higher Education" (Vatat), the "Center for Interdisciplinary Data Science Research" (CIDR), the "Israeli Cyber Authority", and "KLA".
REFERENCES
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint wrXiv:2209.048355 2022. 19
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021."
Yoeng-Jin Chu. On the shortest arborescence of directed graph. Scientia Sinica, 14:1396-1444 1400, 1965.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers embedding space. arXiv preprint arXiv:2209.0255: 2022. 2
Charles Darwin. On the Origin of Species by Means of Natural Selection. Murray, 1859, or the Preservation of Favored Races in the Struggle for Life.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint Xiv::2000. 1929, 2020. 1,7
Jack Edmonds et al. Optimum branchings. Journal of Research of the national Bureau of Standards B, 71(4):
233-240, 1967. 6
Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. arXiv preprini arXiv:2002. 05688, 2020. 2, 10
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14300- 14300-14310, 14310, 2023.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pp. 9259-3269 PMLR, 2020. 19
Harold N Gabow, Zvi Galil, Thomas Spencer, and Robert E Tarjan. Efficient algorithms for finding minimum spanning trees in undirected and directed graphs. Combinatorica, 6(2):109 -22, 1986. 7
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is region in weight space for fine-tuned language models. arXiv preprint a7X 2023.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 1034, 2015. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 7
Kaiming He. Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings the IEEE/CVE conference computer vision and pattern recognition. pp. 16009, 2022.
Eliahu Horwitz, Jonathan Kahana, and Yedid Hoshen. Recovering the mr--fnne-uune tine-tuiing weights of generative models. In ICML, 2024. URL https: atttps::/oppnaee http///ppneeiiweeeeeeeeeeeeeeeep wwt.:/etteet////eee /d=1/1xxx000x 2
Edward J Hu, Yelong Shea, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0000 2021. 4
Miltiadis Kofinas, Boris Knyazev, Yan Zhang, Yunlu Chen, Gertjan J Burghouts, Efstratios Gavves, Cees GM Snoek, and David W Zhang. Graph neural networks for learning equivariant representations of neural networks. arXiv preprint arXiv:2403. 121443 2024.
Tao Kong, Anbang Yao, Yurong Chen, and Fuchun Sun. Hypernet: Towards accurate region proposal generation and joint object detection. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 845-853, 2016. 2
Derek Lim, Haggai Maron, Mare T Law, Jonathan Lorraine, and James Lucas. Graph metanetworks for processing diverse neural architectures. arXiv preprint arX1v:23/2.045001 2023. 2, 10
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pp- 25790-25816. PMLR, 2023a.2 2
Aviv Navon, Aviy Shamsian, Ethan Fetaya, Gal Chechik, Nadav Dym, and Haggai Maron. Equivariant deep weight space alignment. arXiv preprint arXiv:2310. 13397, 2023b. 2
William Peebles, Ilija Radosavovie, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint htt::2209./2999 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021. 7
Konstantin Schürholt, Dimche Kostadinov, and Damian Borth. Self-supervised representation learning on neural network weights for model characteristic prediction. Advances in Neural Information Processing Systems, 34: 16481-16493, 2021. 2
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hyper- repperrepresentatioos as generative models: Sampling unseen neural network weights. Advances in Neural Information Processing Systems, 01227906-77222 2022." 2
Konstantin Schürholt, Michael W. Mahoney, and Damian Borth. Towards scalable and versatile weight space learning. In Forty-first International Conference Machine Learning, 2024. URL https:// openreview. n
10
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448, 2020. 2
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland. Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser. Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carcy, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 1038//41592-01999 -1666-..... 7
Kai Wang, Zhaopan Xu, Yukun 7hou, Zelin Zang, Trevor Darrell, Zhuang Liu, and Yang You. Neural network diffusion. arXiv preprint ar Xoxiv:2002. 13144, 2024. 2
12

---

Published as a conference paper at ICLR 2025
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari Morcos. Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International conference on machine learning, pp. 23965-23998. PMLR, 2022. 17
Runpeng and Xinchao Wang. Neural lineage. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 4797-4807, 2024. 2
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of rep- resentation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910,04867, 2019.
Baoquan Zhang, Chuyao Luo, Demin Yu, Xutao Li, Huiwei Lin, Yunming Ye, and Bowen Zhang, Mctadiff: Meta-learning with conditional diffusion for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16687-16695, 2024.
Allan Zhou, Kaien Yang, Kaylee Burn,, Adriano Cardace, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. Advances in Neural Information Processing Systems, 36, 2024. 2
13

---

Published as a conference paper at ICLR 2025
13 21 18 30
28 10 10
10
13
10
Figure 9: Query
Figure 10: Value
Figure 11: Key
Figure 12: Dense
Figure 13: Effect of Layer Type: We compute the directional score throughout the fine-tuning process. The Dense layer is the only layer that remained consistently (almost) monotonic. Each color represents a different Model Tree root
A MODEL CARDS ANALYSIS
To establish the benefit of using the model weights opposed to the metadata of the model, analyzed over 800, 000 model cards from Hugging Face. We found that least 36% of all models (roughly 290/)) do have model cards. models, about 35% model cards had no useful information about the pre-trainne traininn models. Overall, about 60% of the models (about 470K) have no model cards or have winformative model cards. Even for the (about 330K) models with "informative" cards (about 40% of all models), the cards often did not describe their parent but often just the root node. have an exact percentage
Llama analyze the remaining cards and found
for about 510K remaining
verified the last point manually, and there models that actually have the parent node, but based on a manual inspection of 500 randomly sampled model cards, we estimate this to he less than half of the remaining models (ahout 165K). See Fig. 14 for a flowchart with the above schematic.
ground truth, do
Model Cards on Hugging Face
Rugging Face models (-800k)
No model card (-280K)
Has model card (-510K)
41%
Unintormative or model card (-180K)
'Informaivva modal cards (-330K)
21%
20%
Partisl information" (-16GK)
information (- 165K)
Estimated
Only Model Tree roet information given, Not enough to determine the exaci location the Model Tree
Figure 14
14

---

Published as a conference paper at ICLR 2025
B
DATASET DETAILS
For all the MoTHer dataset subsets, the following models the Model Tree roots taken from Hugging Face:
the FT split, to prevent model overlitting, we use larger datasets 10K samples rather than the original 1K used in the VTAB benchmark. Each model uses a different randomly sampled seed. See Tab. 2 for additional hyperparameters.
Apart from the rank and seeds, both LoRA-F and LoRA-V use the same hyperparameters. LoRA-F uses fixed rank and alpha of 16, LoRA-V ranks sampled randomly of the options shown in Tab. 3. Both use the VTAB IK datasets shown Tab. and random seeds. Tab. for additional hyperparameters.
Table 2: Full Fine-tuning hyperparameters
Name
Value
1r
[6e 3, 9c 3, 2c 4, 5c 4] 64
batch_siit size
epochs
2 -
cifar100, svhn, patch_camelyon, clevr-count, clevr-distance, dmlab
datasets
Table 3: LoRA Varying Ranks Fine-tuning hyperparameters
Name
Value
Jora_rank (r) lora_al pph (a) 1r
8, 16, 32, 64
8, 16, 32, 64
(6e 3, 9c
3, 2e
4, 5e 4]
batch_size epochs
128
[10 20
cifar100, caltech101, dtd, flower102, pet37, svhn, came__c
datasets
clevr-count, clevr-distance, dmlab, kitti, Saprites-location, diprites-orienntttio,,
calllnorb-azimune smallnorb-elevation
C STABLE DIFFUSION MODELS
For Stable Diffusion we used the following versions found on Hugging Face:
15

---

Published as a conference paper at ICLR 2025
1.0
0.9
0.8
Tragefidt
Iragehes-222 2IX
0.5
0% (Onginal)
30%
3099
9090
91%
539
945
85
989
97%
90%
5990
Pruring
Figure 15: Pruning Ablation: Our method robust to model pruning. For example, with 90% pruning, the accuracy decreases by only 4%, and even 95% pruning, it drops by just 9%. Remark- ably, when 99% of weights are prumed, our method still achieves 68% accuracy (random baseline roughly 5%)
Model Graph
# Pruned Params
Non-pruned
Pruning %
ImageNet
ImageNet-21k
MAE
DINO
MSN
Params
0% (Original)
0.9
0.9
0.8
0.85
0.89
85,524.480
10%
0.9
0.9
0.8
0.85
0.89
8,552,438
76,972,042
30%
0.9
0.9
0.8
0.8
0.88
25,657,339
59,867,14
50%
0.9
0.85
0.75
0.8
0.86
42,762,240
42,762.240
70%
0.9
0.85
0.8
0.8
0.87
59,867,141
25,657,339
90%
0.9
0.85
0.8
0.7
0.85
76,972,042
8,552,438
91%
0.95
0.85
0.8
0.6
0.84
77,827,276
7,697,204
92%
0.95
0.85
0.8
0.84
78.682,510
6,841,970
93%
0.95
0.95
0.8
0.5
0.84
79,537,744
5,986,736
94%
0.95
0.95
0.8
0.5
0.84
80,393,027
5,131,453
95%
0.9
0.95
0.8
0.45
0.9
0.8
81,248.261
4,276,219
96%
0.9
0.9
0.8
0.4
0.9
0.78
82,103,495
3,420,985
97%
0.9
0.85
0.8
0.4
0.9
0.77
82,958,729
2,565,751
98%
0.9
0.9
0.6
0.4
0.85
0.73
83,814,012
1,710,468
99%
0.8
0.9
0.45
0.45
0.8
0.68
84,669,246
855,234
Table 4: Pruning Robustness Ablation: Our method is robust to model pruning. For example, with 90% pruning, the accuracy decreases by only 4%, and even at 95% pruning, drops by just 9%. Remarkably, when 99% of weights are pruned, our method still achieves 68% accuracy (random baseline is roughly 5%)
D ADDITIONAL ABLATIONS
D.I ROBUSTNESS TO PRUNED MODELS
We test the robustness of our method to model pruning. Specifically, we fine-tuned new VIT model graph with a structure similar to the FT graph (5 Model Trees, each containing 21 models). We used this Model Graph and incrementally pruned weights from the models using the runttuuuteeed function in torch. nn. ut 1s. p.rpuuu and evaluated our method on the pruned Model Graphs.
The results show that our method is robust to significant pruning (see Fig. 15 and a detailed breakdown Tab. 4). For example, with 90% pruning, the accuracy decreases by only 4%, and even at 95% pruning, it drops by just 9%. Remarkahly, when 99% of weights pruned, our method still achieves 68% accuracy (random baseline is roughly 5%).
16

---

Published as a conference paper at ICLR 2025
Quantization Method
ImageNet
ImageNet-2Ik
MAE
DINO
MSN
Model Graph
None (Original)
0.9
0.9
0.8
0.85
0.89
fp16
0.9 0
0.9 0
0.765 + 0.022
0.85 = 0
1 1
0.883 0,0044
Int8
0.9 0
0.895 + 0.015
0.77 = 0.024
0.85
=
1
0.883 0.0078
Table Quantization Robustness Ablation: Our method is robust to weight quantization, exhibiting less than 1% decrease in performance
D.2 ROBUSTNESS TO WEIGHT QUANTIZATION
We test the robustness of our method to model quantization. Specifically, we fine-tuned new VIT Model Graph similar the FT graph in the paper (5 Model Trees, each containing models). We then applied quantization half of the models with 50% probability, yiclding Model Graph where 50% of the models are quantized. This experiment was repeated 10 times to generate diverse quantized Model Graphs.
We tested 2 quantization methods: i) Simple quantization to fp16, ii) Int8 quantization using bitssadbytess. Our method is robust to weight quantization, exhibiting less than 1% decrease in performance (see Tab. 5).
D.3 DIRECTIONAL WEIGHT SCORE ANALYSIS
To further ablate the directional weight score, fine-tuned set of VIT models under varying learning rates and training steps. Specifically, fine-tuned models for each learning rate in the set [le 5e 3, le - 4, le- 4, 5e 5, Ie 5, le 6]. Each model was fine-tuned on CIFAR- 1A0 with a unique seed for 10 epochs.
Our results show that for all models that successfully converged, the Directional Weight Score was monotonic (see Fig. 16). For the two non-convergent learning rates (5e - 3 and le 2. which achieved an accuracy below 20%) the Directional Weight Score longer monotonically decreases. Notably, we observed that some point during training (which varies across learning rates), the Directional Weight Score becomes noisy. Upon inspection, this corresponds to the model's validation loss plateauing and becoming noisy, indicating convergence.
E MODELS WITH MIXED HERITAGE
We conducted a preliminary experiment explore models with mixed heritage. We started with the ImageNet, MAE, and DINO pre-trained base models and merged each pair of models using standard uniform weight averaging as described in Model Soups (Wortsman al., 2022). Subsequently, we fine-tuned 5 models from each of the original and merged models, yielding a total of 30 fine- tunet models.
We evaluate the ability of our method to handle merged models 2 settings:
Clustering: We first clustered the 30 fine-tuned models into 6 clusters, which resulted in perfect clustering accuracy.
2. Parent Detection:
To determine the parents of each merged model, we calculated the distance between each merged model and the centers of its potential parent clusters. Specifically, we computed the weights across all 5 fine-tuned models each parent group (ImageNet, MAE, and DINO). For each merged fine-tuned model, we calculated the cosine similarity to each parent cluster's center.
In Tab. 6 we summarize the averaged cosine similarity between the merged models and the pre-trained clusters. As shown, the cosine similarity to the parents is significantly higher than the unrelated pre-trained model, which enabled us to correctly identify the parents all merged models with perfect accuracy.
17

---

Published as a conference paper at ICLR 2025
10.34
12.00
30,00
10.00
20146
12.35
20.79
12:00
AP.
1956
TWN
ard vets Hark nata
41000
1177
le
6
(b) 5e
le
5
122
12.0
134
175
27.14
66
1306 195
390
303
200
450 SIX
300 M00
300 -AA 360 /e6
Mi 200 edo 5AW oee 1009
JG3
198
(d) 5e 5
(c) le 4
5e 4
8.0
230
gg) 3
(h) 5e 3
le 2
Figure 16: Directional Weight Score Analysis: We inspect the directional weight score for a wide range of learning rates and number of steps. For all models that successfully converged, the Directional Weight Score was monotonic. For the two non-convergent learning rates (5e 3 and 1e-2, which achieved an accuracy below 20%) the Directional Weight Score longer monotonically decreases. Notably, we observed that at some point during training (which varies across learning rates), the Directional Weight Score becomes noisy. Upon inspection, this corresponds the model's validation loss plateauing and becoming noisy, indicating convergence, In each sub-figure we plot 5 different models that were fine-tuned using the corresponding learning rate
ImageNet
DINO
MAE
ImageNet DINO ImageNet MAE
0.976
0.192
-0.001
0.847
-0,0002
0.524
DINO + MAE
-0.001
0.303
0.940
Table 6: Models with Mixed Heritage:
We show the averaged cosine similarity between the merged models and the pre-trained clusters. As shown, the cosine similarity to the true parents is significantly higher than the unrelated pre-trained model, which enabled to correctly identify the parents of all merged models with perfect accuracy. In bold are the chosen parent models according to the cosine similarity
18

---

Published as a conference paper at ICLR 2025
10 Samples
100 Samples
1k Samples
10k Samples
Pairwise distances (CPU)
0.033 sec
0.504 sec
5,468 see
5.01 min
Pairwise distances (GPU)
0.1 sec
0.697
0.005 sec
34.742 sec
Clustering (CPU)
0.011 sec
0.001 sec
0.134 sec
4.43 min
Table 7: Clustering Running Time: Our method scales even to larger Model Graphs
F RUNNING TIME ANALYSIS
We measure the runtime of the clustering phase. Specifically, simulated ViT Model Graphs of varying sizes observed the scalability the method. Notahly, our approach allows for clustering based on the weights of a single model layer without sacrificing performance, which provides significant speedups. As can be seen, our method scales even to larger Model Graphs (see Tab. 7). Note, that the runtime of running the minimum directed spanning search is negligible compared to the pairwise distance calculation and clustering.
G THEORETICAL INTUITION FOR CLUSTERING
While the primary focus of this work empirical, recent theoretical results provide support for the clustering of weights into trees. The literature on linear mode connectivity (LMC) has shown that models trained on the same data but with different random initializations converge to linearly related weights, up to a permutation of the neurons (Ainsworth et al., 2022). This implies that such models will be significantly distant from one another in &z-norm in weight space.
contrast, Frankle et (2020) demonstrated that models fine-tuned from a shared starting point (c.g., pre-trained foundation model) experience less neuron permutation and tend have weights that remain close to the original model. This establishes a clear distinction: root models our Model Trees (typically foundation models) are far apart in weight space, whereas their fine-tuned descendants remain relatively close to their parent model.
summary, these theoretical findings predict that intra- tree distances (between models within a tree) be smaller than inter-tree distances (between models from different trees), thereby justifying the effectiveness of clustering models into trees.
19


Reviews:
[{"review_id": "cy36VnPZ4p", "reviewer": "Reviewer_LoWV", "summary": "Motivated by the fact that many models have been publicly released, this paper proposes a new problem: studying the relationships between these models. Specifically, the authors aim to build a tree data structure where directed edges connect a parent model to other models that have been directly fine-tuned from it (its children). For each pair of models, this task requires: (i) determining if they are directly related, and (ii) establishing the direction of the relationship. Assuming that all models within the model tree share the same architecture, the authors propose a method based on the distance between model weights. Experiments demonstrate the performance of the proposed method.", "questions": "What is $\\mu$ in eq. (3)? What is the pretraining stage in Figure 2 (I thought it is all about fine-tuning)? Overall, I found section 4.2 is hard to comprehend.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Originality: This paper addresses a new problem: estimating the relationship between models and their fine-tuned versions. However, the significance of this problem for open models is debatable; see the weakness for the detailed comments.  \n\nSimple approach: The proposed approach based on the distance of model weights is simple. But this is based on a well-known fact that fine-tuning makes small weight changes.  \n\nWriting: The clarity is mixed; some parts are easy to follow, but certain important sections, such as Section 4.2, are hard to understand.", "weaknesses": "Limitation 1: The proposed approach can only handle open models, as it relies on model weights. For important open models that have been fine-tuned, information about the pretrained models is often available at the time of release. For models without such information, one can infer relationships based on weight distance. However, it is unclear why this information is needed for all released models.\n\nLimitation 2: The proposed approach constructs the model tree based on the weight distances between each pair of models and is thus limited to the case that all the models within a model tree share the same architecture. It can not be applied to other models that are obtained through distillation, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQpLGpc1Dy", "reviewer": "Reviewer_VKPa", "summary": "This paper targets to analyze the relation between models, aiming to shed light on which is fine-tuned from which model. This has also applications concerning copyright issues or more general licence concerns. Ths author introduces a method coined \"Model Tree Heritage Recovery\", which unravels the \"parent-child\" relations in a set of models. This method is unsupervised. Numerical examples are provided.", "questions": "see the weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Shedding light on the relation of models, in particular, in the LLM regime is crucial.\n* The numerics are convincing.", "weaknesses": "* Due to the importance of such a method for legal aspects, some theoretical underpinning should be given, which is currently missing. \n* The running time of the method is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ApyjBbb8Bu", "reviewer": "Reviewer_Xqqo", "summary": "This paper investigates the problem of finding the relationship between models from the model weights.", "questions": "My review is as follows:\n\n- I think this paper is well-written and investigates an interesting problem.\n\n- The introduction mentions legal disputes over model authorship. Out of curiosity, are there any known examples of this kind of dispute?\n\n- Could you please elaborate on this point? \"Moreover, it can help identify models that resulted from the wrongful use of proprietary training data.\" It is not clear to me how the proposed method for determining model relationships could help with wrongful use of data.\n\n- Could the method successfully find the relationship between a quantized version of a model and the full precision model? Were there quantized models in the dataset?\n\n- The observation that the Directional Weight Score is monotonic with respect to the training steps is interesting but perhaps not concrete enough. I would expect this to strongly depend on the specific learning rate, number of training steps, and perhaps some other hyper parameters used in training. In my opinion, identifying when this observation tends to hold and when it does not would be important in order to solidify the findings of this paper.\n\n- Some follow-up questions on the monotonicity observation: Does this observation generalize across many different model types? On what kind of models has it been verified so far?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please see the \"Questions\" section.", "weaknesses": "Please see the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfJSa08Udp", "reviewer": "Reviewer_eEd1", "summary": "This paper introduces MoTHer Recovery, a method to automatically trace relationships between shared neural network models by analyzing their weights. The approach uses weight distances and distributions to determine which models were derived from others, creating a tree-like structure of model relationships without requiring training data or documentation. The authors validate their method through experiments and provide a dataset for future research in model heritage recovery.", "questions": "- What is the computational complexity of applying this method to large model repositories? Could you provide runtime analysis for different scales (e.g., 100, 1000, 10000 models)?\n\n- How does the method handle cases where models have been fine-tuned with different learning rates or optimization strategies? Is there a threshold where the relationship becomes undetectable?\n\n- For models with mixed heritage (e.g., merged weights from multiple parents), how does the method determine the primary relationship? Can it detect multiple parent relationships?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper writing is good, and presentation is clear\n\n- The paper introduces a novel and timely problem formulation (model heritage recovery) that hasn't been systematically addressed before\n\n- It develops an unsupervised approach that doesn't require access to training data and leverages inherent neural network weights to infer relationships\n\n- It provides empirical validation across different fine-tuning scenarios and demonstrates effectiveness on the Stable Diffusion model family", "weaknesses": "- The paper doesn't address how to handle models with mixed heritage (e.g., models trained on merged weights from multiple parents) or partial weight sharing\n\n- The clustering approach might not scale well to web-scale model repositories - needs more analysis of computational requirements\n\n- It can be interesting to understand how different learning rates or optimization strategies during fine-tuning affect the reliability of weight-based relationships. For example, will aggressive optimization or pruning obscure these signals?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8x8txd9Af", "reviewer": "Reviewer_x9Mb", "summary": "In this paper, the authors introduce the task of Unsupervised Model Tree Heritage Recovery(Unsupervised MoTHer Recovery) for collections of neural networks.", "questions": "1. You seem to use the existing graph algorithm and the model is also not new. Are there any novel points in the graph algorithm parts?\n2. Could you clarify the cluster method using (1) and (2)? Do you have any guarantee of this way? Why use (1) and (2) not other criteria? In addition, is the cluster method reliable in this task? Can you use other alternative ways?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and introduces the history of the model tree well.", "weaknesses": "The paper is good as an introduction paper. However, it seems to lack novelty in the methodology part. The dense matrix construction (6) is not new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)