Neural Phylogeny: Fine-Tuning Relationship Detection among Neural Networks
Authors: Runpeng Yu
Xinchao Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Phylogeny
Finetuning
Primary area: interpretability and explainable AI

Abstract: Given a collection of neural networks, can we determine which are parent models and which are child models fine-tuned from the parents?
In this work, we strive to answer this question
via introducing a new task termed as neural phylogeny detection, aimed at identifying the existence and direction of the fine-tuning relationship. Specifically, neural phylogeny detection attempts to identify all parent-child model pairs and determine, within each pair, which model is the parent and which is the child.
We present two approaches for neural phylogeny detection: a learning-free method and a learning-based method. First, we propose a metric that leverages the distance from network parameters to a fake initialization to infer fine-tuning directions. By integrating this metric with traditional clustering algorithms, we propose a series of efficient, learning-free neural phylogeny detection methods. Second, we introduce a transformer-based neural phylogeny detector, which significantly enhances detection accuracy through a learning-based manner. Extensive experiments, ranging from shallow fully-connected networks to open-sourced Stable Diffusion and LLaMA models, progressively validate the effectiveness of both methods. The results demonstrate the reliability of both the learning-free and the learning-based approaches across various learning tasks and network architectures, as well as their ability to detect cross-generational phylogeny between ancestor models and their fine-tuned descendants.

Published as a conference paper at ICLR 2025
NEURAL PHYLOGENY: FINE-TUNING RELATIONSHIP DETECTION AMONG NEURAL NETWORKS
Runpeng Yu, Xinchao Wang" National University of Singapore r. yuuuuu. mus. edu,, xi xinchao@nus. edu.
Stable Diffusion 1.4
Stable Diffusion 2
Stable Diffusion 2.1
Phylogeny
Acc
Averallcc
Finshane
Lineage Acc
Lineage Acc
Lineage Ace
Directio Ace
Direction Acc
Direction Acc
eroo1dmmode
-bslgmechic-att
Probsb byy
Protaciit
Probibility
51.0%)
scar ottirmmddee 2510 x1
hrantoringaa roods
orbanlonnghhorhhy dreambood
Probab Byy
Prodazilly
Probabillty
senpkoungtiict
Daichil
Probab Ity
Pro
Probabillty
Daich
Prozeaiity
Probability
DatchThiydiuukk
Sogdansoriaal go
Probob lyy
Protebiiiy
Probsbillty
Firehure
?
Findune
Llama
Llama 3
DreamBooth
Ave
Overrl Ace
Overall Acc Linsage Acc
Relationship
Linsage Acc
300
Lineage Acc
Diectoonce
Directin
AcectionAcc
Zhao Limnaaa 500 v2
12biDio Llama sh.v.1 Frocabilty
Stable Difis
Paroit
Fineuno
Chd
Probob Ity
Aspahpii.bbbane hnddin pretrained byy
B--wweden-a Madela Procesiity
000
sarapeired (amaa----- baakkeee
muraimagk//onne lama-3-88
Parer
Tivehue
CNd
allena/uuu-277
Prohab Byy
susts
g grmmiitiamaa ww..-00pnnSoll........
Procabllly sam
Stable Difie on 2.1
Probab Byy
wan
Prosablity nows
abei enjk phokestnchutt
Probab ity
Prozaality scars
(a) Task illustration
(b) Neural phylogeny detection for Stable Diffusion and Llama.
Neural phylogeny detection showcase. Fig. saoshows the idea of neural phylogeny detection. Fig. Tb presents the detection performance among Stable Diffusion models and among Llama models. The detailed experimental setup and discussion are included in the Sec. and Appendix A
Figure 1:
ABSTRACT
Given a collection of neural networks, can we determine which are parent mod- els and which are child models fine-tuned from the parents? In this work, we strive to answer this question via introducing a new task termed as neural phy- logeny detection, aimed at identifying the existence and direction of the fine-tuning relationship. Specifically, neural phylogeny detection attempts to identify all parent-child model pairs and determine, within each pair, which model the parent and which is the child. We present two approaches for neural phylogeny detection: learning free method and a learning-based method. First, we propose to use the norm of the network parameters to infer fine-tuning directions. By integrating this metric with traditional clustering algorithms, we propose a series of efficient, learning free neural phylogeny detection methods. Second, we intro- duce a transformer- based neural phylogeny detector, which significantly enhances detection accuracy through a learning- based manner. Extensive experiments, rang- ing from shallow
networks to open-sourced Stable Diffusion and LLaMA models, progressively validate the effectiveness of both methods. The results demonstrate the reliability of both the learning-free and the learning-based approaches across various learning tasks and network architectures, well as their
"Corresponiing Author.

---

Published as a conference paper at ICLR 2025
ability to detect cross cross-generationa, phyloogg between ancestor models and their fine-tuned descendants.
INTRODUCTION
Recently, the field of deep learning has experienced transformative evolutions. First, traditional "training from scratch" paradigm has shifted to the for more efficient model development, meanwhile, creating a substantial dependency on models and their derivatives. Additionally, with the expansion of the deep learning research community and commercial success, numerous open-source model repositories have been established, and models produced by "machine learning as service" are accumulating. These advancements signify the emergence of a complex, interconnected network of deep learning models. Studying this network can help us better understand the inheritance of knowledge between models and the propagation of bias and unfairness, also facilitating the management of intellectual property. However, such research relies on the ability to first unveil this underlying network of neural networks.
approach, which allows
In this work, we define and investigate the task of neural phylogeny detection, as illustrated in Fig. Specifically, given a parent neural network fp. the model fc. which is fully fine-tuned from it, is referred to its child model. Considering set of neural networks, which includes several parent models and their fine-tuned child models, the goal of neural phylogeny detection is to identify all parent-child model pairs within this set and determine which model is the parent model and which is the child model in each pair.
We propose two approaches address the neural phylogeny detection task: learning-free method and learning-basee method.
To develop the learning-free method, first investigate how to determine the direction of fine-tuning Specifically. given two models with a fine-tuning relationship, we aim to identify which is the parent model and which is the child model. Through theoretical analysis, we discover that the norm of neural network parameters serves as a simple yet effective method for determining the direction of fine-tuning. We validate this theoretical result on both simple ResNet architecture, and more complex models, including Stable Diffusion and LLaMA models. For instance, in the case of Diffusion Models, using parameter norms enables us to identify the fine-tuning direction with an accuracy exceeding 95%. Building on this insight, we propose a learning-free method for phylogeny detection.
In the learning-free free method, phylogeny detection is treated as a clustering task. Parent models and their fine-tuned child models form clusters, with the parent identified as the centroid. Using clustering to achieve phylogeny detection involves two sub-tasks: clustering models into clusters and identifying the parent model within each cluster. The clustering sub-task can be accomplished using traditional clustering algorithms, while the task of identifying the parent model is handled by the norm of the parameters. We explore two workflows to achieve these two sub-tasks. The first workflow performs clustering first, and the parent in each cluster is identified after the clustering algorithm is complete. The second approach integrates the parent model identification into the clustering algorithm, and by the end of clustering, the parent models within each cluster are also identified. We apply these two strategies commonly used clustering algorithms, resulting in a series of learning- ffree methods for efficient neural phylogeny detection.
In the learning- based approach, we frame neural phylogeny detection as a directed graph learning task. Each model is treated as a node, with theettunig relationships forming directed edges between parent and child models. We design a transformer-basee this graph. Unlike the learning-fres method, which relies on specific metric, the bearning-based approach learns to infer the existence and the direction of finetuning, achieving better performance.
detector to predict the adjacency matrix of
The contributions of this work are:
We investigate the novel task of neural phylogeny detection, aiming to identify parent-child model
pairs among neural networks and determine the fine-tuning direction.
We propose two approaches: a learning-fre free method that is convenient to implement and a learning- based method that offers higher prediction accuracy.
We demonstrate the effectiveness of the proposed methods on various deep model architectures from shallow fully-connected nerual network to large open-source models; under different learning
2

---

Published as a conference paper at ICLR 2025
settings, including few-shot learning and imbalanced data learning; and for cross cross-generne generationn neural phylogeny detection.
2 RELATED WORKS
2.1 NEURAL LINEAGE AND DEEP MODEL IP PROTECTION
This work is most closely related to neural lineage detection in (Yu & Wang 2024), which identifies from which parent model a given child model has been fine-tuned. Neural phylogeny detection is more challenging task and has substantial differences compared with neural lineage detection. (1) In neural lineage detection, the sets of parent models and the set of child models are distinct with no overlap, thus the direction of fine-tuning is known; whereas in neural phylogeny detection, all models are mixed together, and the direction of fine-tuning is to-be- oo-be-determined. (2) In neural lineage detection, the finetuning dataset is accessible and the forward inference and gradient propagation of the models are required. Conversely, neural phylogeny detection is data-free and does not require the forward and backward of the model. (3) Neural lineage detection is a classification task, while in this work, neural phylogeny detection is treated as a clustering or graph structure learning task.
On the other hand, by incorporating information related to the training data, neural phylogeny detection can be simplified into neural lineage detection. For instance, knowledge of the input and labels from the dataset used for pretraining or fine-tuning would facilitate the differentiation between parent and child models through measures such as accuracy, loss values, or gradient magnitudes, effectively reducing the phylogeny detection to lineage detection.
Another related task deep model IP protection, which commonly addressed through model watermarking and model fingerprinting. Model watermarking identifies stolen model via an embedded watermark (Uchida et al. 2017 Chen et al. 2019 Li et al. 2020b 2022, Lou et al. 2021
Chen et al.) 2021b] Lounici et al. 2021 Lao et al., 2022 Nagai et al. 2018 Zhao et al. |02 Xie
et al. 2021 1: Darvish Rouhani et al. 2019] Adi et al. 2018 Ye et al., 2023
Jing et al. 2021 2023),
whereas model fingerprinting discovers the source model and stolen model based their behaviors
across a set of conferrable samples (Cao et al. 2021 I; Lukas et al.] [021] Peng et al. 2022 Li et al.
2021 Pan et al. 2021 Zhao al., 2020; Yang al. 2022] Wang & Chang 2021h. Deep IP protection also claims that TP protection methods are capable of identifying source model and its child models
against the fine-tuning attack. However, in deep IP protection, fine-tuning is often conducted using minimal learning rates and few epochs to maintain the child model's performance on the original
2021 Li al. 2020a Cao al. 2021 Pan et al. 2022, task (Darvish Rouhani et al. 2019 Jia et al.,
Fan et al.] 2019; Lee et al. [023]] Adi et al. 2018). which differs significantly from the fine-tuning process in practice. By contrast, the fine-tuning neural phylogeny more aligned with practice, in which finetuning is method for transfer learning.
2.2 NEURAL NETWORK LINEARIZATION
In this work, we build our theoretical analysis upon the framework and conclusions of neural lin- carization. Unlike previous studies that applied neural linearization pruning at initialization (Wang
et al. 2020 Gebhart et al. 2021 Fang et al. 2023 Ma et al. 2023), training-free neural architecture search (Chen et al. 2021a Wang et al. [202 Mok et al. al. 2022). and training time prediction (Zancato) al.). approach represents new of linearization theory, specifically for detecting fine-tuning direction. The thcoretical foundation of neural linearization suggests that, as network width grows, the network can be well represented by its first-order Taylor expansion initialization (Lee et al. 2019a). Further research indicates that these linear approximations can achieve comparable or even superior performance their nonlinear counterparts (Ortiz-Jiménez al.) 2021] Arora et al.) 2020). By leveraging neural linearization in our method, we explored which metrics exhibit monotonic changes during the fine-tuning process, allowing them to be used to determine the direction of fine-tuning.
2.3 GRAPH STRUCTURE LEARNING
In learning-based based methods, treat the problem of neural phylogeny detection novel application of graph structure learning. Unlike previous works that have applied graph structure learning to tasks like automatic diagnosis Cosmo et al. (2020) and protein structure prediction Jumper al.
3

---

Published as a conference paper at ICLR 2025
(2021), our approach specifically targets the identification of phylogenetic relationships in neural networks, which presents unique challenges and opportunities. Graph structure learning generally aims to identify valuable graph structures directly from data, resulting in undirected [Cen & Wu (2022), weighted Chen & Wu (022), or directed graphs (Yu et al. 2019). Prior research has explored aspects such the interplay between structure learning and downstream tasks, the parameterization of adjacency matrices, training loss design, and post-processing methods like discretization and sparsification Liu et al. (2022); Chen al. 20200; Fatemi al. (202T); FFannceschi al. (2019);
et al. [020);; Wang et al. (2021 y Yu et al., (2020). Our work differentiates itself by focusing on the
adaptation of these techniques to capture evolutionary relationships between neural networks.
METHOD
3.1 NOTATION AND TASK
Notation. We use ʃ to denote a neural network and D = (X, to denote the training dataset, where X' = {x : (z. y) D) and y {y (z, y) E D) are the inputs and the targets, respectively. We use 0 to denote the parameters of the neural network and present it in the vectorization form to facilitate notation. For example, we write
R.dox1 where dg is the number of the parameters in Given set of neural network models, we use the superscripts index the models. Specifically, f(i is used to denote the i-th model in the set. The parameter of f()) is denoted by g(+), Given matrix, let Zi,; and ži.j to denote its i-th row and its element in the i-th row and j-th column, respectively.
Given a directed graph G. let ei,j denote the edge from the i-th node to the j-th node. Let 1() denote the indicator function.
Task. The objective of neural phylogeny detection is to first identify all parent-child pairs within set of neural network models. Then, it seeks to determine the direction of fine-tuning between each detected parent and child pair. Specifically, given set of neural networks, (. ===1 containing parent and their child models, neural phylogeny aims find binary matrix A, where A.j = if the i-th model f(8 is the parent of the j-th model f() and Aij otherwise. In this work, neural phylogeny detection is conducted in a completely data-free manner, meaning that we only access the parameters of the neural networks without any information related to their training data.
3.2 DIRECTION OF THE GRADIENT DESCENT
In this subsection, we investigate how simple metric the norm of neural network parameter can be used to detect the direction of finetuning. We begin by evaluating the feasibility of this metric in simplified theoretical setting, and conclude by validating that this simple metric can still achieve reasonable accuracy in predicting the direction of finetuning on open-source models such as Llama and Stable Diffusion.
Neural phylogeny detection comprises two sub-tasks: identifying parent-child pairs and determining the direction of fine-tuning, or, in other words, the direction of gradient descent. The detection of parent-child pairs can be effectively achieved using the similarity between models (Yu & Wang 2024). Therefore, the crux of phylogeny detection is the determination of the direction gradient descent.
We first demonstrate through the Prop. below that, in theory, the norm of neural network parameters increases during the training process. Therefore. the norm of the parameter serves as a theoretically feasible metric for determining the finetuning direction. In Prop. [ we follow the assumption and formulation in neural network linearization Lee al. (2019b) to describe the dynamic of the neural network training. Previous works have shown that, for wide network, the linearized network can be used effectively and efficiently approximate the dynamic of the original non-lineat network in the gradient descent.
Proposition 1. Let fü denote neural network trained using gradient flow on the dataset D (X' ) at time t. Let 0 E Rá denote the vectorized parameters of fo. where dg is the number of parameters. Let fo denote the neural network at the initialization with initial parameters dg. Let n denote the learning rate, Let O denote the neural tangent kernel at the initialization. Let e denote the small term, whose specific definition is included in Appendix D. Then, with MSE loss, the squared
4

---

Published as a conference paper at ICLR 2025
Table By comparing norm of the parameter, the parent model in a parent-child pair can be detected with high accuracy.
Stable Diffusion 1.4
Stable Diffusion 2
Stable Diffusion 2.1
Llama 2
Llama
Accuracy
96.49
95.12
97.33
88.57
58.01
l distance between 0, and z can be approximated as,
1 10:- - =112
dist(O,, z)
V
- + (I)
do
do
which increases over time.
Prop. demonstrates that the norm of parameters O, is a function that increases over time which validates the feasibility of using such norm to determine the direction of neural network training. Furthermore, in the supplementary materials, we extend the above Prop. | to scenarios involving weight decay regularization. It can be proven that the norm of the parameters still increases during training, provided the weight decay coefficient is not large. This is consistent with practical scenarios, as large weight decay coefficient can impair the performance of the network and is therefore not utilized.
To assess the practical usefulness of the theoretical results, we conducted two analyses. (1) We trained eight ResNet18 networks using different combinations of hyperparameters and fine-tuned them. Fig. [ records the norm of their pa- rameters during both the training and fine-tuning processes, The distance increases both during the pre-training phase and the subsequent fine-tuning phase, which validates the conclusion in Prop. [ (2) We downloaded parent Sta- ble Diffusion models and parent LLaMA models, along with their finetuned child models from HuggingFace. In total, we downloaded 366 Stable Diffusion models and 95 Llama models. We evaluated whether the norm of the parameter could effectively detect the fine-tuning direction among open-source models. Tab. D reports the performance, showing that the norm can consistently identify the finetuning direction with high accuracy. To mitigate the curse of dimensionality and reduce the computational costs, in this and subsequent experiments, use subset of the parameters selected based on validation set composed of 10% of the models.
Norma of Pazmudeer
Traiiin Ploee
Pine Pleao
Taining Iteration (x 163)
During training and fine- tuning, the norm of the neural network parameter continually increases, consis- tent with theory.
Figure 2:
3.3 NEURAL PHYLOGENY DETECTION VIA CLUSTERING
In this section, we discuss a learning-fiee free approach for neural phylogeny detection. The main idea is to view the neural phylogeny detection as a clustering problem, combining the detection of fine-tuning direction with traditional clustering methods.
Given set of neural networks, the goal of neural phylogeny detection is to solve for an allocation matrix A, which can be seen as a clustering task. Every model in { LM Ji=1 can be regarded as data point to be clustered. Matrix A allocates these data points into non-overlapping clusters. Each cluster should contain a parent model and the child models fine-tuned from it. For each cluster, the parent model should be identified as the cluster centroid.
However, the problem with previous clustering algorithms is that many algorithm not inherently include the identification of cluster centroids, such as DBSCAN (Hahsler al.) 2019). Even for those algorithms that can define cluster centroids during the clustering process, such as KMeans (Lloyd 1982) and MeanShift Wu & Yang 2007), the cluster centroids identified by the algorithm may not necessarily be the parent model. Take the KMeans algorithm as an example. When the distance between points is measured using & or & norms, the centroid is the mean or median of the points within the cluster. However, from the analysis and empirical findings in Sec. 3.2 it can be seen that the parent model might not locate near the geometric center of the cluster. Because parent model has smaller parameter norm.
5

---

Published as a conference paper at ICLR 2025
There two approaches to integrating the identification of parent models with clustering algorithms. straightforward method is, first, using a clustering algorithm to identify the clusters; then, for each cluster, identify the parent model in the cluster. Specifically, suppose C' C 21. M } is the set of indices of the models in one cluster, the index of the parent model in this cluster can be defined as
arg min 1
For the algorithms that require evaluation of cluster centroids during clustering, the other way of integrating the identification of parent models with the clustering algorithms is by incorporating the assessment of fine-tume fnne-tuning direction directly into the centroid evaluation process. Below we take the KMeans and MeanShift algorithms examples to illustrate. In Appendix B33 Fig. using KMMann algorithm as an example, we provide a diagram of the algorithm's 's workflow to visually illustrate how method integrates the finetuning direction identification into the original clustering algorithm.
KMeans. K.Maan algorithm executes the assignment step and the update step iteratively. During the assignment step, each model is assigned to the nearest cluster. This step is essentially neural lineage detection operation, which can be address effectively using the common distance measurements (Yu Wann 2024) and does not involves the identification of the parent model. Thus, we keep the original workflow in the assignment step and use the & norm as the distance measurement to assign each model to the nearest cluster. In the update step, the KMeans algorithm updates the centroid of each cluster, which in our task should be the parent model. The prediction of the parent model in the cluster can be achieved using Eq. (2). However, during the iterative process, a single cluster may have none or more than one parent model. Therefore, to enhance the robustness of the method, we introduce probability parameter a. When updating the cluster centroid, with a probability of a, the cluster's mean is used as the centroid, and with probability of 1 a, the new centroid is chosen using Eq. Q After the convergence of the algorithm, the final prediction of the parent model is drawn again using Eq. (
MeanShift with a flat kernel. Given a to-be-optimized cluster centroid, mean shift updates the centroid based on the mean of the points within the neighborhood of the centroid. The method for calculating the new cluster centroid can also be represented as finding the point that minimizes the sum of distances from the centroid other points in the neighborhood, similar the update step in the KMeans method. Thus, we employed the same approach as in KMeans by introducing probability parameter and randomly choose to use either the mean of the points in the neighborhood as the new centroid or use Eq. (2) to select the new centroid.
For approaches that identify the parent along the clustering, the choice of parameter a signif- icantly influences the perfor- mance by altering the proportion of iterations using the predicted parent as the cluster centroid. We empirically studied the impact of a values on the final predic- tion accuracy in Fig. Bal When = 0, each update uses the pre- dicted parent as the cluster cen- troid. When = 1, the "identify along" approach degenerates into the original clustering algorithm, always using the internal mean as the new centroid. However, neither of these extreme methods achieves optimal performance. Generally, using relatively small a, primarily relying on the predicted parent as the new centroid while occasionally using the mean, yields the best results.
Eilles Jana Chatwery Kileaas Afhe Chatoen Mossdigg Clastine
Ctjedt
(a) Different a.
(b) Clustering convergence.
Figure 3: Ablation on the value of a and the convergence of the
clustering algorithms.
Besides prediction accuracy, the convergence speed of the clustering algorithm is also a crucial factor in evaluation. We observe that in a clustering task involving 277 models, shown in Fig. 3b both the original clustering algorithm and the one embedded with parent model identification typically converge within 4-5 iterations, demonstrating high efficiency. Additionally, we observe that some dig points appear in the "identify along" approach, marked by red dots in the figure. The occurrence of these dig points is due to the change in the centroid updating method, leading to fluctuations in node

---

Published as a conference paper at ICLR 2025
allocation and consequent accuracy decrease. However, altering the method of centroid updating also corrects accumulated errors during iteration, resulting in higher accuracy upon full convergence.
3.4 NEURAL PHYLOGENY DETECTOR
In this section, we propose learning- baased phy- logeny detector. We treat the task of phylogeny detection as a structure learning problem for directed cyclic graph G. Given a set of neural
Edge Token
Adaptive Pooling
FC to 10 Reshape
networks { Ji== ] each network is a node in
Conv & Norm ReLU
graph G, with corresponding network parameter ((0) being the feature of the node. The adja- cency matrix of G is what we aim to predict in phylogeny detection. The structure the phy- logeny detector, as shown in Fig. 4 includes an edge encoder that takes the source node feature and the target node feature of each edge as input, outputs the latent embedding of each edge, and transformer detector that predicts the probability of the existence of each edge. The following outlines the detailed workflow the phylogeny detector.
Transformer B Boocks
Source Node Weight
Pollion Emtoddiing
Taroe Nod
Weight
Edge Token
Edge Encoder
Transformer Detactor
Figure 4: The proposed phylogeny detector.
First, we vectorize and reshape each weight into matrix 0 E # with shape depending on the specific architecture of the parent and child models. We stack the weights from node and node
j together and use it as the input feature of the edge 6i and denote it by (6.5 & [0(0; 065) E R2*HxW, Such a stacking strategy is more effective than encoding the weights of the parent and child models independently, by allowing for element- comparison of the naturally aligned weights from different models, capturing subtle changes in weights caused by fine-tuning. We use w E Mx x xW to denote the complete input of the edge encoder, which includes all edge features. We use a convolutional network as the edge encoder D, applying batch normalization and ReLU activation after each convolution layer. At the end of 4 we use an adaptive pooling with output size of 1 x 1 to transform each edge's latent embedding into vector, which serves as the token input to the subsequent transformer detector V. The output of O can be written as wene e (w)) E RM xd) where d is the token size for the transformer detector.
We use position embedding EPos E RM (MM to encode the indices of the source and target nodes of each edge, allowing the transformer detector to utilize the relative relationships between edges during processing. First, based on the node indices, we generate the position embedding for each node eoos E x The position embedding for the edge Ci.j the concatenation of the position embeddings of nodes and j. that is, _e e Rd. The transformer's input the sum of the edge tokens and their corresponding position embeddings. Finally, we use linear head to transform the transformer's output for each edge token into a scalar score, which can represented as
final (w + EPPs), s = (ue final E (MX)))x and s E RMM (3)
where s reshaped to ease the notation.
A characteristic of the directed graph formed by neural phylogeny is that there is at most one outgoing edge from each node. This results in the directed graph being very sparse, and using & or binary cross-entropy loss on all edges would lead to imbalanced supervision signals. Therefore, we use cross-entropy as the loss function. For node i, if it a child model, its ground truth label g4, the index of its parent model; if it is parent model, its ground truth label is defined as its own index i. The predicted probability of the edge from node i to node j is defined as Pij - softmax(s;; (st:)) According, during inference, let j A arg max, Pi.j. if j", predict that model is parent model; otherwise, we predict that model is fine-tuned from model j".
4 EXPERIMENTS
Proof of Concept. We experiment on 4 model architectures: a three-layer fully- colly-connected network (FC Net) with ReL.U activation and no normalization; three-layer convolutional network (Cony
7

---

Published as a conference paper at ICLR 2025
Table 2: Phylogeny detection performance for the vision models. The best (the second-bes -bootdd-ees) ones are marked in bold (underlined).
Cifar 5shot
Cifarl0 50shot
Cifarl0 Imbalanced
Method
FMNIST
EMNIST
Cifar10
SVHN
DBSCAN
15.64:10.17
13.16=0443
24.8611.99
31.32:+22,46
7.35=
41.61:1:.90
39, 19.10=0.92
GMM
9,4811.36
3.43-0.16
65.71+1.59
16.59.11.19
9.71=0084
42.34:1185
65.93=112
FC Net
KMMaans
15.80++0.39
12.54-0.47
53.11+2.40
1..2
9.71+0.84
43.80-+2..8
61.54=0.81
MeunShift
14.60:10.55
12.77=0.44
2..12:1..22
33.73:12.40
6.56
42.54:1:.65
40.11=1.12
KMMaanssllog
18.64+00.49
20.09-0.56
6.253+1.26
45.38+1.155
9.81+0.54
88.32-+0.70
62.24-0.61
MeunShift-atong
15,48:10.21
03.55=0045
40.11
34.13:12.49
12.60-10.3
98.87-100.22
56,59 1.43
Phylogeny
87.78:00.25
84.41=0.26
70.86=2.119
50.11:2.111
48.61=00.6
99.52:10.12
68.73=4.25
Detector
DBSCAN
28.30++00.85
45.83 1.20
27.60 1.22
44.53+00.57
12.32+0.83
23.10++2.36
42.89-2.22 2.28
GMM
28.12:1:0.90
0.777=0.70
46.29:1.229
39.82-10.87
1..1
82.79:+1.42
0.74
ConvNer
KMMcans
25.79++0.81
28.29=0.65
45.70=1.52
39.05=00.16
16.33=1.17
77.42++0.99
51.63=0.85
MeanShift
18.15++00.4
15.52+0.32
04.74
21.39
6,20+0.73
94.62++1.23
24.19 1.32
KMcanS-aloog
27.40=t:044
02.36=0.68
1.59
41,21:+0,58
16.76=0.39
78.51+44.34
54.68-0.87
McanShift atong
26.334:0.77
44.28=1.10 1.10
45.99
39.82:1019
37.25=1669
94.62441.3
47.16=0.80
Phylogeny
Detector
83,64:10.77
60.12:1.56
99.17=10229
98.86-10.42
82.54-0..7
99.81-10.15
96.97-0.88
DBSCAN
19.43++1.46
38.50=0.92
43.70
55.17:-2.80
11.95=0.61
28.14433.55
34.69=2.00
GMM
14,29++0..4
28.74-0.65
24.28 1288+1.40 1.44
48.27++2.46
11.80+0.67
28.57+3.52
25.51+2.26
ResNet
KMcans
28.01=11.59
41.38=0.94
48.55
71.55:10.49
11.5620.64
28.9843.60
32.67=2.01
McanShift
18.43++1156
37.36=0.98
43.75-22.74
55.23-+2.488
11.60=0.65
28.84443.52
32.41=1.95
KMcaas-siona
28.57++1.75
43. 10=1-2
04.47
71.68:22.16
11.74=0.59
29,0443.33
33.25=2.78
MeunShift atong
19.43:1.226
38.50=0.92
1.89
55.64:12.40
11.8210.49
29.35443.24
33.68=2.79
Phylogeny
83.01:10.64
88.46 1.57
90.18:10.77
88,39:10.73
1..99
$7.94
87.01 1.30
Detector
DBSCAN
39.14:t2.00
1.56
55.42:44.24
16.76:t1.93
70.01
55.01==1.97
26.46=2.35
GMM
47.83-12.79
1.77
72.80
47.60:12.55
55.01
53. 53,36:12..2
25.01=1.62
KMcans
44.94:22.52
25.38=1.54
70.61
46.33:12.34
50.17=0.05
56,2142.51
1.6
VIT
McanShift
43.49:42.42
26.
47.0924.13
30.87:2..26
65.75=4.76
45.01+8..99
16.
KMcans along
46.38:12.00
69.57=4.15
70.71
70.61+2..9
8..01
80,01114.68
67.27-2.53
MeunShift-nong
47.83-42..79
32.97
73.55 3.033
55.89
70.01 -6.12
45.53-14.28
54.41 1.79
Phylogeny
68.33+11.41
72.51+1..24
89.01 0.86
72.01+1.23
52.89+0.86
51.82++2.02
67.33-1.13
Detector
Net) with ReLU activation and no normalization; ResNet18; and ViT-Tiny. For fully-connected and convolutional networks, the parent models are trained by us on MNIST (Lecun et al. 1998)
and CIFAR-100 100 (Krizhevsky et al., 2009). For ResNet and ViT, the parent models are collected from the timm (Wightman| 2019). Parent models are finetuned on FMNIST (Xiao et al. 2017). CANNIIT-Leii
(Cohen al. 2017). and CIFAR10 (Krizhevsky et al. 2009) datasets to construct the child model sets. The average number of child(parent) models for the four architectures was (644,, 756(44), 106(7), and 64(3), respectively. We compare the detection accuracy of methods in three categories: (1) DBSCAN (Hahsler et al. 2019), GMM (Ouyang et al. 2004), KMeans (Lloyd 1982) and MeanShift (Wu & Yang (007) clustering algorithms with the "identify parent after clustering" strategy: (2) KMeans and MeanShift algorithms with the "identify parent along clustering" strategy. Thereafter, we denote them by KMeans-along and MeanShifftuloog; and (3) the phylogeny detector. prediction is considered correct only if the parent-child pair is accurately identified and the direction of fine-tuning is correctly determined. For learning- free methods. we reported the average accuracy over 5-fold experiments. For phylogeny detector, we split the child models into training, validation, and testing sets in a 7:1:2 ratio and report the average accuracy over five runs on the testing set.
Results are summarized in Tab. 2 with key observations as follows: (1) Instead of using a fixed statistical metric, learning-basse detector generally outperforms the learning-free methods by significant margin. Compared to the best-performing Beamning-free method in each setting, the phylogeny detector's performance is, on average, 29.43% higher. (2) Among the learning- methods, no single clustering algorithm consistently outperforms the others. This is mainly due to the sensitivity of clustering methods to data structure, making the choices hyperparameter significantly affects the performance of the clustering algorithm. (3) By incorporating parent model
8

---

Published as a conference paper at ICLR 2025
Table 3: Cross- Pross-generational phylogeny detection.
Table 4: Ablations.
G2
G4
Embedding Dimension
FMNIST
EMNIST
DBSCAN
38. 08.50220.
38.23++:125
38, 1.95
GMM
28.74==0.65
23.27+2.773
14.29=00.45
84.44
77.42
KMeans
41.38:10.994
23.85+:2.56
14.35:10.52
16
87.51
81.99
MeanShin
37.36:10.98
3..35
37.27:42.00
32
87.78
83.07
87.91
GI
KMeans-atong
43.10=1.992
42.51+11.20
42,86-t:188
(a) Embedding dimeusion.
MeanShift-atong
38.50:20.92
18.82:::115
38.47:1:1.35
Phylogeny Detector
88.46:11.87
87.82:10.59
1.9
Encoder Depth
DBSCAN
40.35:::0.2
38.86 18.86-:1.95
FMNIST
EMNIST
GMM
29.24=11.32
14.86:4:0.53
28.07:1:1.13
26.29:10.70
85.83
79.84
MeunShift
40.35.10.62
40.02.1...21
88.89
84.41
88.33
84.41
KMeans-utong
42.69:+:100
40.86:+11.05
MeunShift along
00.96:1:.65
40.57:1:2.28
(b) Encoder depth.
Phylogeny Detector
77.56:20.74
72.75:t:1.35
DBSCAN
68.011122.09
Transformer Depth
GMM
67.43:1:0335
FMNIST
EMNIST
KMeams
53.71:1:1.59
MeanShift
58, 2.11.6122110
87.51
80.64
87.78
81.99
G3 KMeansalong
56.57:1:1.53
85.83
83.07
MeunShift along
68.05.1.2.12
78.76
Phylogeny Detector
88.46:1:0.92
(c) Transformer depth.
identification into the clustering process, the performance of K.Mcans-along and McanShiff-atong algorithms outperforms the original KMcans and MeanShift. The performance of KMeans and MeanShift methods improved by an average of 9.95% after using the "identify along" strategy.
Few-Shot and Imbalanced Data. We further evaluated the methods on CIFAR 10 datasets under few-shot and imbalanced learning settings in Tab. Generally, the learning-basee method still performs the best, and the strategy of identifying parents along clustering improves the performance of KMeans and MeanShift algorithms. However, in the few-shot setting for ViT models, due the difficulty of fine-tuning we obtained only an average of 16 child models, resulting in insufficient training samples for the phylogeny detector. Consequently, the learning-baid based method perform worse than the learning-free methods in this specific case.
Pross-ennraatonall Phylogeny. We define the 7 parent ResNet18 models as the First Generation (GI) models and perform three rounds of fine-tuning, one after another, on the EMNIST (G2), FMNIST (G3), and EMNIST-Balanced (G4) datasets. Tab. 3 compares the cross cross-generationa. phylogeny detection performance. The learning-based method and the strategy of identifying parents along clustering generally maintain their advantages. the table, we use pink squares to indicate the relative performance of each method under different settings, with darker colors representing higher accuracy and lighter colors representing lower accuracy. It can seen that as the generational gap increases, the performance of all methods generally declines.
Ablation on Detector Architecture, We conduct ablation on the structure of the phylogeny detector to observe changes in accuracy with increasing network complexity. We examine, in Tab. 4 three factors: the embedding dimension of the transformer, the number of layers in the edge encoder, and the number of layers in the transformer. The main observation that due the limited size of training samples, accuracy does not consistently improve with increased network complexity. The combination of 5-layer encoder and a 1-layer transformer generally optimal.
Experiments on Open-Sourced Stable Diffusion and Llama. We conducted experiments on more complex Diffusion Models and Llama Models. Specifically, we collected 366 Stable Diffusion models and 95 Llama models from the HuggingFace. Compared to the models used in the primary experiments, these models have larger parameter sizes, more intricate architectures, and more diverse training configurations. The child models were trained different datasets, using various hyperparameters and training techniques-nut limited full- full-parameter fine-tuning but also including
created by down-sampling the first five categories to its 1/10.

---

Published as a conference paper at ICLR 2025
Table The performance the learning- ferming-free method on open-source Stable Diffusion and Llama models, as well on Stable Diffusion models fine-tuned using DreamBooth.
Identify after Clustering
Identify along Clustering
KMeans
McanShift
KMcans
McanShift
Avreage
Best
Avreage
Best
Avreage
Best
Avrcage
Best
Stable Diffusion
14.92.1..45
18.64
18.97.11001
19.02
78.57121.28
99.73
26.16.124.75
80.54
Llama DreamBooth
0.10
0
0.10
48.83114837
97.67
75.58.137.81
95.34
100.0.10
100.0
100.0.10
100.0
100.0.10
100.0
100.0.10
100.0
Table 6: The performance of the learning-based method on open-source Stable Diffusion and Llama models, well on Stable Diffusion models fine-tuned using DreamBooth.
SD 1.4
SD
SD 2.1
Llama 2
Llama 3
Overall Accuracy
99.44-+0.25
99.12++0.7
99.33+-0.47
92.86++0.70
88.37--0.60
Lineage Accuracy
100.0++0.00
100.0+0.00
100.0+00.00
100.0+0.000
100.0+00.00
Direction Accuracy
98.64110.21
96.55:1.63
97.72:10.27
91.8111.40
80.00:10.44
techniques like LoRA fine-tuning. Subsequently, using DreamBooth (Ruiz et al. 2023), we fine-tuned Stable Diffusion 1.4 and Stable Diffusion 2.1 on the official DreamBooth dataset, resulting in 62 models (30 derived from each parent model and two parent models).
We evaluated the performance of our learning-free technique on these models, with results presented in Tab. [ Consistent with earlier findings, the performance of the learning-free approach significantly improved using the "Identify along Clustering" strategy. However, we observed that the performance of the learning-free method on these models was unstable, with substantial gap between its best and average performance. We further tested our learning-based method on open-sourced models collected from HuggingFace. The experimental results are reported in Tab. 6 where evaluated three metrics: Overall Accuracy, Lineage Accuracy, and Direction Accuracy. Let N denote the total number of models, then, there are Npair = N(NN1) model pairs. For the i-th model pair, gt, is the ground truth label, which can take three possible values: {0 = No Finetuning Relationship. The First Model is Parent, = The Second Model is Parent]. pred, is the predicted label for the i-th pair. The definitions of these metrics are as follows:
N Npacr
Overall Accuracy
1(gh pred,),
Npair
E=1
Npuir
Lineage Accuracy
1((tt = pred; = 0) OR (pred, * 0 AND gt: + 0))
N pair
im
Npair
Direction Accuracy
1(gh: + AND gl. predi).
E 1gt, #
The results in the Tab, demonstrate that the learning- based approach significantly outperformed the learning-free method, achieving nearly 100% accuracy even on complex open-sourced models.
CONCLUSION
We investigate the novel task of neural phylogeny detection, which, given a neural network models, aims to identify parent-child model pairs connected through the fine-tuning behavior and determine the direction fine-tuning. By treating neural phylogeny detection as clustering task or graph structure learning task, we propose both learning-fir and learning-based methods. Extensive experiments demonstrate the effectiveness of our proposed methods across various network architectures and learning settings, and for cross cross- noss-generationall phylogeny detection.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This project is supported by the National Research Foundation, Singapore, and Cyber Security Agency of Singapore under its National Cybersecurity R&D Programme and CyberSG R&D Cyber Research Programme Office (Award: CRPO-GCI-NTU-002,
REFERENCES
Yossi Adi, Carsten Baum, Moustapha Cissé, Benny Pinkas, and Joseph Keshet. Turning your weakness into strength: Watermarking deep neural networks by backdooring. In 27th USENIX Security Symposium, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In NewrIPS, 2019.
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. International Conference on Learning Representations, 2020.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In ACM Asia Conference on Computer and Communications Security, 2021.
Huili Chen, Bita Darvish Rouhani, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar, Deepmarks: A secure fingerprinting framework for digital rights management of deep learning models. In International Conference on Multimedia Retrieval, 2019.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. In International Conference on Learning Representations, ICLR 2021, la.
Xuxi Chen, Tianlong Chen, Zhenyu (Allen) Zhang, and Zhangyang Wang. You are caught stealing my winning lottery ticket! making lottery ticket claim its ownership. In Advances in Neural Information Processing Systems, 2021b.
Yu Chen and Lingfei Wu. Graph Neural Networks: Graph Structure Learning, pp. 297-321. Springer Nature Singapore, Singapore, 2022.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In NeurlPS, 2020.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. CoRR, 2017.
Luca Cosmo, Anees Kazi, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael M. Bronstein. Latent patient network learning for automatic diagnosis. CoRR, abs/2003. 13620, 2020.
Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. Deepsigns: An end-to-end watermarking framework for ownership protection of deep neural networks. In International Conference on Architectural Support for Programming Languages and Operating Systems, 2019.
Lixin Fan, Kam Woh Ng, and Chee Seng Chan. Rethinking deep neural network ownership ver- ification: Embedding passports to defeat ambiguity attacks. In Anmual Conference on Neural Information Processing Systems, 2019.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. 2023.
Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. SLAPS: self-supervision improves structure learning for graph neural networks. In NeurlPS, pp. 22667-22681, 2021.
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. ICML, volume 97 of Proceedings of Machine Learning Research, pp. 1972-1982. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Thomas Gebhart, Udit Saxena, and Paul R. Schrater. A unified paths perspective for pruning at initialization. ArXiv, abs/2101.10552, 2021.
Michael Hahsler, Matthew Piekenbrock, and Derek Doran. dbscan: Fast density-based clustering with R. Journal of Statistical Software, 91(1):1-30, 2019, doi: 1186637/jss.v091..8.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generalization in neural networks. In NeurlPS, 2018.
Hengrui Jia, Christopher A. Choquette- Varun Chandrasekaran, and Nicolas Papernot. En- tangled watermarks as defense against model extraction. In USENIX Security Symposium, 2021.
Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. pp. 66-74. ACM, 2020.
Yongcheng Jing, Yiding Yang. Xinchao Wang, Mingli Song, and Dacheng Tao. Meta-aggregator: Learning aggregate for 1-bit graph neural networks. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 5301-5310, 2021.
Yongcheng Jing, Chongbin Yuan, Li Ju, Yiding Yang. Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern 2023.
Recognition, pp.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin vzidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yingjie Lao, Peng Yang, Weijie Zhao, and Ping Li. Identification for deep neural network: Simply adjusting few weights! IEEE International Conference on Data Engineering, 2022.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS, 2019a.
Jachoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NeurIPS, 2019b.
Suyoung Lee, Wonho Song, Suman Jana, Meeyoung Cha, and Sooel Son. Evaluating the robustness of trigger set-based watermarks embedded in deep neural networks. IEEE Transactions on Dependable and Secure Computing, 2023.
Meng Li, Qi Zhong, Leo Yu Zhang, Yajuan Du, Jun Zhang, and Yong Xiang. Protecting the intellectual property of deep neural networks with watermarking: The frequency domain approach. In International Conference on Trust, Security and Privacy in Computing and Communications, 2020a.
Meng Li, Qi Zhong, Leo Yu Zhang, Yajuan Du Jun Zhang, and Yong Xiang. Protecting the intellectual property of deep neural networks with watermarking: The frequeney domain approach. In International Conference on Trust, Security and Privacy in Computing and Communications, 2020b.
Yiming Li, Linghui Zhu, Xiaojun Jia, Yong Jiang, Shu-Tao Xia, and Xiaochun Cao. Defending against model stealing via verifying embedded external features. In AAAI Conference on Artificial Intelligence, 2022.
Yuanchun Li, Ziqi Zhang. Bingyan Liu, Ziyue Yang, and Yumxin Liu. Modeldiff: testing based DNN similarity comparison for model reuse detection. In ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021.
12

---

Published as a conference paper at ICLR 2025
Yixin Liu, Yu Zheng. Daokun Zhang, Hongxu Chen, Hao Peng, and Shirui Pan. Towards unsupervised deep graph structure learning. In WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022. pp. 1392-1403. ACM, 2022.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 137, 1982.
Xiaoxuan Lou, Shangwei Guo, Jiwei Li, and Tianwei Zhang. Ownership verification dnn archi- tectures via hardware cache side channels. IEEE Transactions Circuits and Systems for Video Technology, 2021.
Sofiane Lounici, Mohamed Njeh, Orhan Ermis, Melek Önen, and Slim Trabelsi. Yes we can: Water- marking machine learning models beyond classification. IEEE Computer Security Foundations Symposium, 2021.
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by conferrable adversarial examples. In International Conference on Learning Representations, 2021.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Ji- Yoon Choi Ji-Hyeok Moon Young-Ilc Mok, Byunggook Na, Ji-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from practical perspective: Can it be trusted for neural architecture search without training? CVPR, 2022.
Yuki Nagai, Yusuke Uchida. Shigeyuki Sakazawa, and Shin ichi Satoh. Digital watermarking for deep neural networks. International Journal of Multimedia Information Retrieval, 2018. 8.
Guillermo Ortiz-Jiménez, Seyed-Mohsen Moosavi-Dezfooll: and Pascal Frossard. What can linearized neural networks actually say about generalization? In NeurlPS, 2021.
Ming Ouyang, William Welsh, and Panos Georgopoulos. Gaussian mixture clustering and imputation of microarray data. Bioinformatics, 20(6):917 (116))917-922 923, 2004.
Xudong Pan, Mi Zhang, Yifan Lu, and Min Yang. TAFA: A task-agnostic fingerprinting algorithm for neural networks. In European Symposium on Research in Computer Security, 2021.
Xudong Pan, Yifan Yan, Mi Zhang, and Min Yang. Metav: A meta-verifier approach to task-agnostic model fingerprinting. In ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2022.
Zirui Peng, Shaofeng Li, Guoxing Chen, Cheng Zhang, Haojin Zhu, and Minhui Xue. Fingerprinting deep neural networks globally via universal adversarial perturbations. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 22500-22510 10. IEEE, 2023.
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. ACM on International Conference on Multimedia Retrieval, 2017.
Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, ICLR 2020, 2020.
Haoxiang Wang, Yite Wang, Ruoyu Sun, and Bo Li. Global convergence of MAML and theory- inspired neural architecture search for few-shot learning. In CVPR, 2022.
Ruijia Wang, Shuai Mou, Xiao Wang, Wanpeng Xiao, Qi Ju, Chuan Shi, and Xing Xie. Graph structure estimation neural networks. The Web Conference, pp. 342-353. ACM IW3C2, 2021.
Si Wang and Chip-Hong Chang. Fingerprinting deep neural networks a deepfool approach. JEEE International Symposium Circuits and Systems, 2021.
13

---

Published as a conference paper at ICLR 2025
Ross Wightman. Pytorch image models. 2019.
Kuo-Lung Wu and Minn-Shen Yang. Mean shift-based clustering. Pattern Recognition, 40(11):
3035-3052, 2007.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmarking machine learning algorithms. 2017.
Chenqi Xie, Ping Yi, Baowen Zhang, and Futai Zou. Deepmark: Embedding watermarks into deep neural network using pruning. IEEE International Conference on Tools with Artificial Intelligence, 2021.
Kan Yang, Run Wang, and Lina Wang. Metafinger: Fingerprinting the deep neural networks with meta-training. In International Joint Conference on Artificial Intelligence, 2022.
Jingwen Ye, Songhua Liu, and Xinchao Wang. Partial network cloning. 2023.
Donghan Yu, Ruohong Zhang, Zhengbao Jiang, Yuexin Wu, and Yiming Yang. Graph-revised convolutional network. Machine Learning and Knowledge Discovery in Databases European Conference, volume 12459 of Lecture Notes in Computer Science, pp. 378-3778 Springer, 2020.
Runpeng Yu and Xinchao Wang. Neural lineage. 2024.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. In International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7154-7163. PMLR, 2019.
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Predicting training time without training. In NeurlPS.
Jingjing Zhao, Qin Hu, Gaoyang Liu, Xiaoqiang Ma, Fei Chen, and Mohammad Mehedi Hassan. Afa: Adversarial fingerprinting authentication for deep neural networks. Computer Communications, 2020.
Xiangyu Zhao, Yinzhe Yao, Hanzhou Wu, and Xinpeng Zhang. Structural watermarking to deep neural networks via network channel pruning. IEEE International Workshop on Information Forensics and Security, 2021.
14

---

Published as a conference paper at ICLR 2025
Neural Phylogeny Supplementary Material
A
Experiments with Stable Diffusion and Llama
16
Collection of Open-Source Models
16
A.2
Collection of DreamBotth-Fietunne Models
16
A.3
Experiments with Open-Sourcee Models in Figure
16
A.4
Performance of the Learning-Free Detection Method with Open-
Source Models
A.5
Experiments with BeitmmBoot--Finttuun Models in Figure
17
B
Other Additional Results and Illustrations
B.
Experimental Results on Pet and DTD
17
B.2
Generalizability of Learning- -Bseed Detector
18
B.3
Diagram Illustrations on the KMeans-along Method
18
Other Implementation Details
18
C.1
Experiments in Table 2 and Table 5
19
C.I.1
Model Set
19
C.1.2
The Choice of Weight
20
C.1.3
Implementation of the Learning-Frig Free Methods
20
C.2
Experiments in Table 3
20
Phylogeny Detector
20
C.4
Experiment in Figure 2
21
Experiment in Figure 2 (Previous Version, Left Here only for Re- viewers Relerence)
D Derivations
22
D.I Derivation of the Proposition
22
D.2 Generalization to Loss with Weight Decay
23
E Further Diseussions
24
E.1 Impacts, Limitations, and Future Explorations
24
B.2 Phylogeny Detection and Attacks
24
15

---

Published as a conference paper at ICLR 2025
A EXPERIMENTS WITH STABLE DIFFUSION AND LLAMA
A.1 COLLECTION OF OPEN-SOURCE MODELS
For the open-source Stable Diffusion and Llama models, we downloaded them directly from Hugging- Face. We selected five parent models in total, diffusion-2 sttablityai/staale-iiff -irbble-dffusion-222
"stabilityailstable- and mmnta-llamaaMetaa Llama-3-8B". Hugging Face provides list of fine-tuned child models for each parent, and we sorted these lists by "trending" and downloaded the top models. For the model "Comp Vis/stable-diffusion v1-4", we attempted to download the top 200 child models. For siaabiltyai/stalleddfffuuoooo has 111 child models, we attempted to download the top 100. For with 152 child models, we attempted download the top 150. As for the LLaMA models "meta- Haama/Llama-2-7777
Wissstaablediffuiionn
which iitiiittaaiiaabldddffssiooooooo
and meeta-llama/Mae Meta--laama-3-83" due to their large sizes (14-16GB) we attempted to download the top 50 child models for each.
We excluded models that could not be downloaded, were mislabeled, or were unusable as standalone models. For example, some 70B models were incorrectly labeled as fine-tuned from 7B models, or certain LoRA-fine-tunee models only released the LoRA parameters, making them incompatible with a unified interface. After these exclusions, we collected 191 child models for "CompVis/stablle Vis/stable- diffusion-vl -l-onn----",,, 68 for for "meta-llama/ Ilama/Llmaaa 2-7h-hf"", and 47 for "meta-llama/Mect--llanna Maa
107 for iaabbiityaistalle-difliiio
48
A.2 COLLECTION OF
MODELS
We use the "diffusers" library to implement the DreamBooth fine-tuning and image generation for diffusion models. The three diffusion models we used are 'Comp www.staalldifffuioon//// -1-44" and Slabbilyaissabll-ddffttoo---1- The fine-tuning dataset is the official DreamBooth release dataset. We fine-tune on all of 30 sets of images, resulting in 60 child diffusion models.
A.3 EXPERIMENTS WITH OPEN-SOURCED MODELS IN FIGURE
In this experiment, we focused on detecting the existence and direction of the fine-tuning relationship between two models. For example, given Stable Diffusion 1.4 and another model f. the detector determines whether a finetuning relationship exists between Stable Diffusion 1.4 and f. and so, the detector further identifies which model is the parent. This is practical scenario, as in some cases, we want to determine whether specific model is fine-tuned from particular open-source model, or we want to detect the fine-tuning relationship between two models without interference from other models. To achieve this, we trained a detector. The network architecture of the detector is consistent with the one described in Fig. 4, but since only two models are involved, we replaced the transformer block with an MLP- prediction head. The prediction head is designed to output three categories: no fine-tuning relationship between the two models, the first model is the parent of the second, and the second model is the parent of the first.
We randomly split the collected child models into training, validation, and test subsets in a 7:1:2 ratio. The models in each subset were paired with the parent models to form the input for the detector. The experiments for Stable Diffusion and LLaMA models were conducted separately. That is, in the Stable Diffusion experiment, possible combinations included (Stable Diffusion 1.4, a child of Stable Diffusion 1.4). (Stable Diffusion 1.4, child of Stable Diffusion 2), (Stable Diffusion 1.4, child of Stable Diffusion 2.1). (a child of Stable Diffusion 1.4, Stable Diffusion 1.4). (a child of Stable Diffusion 2, Stable Diffusion 1.4), (a child of Stable Diffusion 2.1, Stable Diffusion 1.4) and other combinations sharing similar patterns. Other training details are consistent with the experiments in Tab.
We recorded the detector's s accuracy for detecting the existence of the finetuning, accuracy for detecting the direction of the finetuning and the overall accuracy. The reported "Lineage Acc" is defined as the percentage of cases where the existence of finetuning relationship between the input model pair is correctly detected. The reported "Direction Acc" is defined as the percentage of cases where the parent model in detected finetuning relationship is correctly identified. The "Overall Acc" is defined as the percentage of cases where both the existence of a finetuning relationship and the parent model are correctly identified, All reported results are computed from the testing set. The
16

---

Published as a conference paper at ICLR 2025
results are presented in Fig and Tab. 6 each parent model, we also display some of its child models along with the detector's predicted probabilities that they were finetuned from the parent model in Fig 1.
Our method achieves near 100% detection accuracy. For
A.4 PERFORMANCE OF THE LEARNING- FREE DETECTION METHOD WITH OPEN-SOURCE MODELS
We also tested the performance of the learning-free methods on the collected open-source Stable Diffusion and Llama models. The input parameters used were the same in the experiments of the learning-based method, but here the goal was to detect all fine-tuning relationships and directions among the models one pass. The experiments for Stable Diffusion and Llama models were conducted separately. Each experiment was repeated 10 times, and we recorded the average accuracy and the best accuracy across the trials. The specific results are shown in the first two rows of Tab.
In terms of the best achievable performance, the learning-free clustering method, which incorporates the idea of "distinguish along clustering", achieved significantly better predictions compared the original K.Maans and MeanShift methods. However, the learning- feeinng-ree method lacked stability and was highly sensitive to the initialization. Across multiple runs, many failed attempts were observed, leading to low average accuracy and high variance.
A.5 EXPERIMENTS WITH CHEAMBOOTTH-FIETUURR MODELS IN FIGURE
We conduct phylogeny detection experiments using four learning- free strategies. The de- tailed results are shown in the last row of Tab. 5 which are also illustrated in the bottom right part in Figure 1. For the clustering algorithm's input, used the ttransformer__booee Srfocss0.ff..et.22. coder. Our method achieves high accuracy in detecting the fine-tuning relationships and directions among 62 models in one pass.
"down
in the diffusion model's UNet en-
B OTHER ADDITIONAL RESULTS AND ILLUSTRATIONS
B.1 EXPERIMENTAL RESULTS ON PET AND DTD
Table 7:
Phylogeny detection performance for the vision models on the Pet and DTD datasets. The best (the second-best) ones are marked in bold (underlined).
Model
Method
DTD
Pet
DBSCAN
22.93:12.70
46.88+3..74
GMM
22.86++2.63
38.2943.03
Identify after Clustering
KMeans
23.45:3.00
46.8813.75
ResNet
MeanShift
22.74+22.54
46.52+3.66
KMeans
41,691 3.90
47.42+3.74
Identify along Clustering
MeanShift
41.54+1..09
4..36
Phylogeny Detector
50.44:1:.885
4..61
DBSCAN
71.87:14.76
2.08
GMM
59.38
1.25
Identify after Clustering
KMeans
68.7444.669
2.80+1.40
ViT
McanShift
72.52+4.81
2.17+1.09
KMeans 68.8641..30
3.07
Identify along Clustering
MeanShift
72.8744.76
2.91
Phylogeny Detector
82.54+1116
50.79+11007
The experimental results on the Pet and DTD datasets, shown Tab. 7 are consistent with findings other datasets. The ceaning-based method still outperforms the learning methods. Among the
17

---

Published as a conference paper at ICLR 2025
Table Performance of the learning-basse baased detector under across-architecture generalization.
ViiT
Conv Net
Cifar10
SVHN
Cifar10
SVHN
Best Learning-Free Free Method
73.55%
70.61%
46.29%
44.53%
Without Generalization
89.01%
72.01% %
99.17%
99.86%
With Generalization 62.27%
60.67%
83.33%
74.62%
various learning-free clustering methods, it is difficult to determine an absolute best. However, incor- porating parent model identification into the clustering process generally improves the performance of the clustering algorithms.
B.2 GENERALIZABILITY OF LEARNING-BASED DETECTOR
In the main experiments, we controlled the training and testing sets of the learning-based detector to have different child models (Appendix (.3). This implicitly requires the detector to be able to generalize unseen child models. Such experimental setup is based on the perspective of pretrained model owners. They possess parent models and some child models and aim to detect from which pretrained model external child models finetuned.
We further explored the detector's ability to generalize across different architectures. We trained VIT and ConvNet architectures. The results are shown
the detector using FC-Net and tested in Tab. 8 In the table, the row "Best Learning-Free Method" refers to the highest accuracy of learning-fres free methods in the main result. Learning-rree methods do not require training and thus do not face generalization issues. The row labeled "Without Generalization" corresponds to the accuracy when no cross-arts arsssss-cchtttture generalization is required. These models are trained and tested on the same prchitecture-datasee pair. The row "With Generalization" corresponds to the scenario where the detector trained on FC-Net and directly tested on VIT and ConvNet architectures. As shown, the learning-basee -baed detector demonstrates acceptable arostarccite generalizability. Even when generalization to entirely new neural network architectures is required, the detector still achieves acceptable accuracy. Notably, in the ConvNet experiments, the accuracy of the directly generalized detector remains significantly higher than that of learning-fir free methods.
B.3 DIAGRAM ILLUSTRATIONS ON THE KMEANS-ALONG METHOD
In Fig. 5] we illustrate the KMeans-along algorithm. The KMeans-along algorithm follows the same procedure as the original KMeans algorithm during the assignment step. As shown on the left part of Fig. 5 each point is assigned the cluster whose centroid is closest to it based key difference between KMeans KMenns-along and the original K.Maans algorithm lies in the update step, illustrated the right part of Fig. In this step, the centroid of each cluster is updated. When the KMeans-along algorithm reduces to the original K.Maas algorithm, as shown in the upper-right part of Fig. 5 using the geometric center as the centroid for each cluster. However, when < KMcans-alle along updates the centroid by selecting the point most likely to belong to the parent model with probability of a, as shown the lower-right part of Fig.
distance. The
C OTHER IMPLEMENTATION DETAILS
Experiments are conducted on one Nvidia RTX 4090. For the parent and child models used, we followed the method of collecting parent and child models from previous work (Yu & Wang 2024). However, compared to previous work, we collected more models with greater variety of structures and in larger quantities. In this work, consider finetuning all the parameters.
18

---

Published as a conference paper at ICLR 2025
Update Step
with probabiity
Assignment step
Centroki
Updatt: controld .shgg custer mean
mindom
Assign models to cusstrs bised on anly.
probabilty
Update centrold based tnetuning direction
Figure 5: Illustrative diagram for the process of KMeans-along method.
C.I
EXPERIMENTS IN TABLE 2 AND TABLE
C.I.1 MODEL SET
We conducted experiments on four types of neural network architectures: a three-layer fully- canly-co network with ReLU activation; three-layer convolutional network with ReLU activation; ResNet18; and ViT-Tiny.
The hidden layers of the fully-connected network are fixed at the sizes of 1024, 256,, and 128. The convolutional network use convolutional kernels of size 3, followed by two fully-connected layers with latent feature dimensions set to 128. The input for the convolutional network is fixed at 28x28; images not meeting this size are resized accordingly. The output dimensions for both networks are adjusted based on the dataset. For the fully- connecedd and convolutional networks, the parent and child models are trained by ourselves. We first train fully-commn fully-connected and convolutional networks on the MNIST and CIFAR- datasets, creating four parent model categories: fully-connected + MNIST, fully-connected + CIFAR-100, Convolutional + MNIST, Convolutional + CIFAR-100 100. The training process use a learning rate of either 0.01 or 0.001, a batch size of either 256 or 1024, and parameter initialization with either Kaiming Uniform or Kaiming Normal, using eight random seeds. This results in 64 training configurations. The number of training epochs set 50, and Adam optimizer is used. For the MNIST dataset, we only select models with an accuracy greater than 80% as parent models. For the CIFAR- 100 dataset, models with an accuracy greater than 50% are selected as parent models. Finally, we obtain 64 fully-connected models MNIST, 17 fully-connected models on C00AR-100, 64 convolutional models on MNIST, and 44 convolutional models on CIFAR- 100. Starting from these parent models, we fine-tune the child models. Models trained on MNIST fine-tuned on FMNIST and EMNIST, while parent models from CIFAR- 100 are fine-tuned on SVHN, CIFAR-10. and its variants. The fine-tuning learning rate is set to of 0.01, 0.001, 0.0001, with a batch size of either 256 or 1024, using four random seeds. The number fine-tuning epochs set to 30, using the Adam optimizer. We only select models with an accuracy greater than 50% as child models. Finally. the fully-connected network obtain 1202, 1220, 160, 232, 364, 264, and 260 models on FMNIST, EMNIST, CIFAR-1O, SVHN, CIFAR- 10-5sho, 10-AR--0-00shot, and CIFAR- 10- 10-Imbalanced, respectively. The convolutional network obtain 1050, 1 1100. 293, 826, 306, 168, and 448 models on FMNIST, EMNIST, CIFAR- 10, SVHN -IFAR-10-5shor CIFAR- 10-50shot, and CIFAR- I-mbalancdd, respectively. This model set is significantly larger than previous works.
For ResNet and ViT-Tiny models, we directly collect publicly released models from the timm library as parent models. The ResNet models used were "resnet1 8.al __ink".,. "resnetl _22_innk". Fresmetl8.a3_jnmg _k"
"resnet18.ge soanett..gloo_iiii
"resnet18.fb.s. siiisss___seeee__e Los__yyccl0ml 100c ft_inkk", totaling seven models, all trained
"resnet18
and (eanee18.tv_inlle on ImageNet. The ViT-Tiny models used were "vit tiny r_6 p8 224.aggreg in21k ft inlk", "vit tiny I s16 p8_ 22greg in21k", and "vit tiny patch16 224.augreg in21k", totaling three models. We fine-tuned these parent models on all datasets shown in Tables 2 and 5. All input images resized to 224x224. Grayscale images are duplicated three times to convert them into
19

---

Published as a conference paper at ICLR 2025
three-channel images. The network's output dimensions adjusted according to the dataset. The fine-tuning and model selection processes are consistent with those used for the fully-connected and convolutional networks. Finally, we obtain 168 child ResNet18 models on each dataset. we obtain 65, 65, 64, 64, 16, 16, 64, 44, and 60 child ViT-Tiny models on FMNIST, EMNIST, CIFAR-10, SVHN, CIFAR-10- 10-5sht,,
10-A-10-50shot,
LLFAR--0-Ilmallncee Pet, and DTD, respectively.
In our experiments, we combined the child models with their corresponding parent models to form the model for phylogeny detection.
C.1.2 THE CHOICE OF WEIGHT
Using all the neural network parameters would result in significant complexity. In our experiments, select only subset of the neural network parameters for the phylogeny detection task. For fully- connected networks, we use the weight tensor of the first linear layer. For convolutional networks, the
use the convolutional kernels of the first convolutional layer. For ResNet networks, convolutional kernels of the first convolutional layer in the second ResNet block. For ViT, we used the weight tensor of the last linear transformation in the third transformer block.
C.1.3 IMPLEMENTATION OF THE LEARNING-FREE METHODS
We use the DBSCAN and GMM implementations from the sklearn library. The search range for the eps parameter in DBSCAN is 0.5, 0.1, 1.0, 2.0, and 5.0. We implement the KMeans and Mean Shift algorithms ourselves. The search range for the bandwidth parameter in Mean Shift 0.5, 1.0, 2.0, 3.0, and 5.0. For the identify parent along the clustering strategy, the search range for is 0.0, 0.2, 0.4, 0.6, and 0.8. For algorithms that require the number clusters as input, such as K.Means and GMM, provide the actual number of parent models as the number of clusters. The maximum number of iterations for clustering is set to 50, but the algorithms often converge in fewer than 10 iterations.
C.2 EXPERIMENTS IN TABLE 3
This experiment aimed to test the ability to perform phylogeny detection across generations. The method for constructing the model set to be detected followed the same approach as in previous work (Yu & Wang 2024). using ResNet18 as the model architecture. The first generation consist of the original seven ResNet models. The second generation models are obtained by fine-tuning the first-generation models on the EMNIST-Letters dataset, using the child models collected in Table 2 experiments. From the second generation, we select seven models with highest accuracy as the parent models for training the third generation, and ensure that these models have different first-generation parent models. These seven models are fine-tuned the FMNIST dataset obtain the third- thirr-genrration models. From the third-generation models, we again select seven models with the highest accuracy, ensure different first-generation ancestors, and fine-tune them on the EMNIST- Balanced dataset to obtain the -ortth-generation models. The fine-tuning settings and model selection methods are consistent with those in Table 2 experiments. We obtain 168 models each for the third and fourth generations.
For the experiments involving the first and third generations, we combine the original seven ResNet models with the third-generatic models fine-tuned on FMNIST to form the model set to be detected. For the first and fourth generations. we combine the original seven ResNet models with the fourth- generation models fine-tuned on OMNIST-Ballacee third generations, we combine the seven models selected from the second generation fine-tuned on FMNIST with the third third-generation models fine-tuned FMNIST. The experiments between the third and fourth generations and between the second and fourth generations are constructed similarly.
For the experiments between the second and
The implementation of learning-free and learning- based methods and the hyperparameter search in these experiments are consistent with previous experiments.
C.3 PHYLOGENY DETECTOR
Given a set of neural network models, similar to the learning-fre free method, we only extract part of the parameters from each model as the phylogeny detector input. The specific parameter selection
20

---

Published as a conference paper at ICLR 2025
remains consistent with the learning-fre free method. This allows us to obtain features for all nodes in the directed graph, and we need to predict the edges in the graph. Similar to previous methods for node detection and edge detection on single graph, we split the graph into several subgraphs, using a portion for training and the remaining parts for validation and testing.
Specifically, for each subgraph, we randomly select P parents and C child models for each parent, combining P P x C' models to form a subgraph. The values P and C' depend on the total number of parent and child models and the GPU memory limitations. For fully-connected. convolutional, and ResNet networks, P is set to 4, and for ViT models. P is set to 3. C' is always set to 6. The resulting subgraph sizes are 28 or 21. We divide the obtained subgraphs into training, validation, and test sets in 7:1:2 ratio. Since the number of parent models is limited, we use it in the training, validation, and test sets, but we ensure that child models are unique to each set. If child model is included in a training subgraph, it doess not appear in the validation or test subgraphs. If a child model is included in validation subgraph, it does not appear in the training or test subgraphs.
To increase the number of training samples, use a strategy of repeated sampling of the parameters to create more samples. For example, in the experiment with fully-connected networks trained on MNIST and their child models, we use the weight matrix of the first linear layer, which has size of 1024x784. Instead of setting the input size of the phylogeny edge encoder to 2x1024x784, we set it to 2x32x32. Thus, we used only part of the weight matrix, allowing us to sample multiple times from this matrix to create more than one samples. Specifically, once which models are included in a subgraph is determined,
sample 32x32 indices based on the shape of the weight matrix of the first linear layer. For each model included in the subgraph, we pick the elements from the weight matrix according the sampled indices. Different subgraphs have different index sets. Given the models in subgraph, we sample multiple times to create several data samples. We typically repeating the sampling 10 times. In the few-shot experiment with the ViT structure, due to the small number of models, we repeated the sampling 40 times. Even with this, we could not obtain large enough dataset improve the performance of the phylogeny detector in the ViT few-shot experiment.
Except in the architecture ablation experiment, we use 5-layer edge encoder and 1 1-layer trans- former detector in all other experiments. The edge encoder's s kernel size set to 3, and the latent embedding size is set to 32. The training learning rate is 0.01, the batch size is 1, and the epoch is 100 for all experiments. The Adam optimizer used.
C.4 EXPERIMENT IN FIGURE 2
We train eight ResNet18 models from scratch with learning rate of 0.01 or 0.001, batch size of 256 or 1024, using two random seeds. The training dataset CIFAR- 100, and the Adam optimizer is used, with the number of iterations set to 12,000. Subsequently, from each parent model, four child models with a learning rate of 0.01 or 0.001, using two random seeds. The fine-tuning dataset is CIFAR-10. The dark lines in the figure represent the average performance of the eight parent models and thirty-two child models. The lighter lines depict the trajectories of some specific parent and child models.
fine-tune
C.5 EXPERIMENT IN FIGURE 2 (PREVIOUS VERSION, LEFT HERE ONLY FOR REVIEWERS' REFERENCE)
Color Uniform Gesasiun
Distugie
0.6
022
Pto Toitznng Plaze
Tuning
0.0
0
6
12(0) 2
Training Inteaaton (x 163)
Figure 6: During training and fine-tuning, the distance between parameter and the fake initialization continually increases, consistent with theory.
21

---

Published as a conference paper at ICLR 2025
We train eight ResNet18 models from scratch with learning rate of 0.01 0.001, batch size of 256 or 1024, using two random seeds. The training dataset 100, and the Adam optimizer is used, with the number of iterations set 12,000. Subsequently, from each parent model, fine- tune four child models with a learning rate of 0.01 or 0.001, using two random seeds. The fine-tuning dataset is CIFAR-10. The dark lines in the figure represent the average performance of the eight parent models and thirty-two child models. The lighter lines depict the trajectories of some specific parent and child models.
In our theory, the only restriction on the distribution of the fake initialization is that it has zero mean, with no constraints on the variance and the type of the distribution. We tested with two types of distributions and with various variances. including {le - 3, le 2, 1e - 1, 1, 10}. and found that the resulting monotonic increasing trend remains stable. The plotted graphs are consistent with those in Figure 3. Note that, the variance of 1e - 3 is smaller than the variance when using Kaiming initialization, while a variance of 10 is much larger than the scale of the neural network parameters after training. Thus, this confirms that the choice of the distribution for z is, as predicted by theory, highly robust.
D DERIVATIONS
D.I DERIVATION OF THE PROPOSITION
Proposition 2. Let ft denote neural network trained using gradient flow on the dataset D (X, 3) at time t. Let 0, E Rd denote the vectorized parameters of fo. where dg is the number of parameters. Let So denote the neural nelwork at the initialization with initial parameters Og. Let n denote the learning rate. Then, with MSE loss L - 211 ->11 the Euclidean norm O2 can be written as,
1 (I-
+
de
which increases over time
Proof. In the derivation, we use the dynamic of the linearized fe to approximate the dynamic of the original non- non-linear fi. Such linearization has been demonstrated to be an effective approximation of the dynamic of the original f, with high accuracy (Lee et al.) 2019b). To simplify the notation, we keep using the fe denote linearized network. Given the input X, the output of the linearized network defined as (X)) = fo(x) + Vo, Ou). Under the gradient flow, the dynamic of the parameters follows
0, = Vo. V(x)
(5)
Under MSE loss the solution of 0, is
0. 0 Ve. (1f6)))) -)),
(6)
where I is the identity matrix, and e tion. Thus, the & norm of the parameters can be written as
is the tangent kernel at the initializa-
1
10,12
- 1100 - Vo.6("))0 - 78 - (fo(X) - >)
dp
do 1
10o ve. - 10t - (fo(t) 2')""
de
x 10
- V - - (fo(t) 2"
n00
(8)
1
[o. 18l ( J))TT
de
x (1--6=1)0-0111--------
- -J']
2
[o. (fo(())))))))))))))) - -))
de
0 0
(9)
dg
22

---

Published as a conference paper at ICLR 2025
There are three terms in the equation, representing the second component, first-order com- ponent and the zero-order component of the changes in the Euclidean distance introduced by the gradient descent. Because in practice, On is initialized from distribution with mean and small variance, the second term can be ignored. The third term is time-independent and does not affect the monotonicity in our subsequent analysis, it can also be disregarded. We denote the second and third terms as e. Thus the &g distance can be approximated by
1
0 1
- (6-10)0))))))) -
dg
de
x (1 - - - -))] +e.
(10)
To evaluate the evolution of the distance along the time, we evaluate its derivative of as
1 d
2 - - E - (fo(x)) ym
(11)
do dt
de
Because all eigenvalues of ne are between and 1, both net and - net are positive definite. Because (e (T e e = (7 e (e (T - e ( is also positive definite. Thus, the derivative positive, the norm of the parameters increases along the time.
the assumptions that the width of neural network goes to infinity, which is a common assumption used in the work in neural tangent kernel (Jacol al.) 22188- Arora al. 2019) and neural network linearization (Lee el al. 2019h). Practically, for example, the theoretical properties align well with the behaviors of architecture, like 3layer fully-connected network with layer width of 8196 (Lee et al. 20195).
Remark 1 (Remarks the assumptions.). In the theorem,
In the theorem, we use the assumption that the loss is MSE, which is also common assumption used in the work neural tangent kernel (Jacot et al. 2018 Arora al. 2019 and neural network linearization (Lee et al.) (019b). Only with the MSE loss can the analytic solution of the dynamics of neural network training be derived.
D.2 GENERALIZATION TO LOss WITH WEIGHT DECAY
Proposition 3. Let St denote neural network trained using gradient flow on the dataset D = (X' )) at time t. Let 0, E Rd denote the vectorized parameters of ft. where dg is the number of parameters. Let fo denote the neural network at the initialization with initial parameters Og. Let denote the learning rate. Let À denote the small positive balancing parameter for the weight decay term. Then, with MSE loss L 2 211 212 and weight decay L A 100 12, the Euclidean norm Or can be written as,
2
J/111 \o- (fo(2) - +E,
(12)
which increases over time
Proof. Similarily based on the linearization, the dynamic of the parameters follows
0, = V
(13)
Under MSE loss the solution of 0, is
0. 0 0
(o) -0- ((6()) )).
(14)
where I is the identity matrix, and θ & (o. is the tangent kernel at the initializa- tion. Thus, the f2 norm of the parameters can be written
1 + [o
de 10.12
(0t (f6(2))))
de 100
- -))]
2
(110-110((6((8 -2)j0,
de
1
0
(15)
dg
23

---

Published as a conference paper at ICLR 2025
Collecting the high order term into e, the E2 norm can be approximated by
110112 ~ da
((6)))))))0))))))))00--)) - -2)
dg
[Vo.66l33) (-0000000000)))600000000 V)] + (16)
Its derivative of f can be written as
d d
11
2n do
-)) V Ve)
(
de
dt
x |1 - -1-le-ntee 1 1: "O0 y).
(17)
Because À is small the term / je - can be approximated as (0 + A/).. Similarly to the discussion in Appendix D.I] the derivative is positive definite. Thus, the derivative positive, the norm of the parameters increases along the time.
E FURTHER DISCUSSIONS
E.1 IMPACTS, LIMITATIONS, AND FUTURE EXPLORATIONS
Neural phylogeny detection is the first task that identifies fine-tuning pairs and directions directly from collection of models without relying on generational annotations. Identifying the phylogeny of models has significant implications for research and model governance. The information about model relationships weaknesses among models. Practitioners can use detected phylogeny to regulate model usage. Since phylogeny detection only requires parts of the model parameters, extracting the knowledge learned by the models or related information used for training is difficult, thus minimizing the risk of privacy or intellectual property breaches. However, the ability of phylogeny detection uncover fine-tuning relationships between models can be applied to research on model attacks, defenses, and privacy extraction and protection. For example, when designing black-box attack methods, more effective proxy model can be selected from model library using phylogeny detection.
be used study the inheritance of knowledge, and the spread of bias and
As the first attempt to address novel task, our solutions have many areas for further exploration. First, due to the reliance on the natural alignment between parent and child model parameters, our models cannot handle fine-tuning with additional parameters introduced or changes neural network structures. However, this strategy is widely used in the efficient tuning of large models, making highly relevant for practical applications and a promising direction for future research. On the other hand, our learning-fir free method treats phylogeny detection a clustering task, which is highly dependent on data structure and sensitive to hyperparameter design. Although our learning- -free method achieved good results, in practical applications, it is worth exploring how introduce more effective hyperparameter selection mechanisms or approach the phylogeny detection problem from different perspectives to avoid relying on clustering methods.
E.2 PHYLOGENY DETECTION AND ATTACKS
Our method requires access to model parameters and assumes that both the parent and child models are fully accessible. Therefore, our finetuning relationship detection is primarily designed for open- source models, such the publicly available models on HuggingFace. For these models, white-box attacks can be performed directly with high success rate.
However, the proposed Phylogeny Detection does not enhance the white-box attack. While it may enhance black-box box and gray-box attacks by improving the selection of proxy models, these improved attacks will still be less efficient and effeetive than white-box attacks.
On the other hand, the issue of improved proxy model selection leading to more effective black- box and gray-box attacks will only arise if black-box gray-box finetuning relationship detection becomes feasible. We recognize black-box or gray-box finetuning relationship detection as an important direction for future research and will address it in the Future Work and Limitations section.
24


Reviews:
[{"review_id": "eHAPYwhLRT", "reviewer": "Reviewer_fSGg", "summary": "This study introduces the concept of neural phylogeny detection, which is a new task, focusing on identifying the parent-child relationship between neural networks and determining the direction of fine-tuning between them. These two methods have been tested in different model architectures and learning environments, and proved to be effective and reliable in detecting the fine-tuning relationship of transgenerational neural networks.", "questions": "- How robust is the proposed fake initialization metric across different types of neural architectures beyond ResNet, Stable Diffusion, and LLaMA? Has it been tested with a variety of non-fully connected networks or transformers?\n\n- Given that the choice of $\\alpha$ impacts clustering accuracy, could a systematic method be developed for selecting an optimal $\\alpha$ value without relying on empirical tuning?\n\n- How does the proposed framework handle situations where multiple child models are fine-tuned from a single parent model, or when models have multiple parent models (e.g., ensembling scenarios)?\n\n- Would the approach be applicable or require modifications when dealing with pruned or compressed models, which might have distinct parameter distributions from standard fine-tuned models?\n\n- The author used fmnist, emnist, cifar10, svhn as datasets in the experimental part. However, the structure of these datasets is too simple. Can the author prove the method and rationality proposed in this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Novelty of neural relationship detection: this work provides a groundbreaking method for understanding the inheritance of knowledge and connections in the growing neural model network, and solves the key challenge of deep learning traceability.\n\n- Dual methods of flexibility and accuracy: by providing learning free and learning based methods, this research provides the option of giving consideration to easy implementation and high detection accuracy to meet various use cases.\n\n- Robust evaluation: a large number of experiments in various network architectures and learning scenarios have confirmed the universality and robustness of the proposed method, showing the applicability of the framework to different model structures and generational relationships.", "weaknesses": "**Some major comments:**\n\n- The metric relies on a “fake initialization” since the true initialization parameters are unavailable. This introduces a level of approximation that may impact the accuracy, particularly in scenarios where the proxy distribution used for fake initialization does not align closely with the true initialization distribution.\n  \n- Integrating fine-tuning direction into clustering algorithms introduces added complexity, especially with algorithms like KMeans and MeanShift. Adjusting the clustering to identify the parent-child direction requires modifications that could add computational cost and reduce interpretability of traditional clustering processes.\n\n- The effectiveness of the clustering approach depends heavily on the probability parameter $\\alpha$. The need for empirical tuning of $\\alpha$ to achieve optimal results adds another layer of complexity, which may limit the method’s generalizability across different datasets and model types.\n\n- The proposed methods require iterating across high-dimensional parameter spaces and maintaining computational efficiency, which may be challenging with larger networks or model sets, potentially impacting scalability and real-world applicability.\n\n**Some minor comments:**\n\n- Related work should introduce the differences between this paper and previous work and explain the advantages of this research over previous work. It should not only list the work.\n\n- The objective of this study has not been clearly described (in Section 3.1)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "spNOsJsUXj", "reviewer": "Reviewer_WN3r", "summary": "This paper proposes a new problem, neural phylogeny, which aims to predict the fine-tuning relationship of models. Specifically, given a set of neural networks, the authors predict clusters of models such that all models were fine-tuned from a common pre-trained (or parent) model. Furthermore, they predict which model is the parent (pre-trained model) and which are the children (fine-tuned models). The authors first define a method for predicting the parent model in a given (parent, child) pair. Next, they propose a few methods for predicting model clusters and parents within those clusters, including unsupervised clustering methods and a supervised neural network method. They perform experiments on several model architectures and fine-tuned datasets and find that the supervised neural network method is the top performing method.", "questions": "Questions\n1. Can you please clarify how you found the results in Table 2? Specifically, are models trained on the same dataset with different learning rates and/or batch sizes considered to be different parent models? Are the results averaged over the 8 seeds mentioned in the appendix or something else?\n1. In equation 2, why do you select the model that is most frequently identified as the parent instead of just taking the model with the minimum distance to $\\mathbf{z}$ (i.e., $\\arg\\min_{i} dist(\\boldsymbol{\\theta}^{(i)}, \\mathbf{z})$)? \n2. Why did you choose to use a transformer model over other supervised models?\n3. I assume \"Uniform\" and \"Gaussian\" in Figure 2 refer to the random distributions used for $\\mathbf{z}$. On line 256-258, you say that \"this theorem only requires z to have zero mean, which makes the choice of proxy distribution robust when calculating the distance, without the need to precisely know the initial parameter values of the neural network\". Have you validated this empirically with other initialization techniques that are common for neural networks, like Kaiming or Xavier? \n\nSuggestions\n1. To understand Figure 1, we must read the Appendix. I think this is a main contribution of the paper and the experimental details should be clearly stated in the main body. Additionally, the metrics (Overall acc, lineage acc, and direction acc) should be more clearly defined, preferably using math notation.\n2. In Figures 2 and 3 the legends are unclear. I suggest replacing legend titles like \"color\" and \"line style\" with more useful explanations and clarifying method names (\"Along clustering\" and \"After Clustering\") as you define them.\n4. The notation in Figure 4 does not match the notation in the main text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem of neural phylogeny is an interesting concept and may be more practical than the previously defined neural lineage detection because (a) it is data-free (i.e., does not require access to the fine-tuning dataset) and (b) it does not require prior knowledge of whether a model is a model is a parent or child. \n2. The detection model used for Figure 1 and the metric for determining the fine-tuning relationship between two models (proposition I) is novel and the empirical results show these are effective.\n3. The paper considers multiple methods for predicting clusters and parent node, including unsupervised and supervised methods. Additionally, the authors test their methods on several model architectures with varying sizes and types of layers (FCNet, ConvNet, ResNet, ViT, Llama, Diffusion models).\n4. The paper is well-written.", "weaknesses": "1. These methods require that the practitioner has access to all the network weights. Furthermore, the supervised model \"Phylogeny Detector\" requires having access to a training set that includes the weights of the parent model and several children models. These assumptions do not seem practical and the unsupervised performance is poor.\n2. The methods that the authors propose for solving the neural phylogeny problem require that all of the node models have the same model architecture (i.e., if the fine-tuned model has additional layers or parameters, these methods won't work). \n3. The experimental design and results are hard to understand, even after reading the relevant sections in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GDhrr0oJEm", "reviewer": "Reviewer_MurH", "summary": "This paper proposes an interesting concept Neural Phylogeny, which aims to determine the parent-child relationship in a given set of models, which can play a role in model copyright protection in context of fine-tuning. Simply speaking, this article determines the length of the training cycle based on the distance from the model weight to the initial weight, which can be used as evidence to determine the parent-child relationship. Based on this idea, this paper proposes two specific methods and conducts comprehensive tests.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The concept of neural phylogeny detection is a new, well-defined task, advancing beyond neural lineage to focus on detecting both the existence and direction of fine-tuning relationships among neural networks. This novel concept is valuable for understanding model evolution and intellectual property tracking​.\n\n2. The design motivation of the solution is well-founded with theoretical analysis and empirical evidence. \n\n3. Extensive evaluations are conducted to indicate the superiority of the proposed methods in detecting fine-tuning relationships.", "weaknesses": "1. The proposed method is only applicable to relationship traceability in white-box scenarios. However, in practical applications, especially in IP-sensitive environments, only partial model information may be available, and the effectiveness of the proposed methods under such constraints was not evaluated​.\n\n2. Whether different sampling methods of the initial weight will affect the detection results needs to be discussed. \n\n3. I note that the performance of the proposed method varies greatly across architectures and datasets (see Table 2). It is worth exploring the reasons behind this unstable detection and whether this instability is common or unique to the proposed method.\n\n4. The learning-based approach achieved high accuracy on tested models but might overfit specific architectures or datasets used for training. This raises concerns about generalizability, particularly when applying the model to untested or proprietary architectures not represented in the current experiments​.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTdQoS5EMG", "reviewer": "Reviewer_FjhM", "summary": "The paper presents an algorithm to identify a relationship between networks in the form parent-children pair. In contrast to the existing neural lineage detection, the proposed algorithm proposed a new notion termed neural phylogeny where the child and parent model can have the same structure. The paper proposes a learning-free neural phylogeny detection method. The experiments are performed using multiple shallow and deep models.", "questions": "- The primary weakness of the proposed research is the lack of motivation and application of the current work. What is the use of neural phylogeny and where it can be applied?\n\n- Why the fake initialization has been used to compute the distance? Is the proposed metric sensitive to this initialization?\n\n- The paper mentions \"fine-tuned child model undergoes a longer gradient optimization\". Is the phylogeny depend on the number of fine-tuning steps/processes?\n\n- How does the fake random initialization help in finding the fine-tuning direction?\n\n- Comparison with neural lineage algorithms (discussed in section 2.1) must be performed. Along with the graph link prediction algorithm also needs to be added. \n\n- Another primary limitation is the limited success of the proposed algorithm (especially ResNet and ViT) on the low-resolution color images (CIFAR10- 5/50 shots). It raises the question of the success of different deep transformers and CNNs on high-resolution datasets such as CIFAR100, ImageNet, and Tiny ImageNet on different shot settings.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper presents an effective approach for the identification of parent-children relationships among the deep learning models.\n\n+ The paper is easy to read and follow.", "weaknesses": "- The primary weakness of the proposed research is the lack of motivation and application of the current work. What is the use of neural phylogeny and where it can be applied?\n\n- Why the fake initialization has been used to compute the distance? Is the proposed metric sensitive to this initialization?\n\n- The paper mentions \"fine-tuned child model undergoes a longer gradient optimization\". Is the phylogeny depend on the number of fine-tuning steps/processes?\n\n- How does the fake random initialization help in finding the fine-tuning direction?\n\n- Comparison with neural lineage algorithms (discussed in section 2.1) must be performed. Along with the graph link prediction algorithm also needs to be added. \n\n- Another primary limitation is the limited success of the proposed algorithm (especially ResNet and ViT) on the low-resolution color images (CIFAR10- 5/50 shots). It raises the question of the success of different deep transformers and CNNs on high-resolution datasets such as CIFAR100, ImageNet, and Tiny ImageNet on different shot settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)