Model Provenance Testing for Large Language Models
Authors: Ivica Nikolic
Teodora Baluta
Prateek Saxena
Venue: NeurIPS 2025 poster ()
Keywords: model provenance
large language models
Primary area: deep_learning
TL;DR: Develop a test to check if model is obtained by fine tuning of another model

Abstract: Large language models are increasingly customized through fine-tuning and other adaptations, creating challenges in enforcing licensing terms and managing downstream impacts such as protecting intellectual property or identifying vulnerabilities. 
We address this challenge by developing a framework for testing model provenance. 
Our approach is based on the key observation that real-world model derivations preserve significant similarities in model outputs that can be detected through statistical analysis. Using only black-box access to models, we employ multiple hypothesis testing to compare model similarities against a baseline established by unrelated models.
On two comprehensive real-world benchmarks spanning models from 30M to 4B parameters and comprising over 600 models, our tester achieves 90-95% precision and 80-90% recall in identifying derived models.
These results demonstrate the viability of systematic provenance verification in production environments even when only API access is available.

Model Provenance Testing for Large Language Models
Ivica Nikolić
Teodora Baluta
National University of Singapore Singapore inikolic@nus saaikoolc@nuseeuu.sga.... edu.sg
Georgia Institute of Technology Georgia, USA
teobaluta@gatech.ek
Prateck Saxena
National University of Singapore
Singapore
prateeks@comp mus aus. sg
Abstract
Large language models are increasingly customized through fine-tuning and other adaptations, creating challenges in enforcing licensing terms and managing down- stream impacts such as protecting intellectual property or identifying vulnerabil- ities. We address this challenge by developing a framework for testing model provenance. Our approach based on the key observation that real-world model derivations preserve significant similarities in model outputs that can be detected through statistical analysis. Using only black-box access to models, we employ multiple hypothesis testing compare model similarities against baseline es- tablished by unrelated models. On two comprehensive real-world benchmarks spanning models from 30M 4B parameters and comprising over 600 models, our tester achieves 90 95% precision and 80 90% recall in identifying de- rived models. These results demonstrate the viability of systematic provenance verification in production environments even when only API access is available.
Introduction
Platforms such Amazon SageMaker and Hugging Face have enabled wide scale distribution of
ML models, most notably large language models (LLMs) [39, 47, 22]. The increase of publicly available foundation models and datasets, however, has also triggered concerns over unauthorized use of intellectual property (IP) and concerns of compromised datasets [7] or models [20]. These issues are present just in open-source ecosystems, but also for proprietary models that are hidden behind APIs [3, 32). For instance, concerns about model stealing attacks wherein one can extract the model parameters even for production-lever models are on the rise [8. 42, 33]. Similarly, there is growing concern that models may contain backdoors or vulnerabilities, making them susceptible to jailbreaking [20, 2, 59]. Despite best efforts to create safe environment for the development of foundation models, there have already been instances of reported misuse [38, 16. 41].
This landscape highlights the growing need for model provenance testing. The problem of model provenance testing is as follows: Determine whether target model has been derived from a foun- dational model by customizations such fine-tuning. This problem has applications in tracking reuse models not just in open marketplaces but also across product teams in large organizations. When a security or privacy audit finds problem with foundational model, it becomes important to identify which other models in use by the organization may be derived from the problematic one and take appropriate remedial actions (e.g. revoke, retrain, fortify) to mitigate the risk of further
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

non-compliant use. Model provenance tracking is often useful after the fine-tuned model has been deployed, and when authentic ground truth unavailable or unreliable.
One challenging aspect of designing a model provenance tester is achieving high accuracy. There is cost associated with a provenance verdict. For instance, as result of provenance tracking, a company may initiate legal action or investigation. For use cases within the same organization, developers might have to revoke the use of an existing model and even retrain from a clean parent model. False positives, i.e., the deployed LLM is wrongly flagged as a derivation of a problematic LLM, thus entail a downstream cost. At the same time, false negatives, i.e., not being able to flag that the LLM is customized from a problematic parent, also increase the risk of non-compliance. Therefore, we want a principled way to decide provenance and make accuracy trade-offs.
Another challenge is that a practical provenance tool needs to have minimal assumptions to be readily usable in many post-deployment settings. We focus techniques that do not change typical training and data pipelines, and can be integrated for current state-of-the-art LLMs. The tester is expected to only have black-box query access to the models and has no additional information, such as the training dataset, test set, or the algorithm used for training. We are not aware of any prior work addressing the question of model provenance testing systematically and in such practical setups.
Contributions. In this paper, we design the first practical model provenance tester for LLMs that requires only query access. Our proposed techniques stem from a key empirical observation: The output distribution of fine-tuned models is often close to that of their parent model. This distance between a model and its true parent is typically smaller than that between the model and other unre- lated models, making it possible to reliably trace back a derived model to the original parent. In order to keep assumptions to a minimum, we propose to employ the principled framework of statistical hypothesis testing. Specifically, we use black-box sampling and estimation to determine whether the distribution captured by the given model is close to that of the candidate parent. Such estimation can provide formal statistical significance measures, which can be used to check for the null hypothesis, ie., the customized LLM is not close to the given parent model. Our approach is agnostic to the chosen metric for closeness and returns with the guarantee that the false positive rate is less than or equal to a user-specified threshold. We conduct an extensive empirical evaluation across two com- prehensive benchmarks comprising over 600 models from Hugging Face, ranging from 30M to 4B model parameters and across diverse domains. Our tester achieves 90-95% precision and 80-90% recall in detecting model provenance, even with limited number of queries.
2 Model Provenance Testing
Pretraining LLMs involves significant investment, requiring substantial computational resources costing millions of dollars in infrastructure and thousands of GPU hours. When Company A re- leases pretrained LLM denoted as f. it employs specific licensing terms crucial for protecting this investment, maintaining competitive advantage, and controlling the model's usage [32, 3). Startup B might download f. perform only fine-tuning other light customization (mixture-of-experts, prompt engineering), but claim to have pretrained their model from scratch, thereby circumvent- ing licensing requirements and mistepresenting company A's work. In such cases, want to be able to determine if g is derived through fine-tuning of f and resolve the model provenance problem.
We consider model provenance testing framework with minimal assumptions only, where the tester can query the models f.g on arbitrary prompts (c.g., through an API) and get responses. The tester has no access to the training datasets, embeddings, hyperparameters used by either company, or any information about potential modifications performed by Startup B. This mirrors real-world conditions where companies do not always disclose their training procedures, data sources, mod- ification techniques, making the provenance testing problem both practical and challenging.
Our proposed methods are evaluated for non-adaptive adversary, i.e., the Startup B is not aware of the strategy of the model provenance tester. On the other hand, it not clear if more advanced techniques for evasion of provenance detection deployed by adaptive adversuries can provide real benefit due to the randomness used in our tester (sampling from infinite set of prompts) and the fact that adversarial fine-tuning may come at the cost of reduced model performance on intended task.

---

2.1 Related Work
The problem of testing provenance is similar to designing schemes to prove ownership of a model. We provide more details on related work in Appendix and summarize main approaches here.
Fingerprinting. Fingerprinting aims create an identifier of the model for different downstream purposes such as IP protection. Most prior work require white-box access or additional knowl- edge such as intermediate LLM modules [55], access to training and testing datasets and changes to the fine-tuning [49]. One type of approaches detect copies or tampering by hashing the model weights [9, 48]. Other works aim to create fingerprints by estimating the decision boundary via adversarial examples [27, 50, 37]. However, adversarial-bai -baedd approaches can still have high false positive rate, i.e., two unrelated models have the same fingerprinting samples [6]. Follow-up work generates universal perturbations that can be added to samples to match the target model [27]. This can be prone to adversarial evasion [19]. Thus, recent work constructs a set of adversarial examples that are connected with each other and that characterize the decision boundary more robustly [50]. In addition to model fingerprinting via adversarial perturbations, there are works that focus on measur- ing the model similarity [19, 17, 11, 23, 14]. For example, work selects samples with different prediction results across two groups of reference models [17]. This approach utilizes their pair- wise correlation to identify target stolen image classification models. Other approaches also rely on the internal representations to compare models, without focusing on provenance. For instance, one work locally approximates the models under test with linear models and then computes the cosine distance between the weights of the linear model [23]. One recent work in black-box access requires specialized prompting and an additional model's embeddings identify models [34]. We highlight that these approaches do not focus on the question of model provenance as fingerprints may not be robust fine-tuning and other customizations. Our tester also has minimal assumptions of black- box access to only next-token predictions. While our work considers this measure of similarity, our approach returns with a bounded false positive rate, which is metric-agnostic.
Watermarking. Other approaches prove ownership via embedding watermarks in the model weights [10, 45, 43], hidden-layer activations [13] or other model features [26, 12]. Another type of watermarks are achieved by learning triggers or backdoors that produce predefined out- puts [24, 40, 52, 36, 6, 56]. These rely on implementing changes to the white-box model, fine-tuming or training the model on specialized dataset and can affect the model's performance. Specialized model ownership schemes have also been proposed for other models such as graph neural net- works [58, 44). Most existing approaches do not have provable guarantees that model ownership can be verified with given confidence, the verification is often empirically determined. It is par- ticularly challenging to preserve the watermark detection after applying model customizations [25].
The focus of prior work has not been on designing tests for determining provenance under model customizations and do not work under the same minimal assumptions. For instance, recent work proposed a detection framework for fine-tuning and pruming that still requires white-box access the victim model generate test cases and it does not consider LLMs [11]. Moreover, our benchmarks are much more extensive with hundreds of models under diverse customizations.
3 Approach
Our approach to testing model provenance is based on key observation: fine-tuning and other model derivation techniques typically result in only limited changes to the original model, as they primarily adapt the model for new tasks. After fine-tuning, the derived model g may remain similar to its parent model f, as the process focuses on refining specific capabilities rather than creating fundamental changes to the model distribution.
3.1 Model Provenance Tester
Since the tester has only query access to the models, the only data it can collect from provid- ing inputs (called prompts) and analyzing the corresponding output tokens. Furthermore, due to our minimal assumptions and lack of information about the training datasets, the tester queries the models on randomly chosen short sequences, further denoted as a prompt space Ω.
3

---

The tester independently samples from Q a set of T prompts .1...., 2r, queries each model on the same set of prompts, and then compares their oulput tokens pairwise. For each prompt, we compare only the first output token generated by each model; however, n-grams could also be considered. The similarity between two models f and g then calculated as the proportion j of prompts on which the models produce the same output token: μ = 1
T Σ}= 1(f(z)) = g(z)))
However, the ratio μ. by itself may not be sufficient determine whether f and g are similar, since even two unrelated models might agree on some proportion of
Prompts
Models
Simiartios
Hypothesis
Cutcome
outputs by chance. To assess the significance of j4, we introduce set of control models C {ci..... Cm for the model if and compare their similarities to g- These control models are sampled from the set of available LLMs with distinct architectures or train- ing data, to ensure diversity. Our aim is to compare μ to the similarity ratios μui between each control model c, and g. The number m of control models and their sampling strategy depend on availability; generally, the larger the set of control models, the better the es- timation of similarity among unrelated models.
Figure 1: Our model provenance tester that decides if model g is derived from model f.
The final step of the tester is to verify that the similarity ratio j between f and g exceeds all similarity ratios 14: However, we want to ensure this is not merely due to random chance, but rather reflects a true difference. To establish such theoretical guarantees, we employ multiple hypothesis testing. More precisely, for each control model G. we formulate the following hypothesis test H":
H 1i:
/ S Ps:
P > Pi:
where Hj is the null hypothesis that the similarity between f and g is less than or equal to the similarity between e, and g. and Hi is the alternative hypothesis that the similarity between I and is greater. To test each of the hypothesis H', we employ a z-test, which is standard statistical test well-suited for comparing proportions like our similarity ratios when working with large samples. The z-test produces a p-value, which represents the probability of observing such a difference in proportions there were truly no difference between the models (i.e., if the null hypothesis were true). A small p-value (typically less than predetermined significance level a:) indicates that the similarity μ between f and g is indeed significantly larger than the similarity μ4 between g and q,
When conducting multiple hypothesis tests simultaneously. we want to maintain this same overall risk level of a, regardless of how many tests we perform. However, running multiple tests increases our chances of obtaining at least one false positive across all tests, known as the family-wise error rate (FWER). To control this cumulative risk, we employ the Holm-Bonferroni method [18), which adjusts the significance thresholds ag for individual tests //& to ensure the overall false positive rate remains at or below our desired level of a. We illustrate this basic tester idea in Figure 1.
The provenance tester described thus far assumes we have one candidate parent f that we want to test our child model g against. We now extend to the general case where the goal is to determine whether model y is derived from some model from the set Ji,..., of candidate parent mod- els. While running the basic tester a times (once for each provenance pair (f(gg) would solve the unspecified parent problem, it would also would require additional correction for multiple testing to maintain the same level of confidence. The probability of false positives would grow with the number of candidate parents unless appropriate adjustments are made to the significance level.
We thus propose the improved tester in Algorithm Our tester avoids this issue by conducting a single set of hypothesis tests after identifying the most similar candidate. First, it finds the most similar model to the given model among all the control models C and candidate parents F. If that model is a control model, the algorithm terminates with False. Otherwise, it goes on to test whether the FWER of this model is overall below a, the desired significance level. When this test procedure returns False, it indicates that we could not establish this higher similarity with the desired level of statistical significance. This may occur either because there genuinely no significant similarity indicative of provenance or because the test lacked sufficient power under the given parameters (e.g.,

---

Algorithm Provenance Tester for Given a Candidate Parent Set
Require: Model 9, candidate set F (fi...,yfa) Se), of control models C' {cl, Cm.
prompt space f2 number of prompts T,, significance parameter a, statistical test ZTest.
iid 21..... 27 ~
B Sample T prompts
for i + to do
μ4 < 7 د== 1(1(x))) - g(x,))
Calc sim of candidates
end for
for i <- to m do
11 - Σ}= ((0((1z)) = g(z,))
P Calc sim of controls
end for M (MI,..., Me} (14 )
Set of all sims D Find highest sim
umaa + M)x(M)
if Umaz $ (MI...., Ho} then return FALSE parent
Highest not from F, but from C, cannot be
end if
for μ' E .M' ) {Hmaa) | do
Pı ZTest(HHmar /l T)
Compare against other sims
end for
(P(I).... ................................ P(s+m-1) 11) Sort . Ps4m-1)
for f 1 to 8 + m do
OA a (8 + 1--m-h k)
Monf-rooferroni adjustment
if P(k) > dg then return FALSE
end if
end for
return (TRUE, arg max (1)
> Return parent
sample size number prompts) to detect it. When the algorithm return True. has the guarantee that the most similar model is one of the candidate models and that the total significance level across all hypotheses meets the threshold a.
3.2 Understanding Sources of Error
The overall error rates of our tester depend on the combination of errors from both our statistical hypothesis testing and the two core assumptions which we describe below.
Assumption 1: Derivation implies similarity. We assume that when model g is derived from f. they will exhibit above-average similarity in their outputs. This assumption leads to two potential types of errors: false negatives and false positives. False negatives occur when derived model shows insufficient similarity to its parent. This can happen when a model customizer applies exten- sive modifications that significantly alter the model's behavior. While resource constraints typically prevent such extreme modifications (as they would approach the cost of training from scratch), some legitimate derivation relationships may still go undetected. False positives arise when independently developed models exhibit high similarity. This typically happens when models are trained on similar datasets or designed for similar specialized tasks for instance, two independent medical diagnosis models may produce very similar outputs due to their shared domain constraints.
Assumption 2: Control models establish a valid baseline. We assume our control models provide reliable baseline for the similarity we should expect between unrelated models. Similarly, poor selection of control models can lead to two types errors. False positives occur when our control models are too dissimilar from the domain of f. For example, using general language models as controls for specialized code generation models sets an artificially low baseline, making normal domain-specific similarities appear significant. False negatives happen when control models themselves derived from ff or trained on very similar data. This establishes an artificially high baseline that masks genuine derivation relationships.
While we can provide theoretical guarantees controlling error rates in hypothesis testing, we cannot derive analytical bounds for enrors arising from the assumptions about derivation implying
5

---

similarity or the validity of the control model baseline. The assumption coaumptioo--aeedd based error rates can only be evaluated empirically. We find these hold in practice through extensive experiments (Section 4).
3.3 Reducing Query Complexity
Most of LLMs available currently allow cheap (even free) API access, thus the monetary query cost of running our testers is insignificant. However, sometimes this is not the case, for instance when the cost of queries is high (e.g. one query to OpenAl model O1 can cost more than $1 [31]), when the models have some rate restrictions. Furthermore, there are use cases when query complexity can be reduced without any side effects, thus it makes sense from optimization perspective.
We can divide the queries used in the tester into two distinctive types: online queries made to the tested child model g. and offline queries made to the parent model (or models fi...., and to the control models Cm We make this distinction for two reasons. First, often offline queries much cheaper,
the potential parent models (and the control models we will see in the Section 4) are well established, and available from multiple sources, thus they are usually cheaper or free. Second, in some cases, we can reuse the offline queries to perform many provenance tests of different 9i- We analyze separately these two scenarios.
Reducing Online Complexity. Since our tester is fundamentally based on statistical hypothesis testing, any reduction in query complexity must be compensated by increasing the statistical power of individual queries. Rather than querying model g with T random prompts, select a smaller set of T" < T prompts that yield comparable statistical power for detecting model provenance! We achieve this through an informed sampling approach: instead of uniform sampling from f2, we employ rejection sampling with an entropy-based selection criterion. Specifically, to generate each prompt in T", we sample i candidate prompts from P and select the one that max- imizes the entropy of output tokens across all parent and control models. The selection criterion dynamically weighted favor prompts that have stronger discriminative power between similar models. While this approach introduces dependencies between the sampled prompts (so the theo- retical guarantees of hypothesis testing used in Algorithm I do not carry over), our empirical results in Section 4.4 demonstrate its practical effectiveness. We detail our approach in Appendix H.
strategically
Reducing Offiine Complexity. We further consider the case of reducing offfine complexity in settings where offline queries cannot be reused. The key observation for reducing offline query complexity is that may not need equal number of queries to all parent/control models to identify the most similar one. If particular parent model shows consistently higher similarity to g compared to other models, we might be able confirm it as the top candidate with fewer queries to the clearly dissimilar models. The challenge lies in determining when we have sufficient statistical evidence to conclude that one model is significantly more similar than the others, while maintaining our desired confidence levels. This observation naturally leads us to formulate the problem as a Best Arm Identification (BAI) [5] problem in the Multi-Armed Bandit (MAB) setting. In this formulation, each parent or control model represents an "arm" of the bandit, and querying a model with a prompt corresponds to "pulling" that arm. The "reward" for each pull is the binary outcome indicating whether the model's 's output matches that of the tested model g. The goal is to identify the arm (model) with the highest expected reward (similarity to g) while minimizing the total number of pulls (queries). We implement an optimization for our tester based on sfate-of-the- art BAI approaches (see Appendix for details) and show it impact in Section 4.4.
4 Evaluation
We evaluate our proposed provenance testing approach experimentally and seek to answer the fol- lowing research questions:
(RQ1) How accurate is our provenance tester in practice and how does the number of prompts affect its performance?
(RQ2) To what extent do derived models maintain similarity to their parents?
Itt means in Algorithm 1, instead of random sampling ,i,..., 2r ~ St the goal is to find set Zi, Zr from Ti, .1....,27 2r and F, C.

---

(RQ3) How does the size and selection of control models impact the tester?
(RQ4) How effective are the query reduction approaches?
4.1 Models and Provenance Testing Parameters
We collect model candidates for all provenance pairs from the Hugging Face (HF) platform [21]. To avoid selection bias, used download counts selection criterion, taking the most popular models subject only to hardware constraints on model size,
To increase variety of candidates, create two distinct benchmarks BENCII-A and BENCH--,, that differ in aspects such as model sizes, choice of pre-trained models, and ground-truth verification procedure (refer to Tbl. 1). The full procedure of collection of models and constructions of bench- marks is described in Appendix C. We use the standard significance a = 0.05 (see Appendix for other values). Sampling of prompts is given in Appendix D.
The implementation of the tester along with the two benchmarks can be found at https://githhtl htt.iiiiiiillllioccccooo model wwtppppoonneeeeettteeee testiig..
Selection of control set. In all of our provenance tests, use the complete set of available pre-trained models from the benchmark as control models 10 models for BENCH-A and 57 for BENCH-B. This se- lection was done to demonstrates that effective con- trol sets can be constructed without careful manual curation or domain-specific analysis. Manual cu- ration would not have been feasible since two benchmarks have over 600 candidates. Specifically, make effort to align control models with par- ticular parent models' domains or capabilities, We neither analyze the outputs of parent models attempt to match control models to specific use cases. Instead, we simply include all pre-trained models that rank among the most popular on the Hugging Face platform. This sampling approach, while simple, helps avoid introducing obvious selection bias while ensuring broad coverage of model types and capabilities. Due the large benchmark size, we find diverse examples of candidates for domains such as financial, medical and more (see Appendix C for details). We have not evaluated other selection strategies, which remains direction for future work. This straightforward selection strategy, however, allows us to evaluate whether provenance testing can be effective even without carefully chosen control sets.
Table 1:
Comparison of BENCII- to BENCH-B on different features.
Feature pre-trained models
BENCH-A BENCH-B
10
57
derived models total models model parameters
100
383
100
531
IB-4B
<
IB
compilation method
manual higher
automatic
ground-truth verification
lower
4.2 Accuracy of Model Provenance Tester
We evaluate the accuracy of the provenance tester by examining its performance on both BENCI--A and BENCH-B B under different numbers of prompts. Figure shows the precision and recall results from these experiments.
The precision is notably high (approximately 0.95) when the tester uses up 1, 000, and it is sig- nificantly higher than the baseline precision of randomly guessing the parent which is 104I P 0.09 for BENCH-A A and 5741 x 0.02 for BENCH-B. Interestingly, however, the precision reduces as the number of prompts (test samples) increases. This is in direct contrast common hypothesis testing, where larger sample size leads to smaller standard errors, thus higher precision. We get different results because our model provenance tester relies detecting similarities of models. When using a smaller number of prompts, it can detect only the stronger similarities which are usually due to model provenance. However, necessarily have provenance relation. This leads to misclassification and reduced precision.
increase the prompts, it starts detecting similar models that not
The recall behavior shows an opposite trend it improves with larger number of prompts, even- tually reaching 80% 90% depending on the benchmark. This follows expected behavior: more prompts increase the statistical power of our hypothesis tests, enabling detection of small but signif- icant differences in similarities. This increased sensitivity leads higher recall rates, as the tester can detect more subtle provenance relationships that might be missed with fewer prompts.
7

---

We also examine the impact the randomness of prompt sampling on the tester's accuracy. We con- duct experiments on both benchmarks using five different randomly sampled sets of 1, 000 prompts, with the same set of prompts used in all testers, and record the precision and recall for each run see Table 6 of Appendix E. The results show that these values vary by 1 4% between runs, indicating consistent performance across different prompt samples.
(RQI): Our model provenance tester demonstrates high accuracy across different benchmarks, achieving precision of 90% 95% and recall 80% 1566-9000 90% with 3, 000 prompts per model. Simply increasing the number of prompts does not guarantee uniformly better results, reflecting a funda- mental trade-off: gains in recall might be accompanied by losses in precision.
4.3 Correctness of Assumptions
As discussed in Section 3.2, our approach relies on two key as- sumptions. While the high accuracy demonstrated in the previous section indirectly validates these assumptions, we provide here a detailed experimental analysis of both.
1.0
0.6
Precision
Our first assumption posits that derived models maintain significant similarity to their parent models. To evaluate this, we analyzed the similarity rankings across all provenance tests using 3, 000 prompts. For each derived model, we examined where its true parent ranked among all models in terms of similarity ratio Jr. The results strongly support this: in BENCH-A, the true parent had the highest simi- larity ratio in 93% of cases, while in BENCH-B B this occurred in 89% of cases. When considering whether parents ranked in the top 50th percentile by similarity, these percentages increased to 98% and 96% respectively. Thus we can conclude that our experiments indicate that derived models do indeed maintain strong similarity patterns with their parent models. Inadvertently, we have shown as well that with 3, 000 prompts the tester almost approaches the statistical limit (only the model with highest similarity ratio can be identified as parent), the recalls are very close to the per- centages of highest similarity (89% recall vs. 93% highest parent similarity, and 82% recall vs. 89% similarity, for the two benchmarks, respectively).
0.2
Recall
10
Control set size
1.0
0.6
0.4
Precision
0.2
Recall
0
10
20
30
40 50
Control se: size
Figure 2: Precision and call of BENCH-B (left) and BENCH-A (right) with spect to smaller control set size.
(RQ2): The assumption of similarity between derived and parent models is valid for most prove- nance pairs.
Our second assumption concerns whether control models can effectively establish a baseline for similarity between unrelated models. We stress that in our experiments we have chosen the control models to be simply the set of all pre-trained models in an unbiased way, without any special selection or optimization for particular parent models they are tested against. We empirically observe that such unbiased selection of control model establishes a good baseline similarity as evident from the accuracy results presented thus far.
1.0
0.6
0.4
Precision
Recall
0.0
1000
2000
3000
4000
5000
Prompss
1.0
0.8
We further examine how the size and quality of the set of control models might affect tester accuracy. We conducted experiments varying the size of the control set while keeping other parame- ters constant (3, 000 prompts per test). We randomly sampled different-sized subsets from our full control sets (10 models for BENCH-A and 57 for BENCH-B) B) and ran 100 complete bench- mark tests for each size, and averaged the outcomes. The results, shown in Figure 2, reveal that both size and quality of the con- trol set significantly impact tester performance. For BENCH-A, even with just control models, the tester achieved 55% pre- cision. This relatively good performance with few controls can
0.4
Precision Recall
0.2
0.0
1000
3000
5000
Prompts
Figure 3: Precision and re- call of the model provenance tester with different number of prompts on BENCH-A (top) and BENCH-B (bottom).
8

---

1.0
Promots kw64
Prompts kw64
200
400
600
800
1000
200
400
600
800
1000
0.8
1.0
k-64
0.8
0.4
k-16
0.6
Recall k=1
0.6
Recall k=1
k-4
0.4
Precision kw64
0.4
Precision k-64
Recall km64
0.2
Recall km64
0.0
0
200
400
800
B00
1000
0.0g
1000
2000
3000
4000
5000
0.0
1000
2000
3000
4000
Prompts
Promots k=1
Prompts k=1
Figure 4: Recall for BENCI-BB with different values of advanced prompt sampling (k).
Figure 5: Precision/recall for BENCII-B (left) and BENCI-B (right) when advanced online prompt sampling with k = 64 uses four times less prompts than advanced sampling (k: 1).
be attributed to BENCH-A consisting entirely of seneral-purposse well-trained LLMs thus any subset of these models provides reasonable baseline for measuring similarity between unrelated models. However, for BENCH-B, the randomly sampled 4-model con- trol set yielded less than 10% precision. This poor performance stems from BENCH-B containing a much more diverse set of models, including domain--peciffi models (e.g., for medical or coding tasks) and smaller models with varying capabilities. With such diversity, small random subset of control models is unlikely to establish good baselines for all test cases for instance, when testing a foding-focused model, we need coding-related models in the control set to establish proper base- lines Performance improves steadily as control set size increases in both benchmarks, since larger control sets are more likely to include appropriate baseline models for each test case.
(RQ3): The tester's performance degrades when the control set is too small or poorly selected.
4.4 Reducing Query Complexity
To reduce the online complexity. we implement an advanced rejection prompt sampling strategy as detailed in Section 3.3. We evaluate this strategy using different parameter values k = 4, 16, and 64 (recall, 1 defines how many random samples are used produce one selected sample), comparing it to the standard provenance testing without rejection (k- 1).
Figure compares the tester's recall across different values of k. Notable improvements are visible even = 4, with higher values of k showing better re- sults (though with diminishing returns). Specifically, the recall achieved with 1, 000 prompts at k can be matched using only about 250 prompts at k 64, representing four-fold reduction in on- line complexity. Figure 5 provides comprehensive comparison between
Table 2: Precision and recall of the base vs BAT tester on BENCH-A and BENCH-B.
Allowed
Benchmark
Tester
Avg Queries
Precision
Recall
Queries T
500
BENCH-A A
base
500
1.00
0.81
500
BENCH-A
BAT
450
0.98
0.29
500
BENCH B
buse
500
0.95
0.56
500
BENCH-R
452
0.98
0.29
1 and k 64 for both precision and recall across both benchmarks, using 5 5 times fewer queries for = 64 (note, in Fig- ure 5 the number of prompts for A 64 are given at the top of the plots). The results demonstrate that the tester maintains its effectiveness despite the signifi- cant reduction in queries to the tested models. For example, ad vanceed prompt sampling achieves high levels of 90 95% precision and 80 - 90% recall while reducing the required number of prompts from 3, 000 just 500 per model.
1,00
BENCH-A
1,000
0.99
0.86
L,00
BENCH-A
BAI
605
1.00
0.63
1,000
BENCH-B
base
1,000
0.94
0.68
1,000
BENCH-B -B
BAI
809
0.98
0,42
2,000
BENCH-A
base
2,000
0.98
0.89
2,000
BENCH-A
1,482
0.97
0.54
2,000
BENCH-B B
base
2,000
0.92
0.77
2,000
BENCH- B
BAI
1,482
0.97
0.54
We next evaluate strategies for reducing offline complexity, which refers the number of queries made to pre-trained models during testing. We implement this reduction using BAI, described in Section 3.3 and given in Algorithm 4. We test this approach on both benchmarks by setting target budget of T queries (prompts) per pre-trained model. For example, with T = 1000 on BENCIIA.
Note that in practice, unlike our random sampling experiments, one can deliberately select control models matching the domain and capabilities of the suspected parent model, thus reducing significantly the impact of size of control sets, and leaving quality of the control set as the main factor efficiency of the tester.

---

which contains 10 pre-trained models, the BAI-based provenance tester has maximum budget of 10- 000 10, 000 total queries to make its decision.
Table 2 compares the performance of the base tester and the BAI-enhanced version across different query budgets T (500, 1000, 2000). The results show that the BAI tester successfully reduces offline complexity by 10% 30% (as shown in the "avg queries" column). However, this reduction comes significant cost recall, while precision remains largely unchanged. For instance, with T' = 1. 000 on BENCHI-A, A, BAI reduces the average number of queries from 1, 000 to 605, but recall drops from 0.86 to 0.63. Similarly, on BENCH-B, the average queries decrease from 1, 000 to 809, but recall falls from 0.68 to 0.42. This pattern persists across different values of ' and both benchmarks, suggesting that the trade-off between query reduction and recall preservation is not favorable in most cases.
(RQ4): The online query optimization strategy leads to 4-6 fold query reduction without accuracy drop, whereas the offline approach performs only marginally better and has a negative impact on recall.
Conclusion
Our work formulates the model provenance testing problem for LLMSs which has many applications such as in detection of misuse of terms of use or vulnerable customized models. We present an ap- proach based on statistical testing with minimal assumptions that has high accuracy for real-world benchmarks. Our key insight is that models derived through standard customization approaches maintain a level of similarity to their parent model that is statistically distinguishable from unrelated parents. We evaluate this observation empirically, together with our approach and several optimiza- tions. We find our proposed method to be practical for deciding LLM provenance.
Acknowledgments
We thank Xu Louise, Bo Wang, Jason Zhijingcheng Yu, and Mallika Prabhakar for their valuable feedback on the draft. This research supported by Singapore Ministry of Education (MOE) Tier 2 grant Science Foundation under Grant No. 2531010. Any opinions, findings, and conclusions or rec- ommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
24-0007. This material is based upon work supported by the U.S. National
References
[1] Sacif Alhazbi, Ahmed Mohamed Hussain, Gabriele Oligeri, and Panos Papadimitratos. Llms have rhythm: Fingerprinting large language models using inter-token times and network traffic analysis. arXiv preprint arXiv:2502.22 20589, 2025.
[2] Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton. Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Pas- sos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Anonymous. Invisible traces: Using hybrid fingerprinting to identify underlying LLMs in genAl apps, 2025.
[5] Jean-Yves Audibert and Sébastien Bubeck. Best arm identification in multi-armed bandits. In Conference on Learning Theory (COLTT), pages 13-p. 2010.
[6] Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. IPGuard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In Proceedings of the 2021 ACM Asia Conference on Computer and Communications Security (AsiaCCS), pages 14-25, 2021.
10

---

[7] Nicholas Carlini, Matthew Jagielski, Christopher A Choquette- Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web- scale training datasets is practical. In 2024 IEEE Symposium on Security and Privacy (SP), pages 407-425, 2024.
[8] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of production language model. arXiv preprint (xXiv:2433.06634, 2024.
[9] Haozhe Chen, Hang Zhou, Jie Zhang, Dongdong Chen, Weiming Zhang, Kejiang Chen, Gang Hua, and Nenghai Yu. Perceptual hashing of deep convolutional neural networks for model copy detection, ACM Transactions on Multimedia Computing, Communications and Applica- tions, 19,3):1-20, 2023.
[10] Huili Chen, Bita Darvish Rouhani, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deep- marks: A secure fingerprinting framework for digital rights management of deep learning models. In Proceedings of the 2019 on International Conference on Multimedia Retrieval, pages 105-113, 2019.
[11] Jialuo Chen, Jingyi Wang, Tinglan Peng, Youcheng Sun, Peng Cheng, Shouling Ji, Xingjun Ma, Bo Li, and Dawn Song. Copy, right? testing framework for copyright protection of deep learning models. In 2022 IEEE symposium on security and privacy (SP), pages 824 841, 2022.
[12] Xuxi Chen, Tianlong Chen, Zhenyu Zhang, and Zhangyang Wang. You are caught stealing my winning lottery ticket! making a lottery ticket claim its ownership. Advances in neural information processing systems, 34:1780- 1791, 2021.
[13] Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. Deepsigns: An end-to-end water- marking framework for ownership protection of deep neural networks. In Proceedings of the twenty-fourth international conference on architectural support for programming languages and operating systems, pages 485-497, 2019.
[14] Tian Dong, Shaofeng Li, Guoxing Chen, Minhui Xue, Haojin Zhu, and Zhen Liu. Rai2: Responsible identity audit governing the artificial intelligence. In NDSS, 2023.
[15] Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Jour- nal of machine learning research, 7(6), 2006.
[16] Dan Goodin. Hugging Face, the GitHub of AI. hosted code that backdoored user devices. / httpa///wwaaaaaaaaa///////////// face---aaceeeee the faace-the-githube github-ort of-a bacccrddddd-------ccccoooeeeeeee
2024. Accessed: 2025-01-20.
[17] Jiyang Guan, Jian Liang, and Ran He. Are you stealing my model? sample correlation for fingerprinting deep neural networks. Advances in Neural Information Processing Systems, 35:36571-36584, 2022.
[18] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
[19] Ziheng Huang, Boheng Li, Yan Cai, Run Wang, Shangwei Guo, Liming Fang. Jing Chen, and Lina Wang. What can discriminator do? towards box- box-free ownership verification of generative adversarial networks. In Proceedings of the IEEE/CVF international conference computer vision, pages 5009-5019, 2023.
[20] Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M Ziegler, Tim Maxwell, Newton Cheng, et al. Sleeper agents: Train- ing deceptive 1lms that persist through safety training, arXiv preprint arXiv:2401.05566, 2024.
[21] Hugging Face Team. Hugging Face: The community building the future. https:/ //hugg ingface. co, 2023. Accessed: 2024-01-18.

---

[22] Amazon Artificial General Intelligence. The Amazon Nova family of models: Technical report and model card. attps://wuxd amazon. www.aa/ww..ccccccccciccaaaaaaaaa
ooteeheenneteeeeeeeeeeeeee amazon amazon-nova-1
2024. Accessed: 2025-01-15.
[23] Hengrui Jia, Hongyu Chen, Jonas Guan, Ali Shahin Shamsabadi, and Nicolas Papernot. A zest of lime: Towards architecture- -nnpeteddenneeeeeeentt model distances. In International Conference on Learning Representations, 2021.
[24] Hengrui Jia, Christopher A Choquette-Choo, Varun Chandrasekaran, and Nicolas Papernot. Entangled watermarks as a defense against model extraction. In 30th USENIX security sympo- sium (USENIX Security 21). pages 1937-1954, 2021.
[25] Torsten KrauB, Jingjing Wang, Jasper Stang. Jie Wan, Alexandra Dmitrienko, Lin Long, Zhiyu Zhang, Ziqi Yang. Yeonjoon Lee, Guangdong Bai, et al. ClearStamp: A Human-Visible and Robust Model-Ownerne Proof based on Transposed Model Training. In 33rd USENIX Secu- rity Symposium (USENIX Security 24), pages 5269-5286, 2024.
[26] Yiming Li, Linghui Zhu, Xiaojun Jía, Yong Jiang, Shu-Tao Xia, and Xiaochun Cao. Defending against model stealing via verifying embedded external features. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1464,147,, 2022.
[27] Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by conferrable adversarial examples. arXiv preprint wrXiv:191..0080. 2019.
[28] Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. CoRR, abs/2104. 10706, 2021.
[29] Hope McGovern, Rickard Stureborg, Yoshi Suhara, and Dimitris Alikaniotis. Your large lan- guage models are leaving fingerprints. arXiv preprint arXiv:2405.1403 2024.
[30] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine- generated text detection using probability curvature. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages -4950-24962. PMLR, 2023.
machine-generated
[31] OpenAl. Openai API 1.
2023. Accessed:
[32] OpenAl. Using logit bias to alter token probability with the OpenAl API. https://help... httaaa/a.////aaatttee/////////// -ity--ithh-thh---peaa---aap 2023, Accessed: 2024-12-15.
[33] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4954-4963, 2019.
[34] Dario Pasquini, Evgenios M Kornaropoulos, and Giuseppe Ateniese. Llmmap: Fingerprinting for large language models. arXiv preprint arXiv:2407.1588 2024.
[35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages 8024-8035. 2019.
[36] Wenjun Peng, Jingwei Yi, Fangzhao WU, Shangxi Wu, Bin Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, and Xing Xie. Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7653-7668, 2023.
12

---

[37] Zirui Peng, Shaofeng Li, Guoxing Chen, Cheng Zhang, Haojin Zhu, and Minhui Xue. Finger- printing deep neural networks globally via universal adversarial perturbations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13430-13439, 2022.
[38] James Pomfret and Jessie Pang. Chinese researchers develop Al model for military use on back of Meta's 's Llama. 2024. Accessed: 2024-12-18.
[39] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[40] Shuo Shao, Yiming Li, Hongwei Yao, Yiling He, Zhan Qin, and Kui Ren. Explanation as a watermark: Towards harmless and multi-bit model ownership verification via watermarking feature attribution. In Network and Distributed System Security Symposium (NDSS), 2025.
[41] David Thiel. Investigation Finds Al Image Generation Models Trained on Child Abuse. http -iddeti-----d-----------
s: 8://cyber. fsi www.ssrd.ddddeeoe wdw/ htttt////////////eeeeeeeeeeeeeee 2024. Accessed: 2024-12-20.
[42] Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristempart. Stealing machine learning models via prediction APIs. In 25th USENIX security symposium (USENIX Security 16), pages 601-618. 2016.
[43] Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin ichi Satoh. Embedding water- marks into deep neural networks. In Proceedings of the 2017 ACM international conference on multimedia retrieval, pages 269-277, 2017.
[44] Asim Waheed, Vasisht Duddu, and N Asokan. Grove: Ownership verification of graph neural networks using embeddings. In 2024 IEEE Symposium on Security and Privacy (SP), pages 2460 2470,2247,000 2024.
[45] Tianhao Wang and Florian Kerschbaum. Riga: Covert and robust white-box watermarking of deep neural networks. In Proceedings of the web conference 2021, pages 993-1004, 2021.
[46] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's 's transform- ers: State-of of-the-rrt art natural language processing. arXiv preprint
2019.
[47] Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc Ilama 7b, 2024. Accessed: 2024-07-26.
[48] Cheng Xiong, Guorui Feng, Xinran Li, Xinpeng Zhang, and Chuan Qin. Neural network model protection with piracy identification and tampering localization capability. In Proceedings of the 30th ACM International Conference on Multimedia, pages 2881-2889 2022.
[49] Jiashu Xu, Fei Wang, Mingyu Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instruc- tional fingerprinting of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 3277-3306, 2024.
[50] Tianlong Xu, Chen Wang, Gaoyang Liu, Yang Yang, Kai Peng, and Wei Liu. United we stand, divided we fall: Fingerprinting deep neural networks via adversarial trajectories. Advances in Neural Information Processing Systems, 37:69299-69328. 2024.
[51] Shojiro Yamabe, Tsubasa Takahashi, Futa Waseda, and Koki Wataoka. Mergeprint: Robust 2024.
fingerprinting against merging large language models. arXiv preprint
[52] Yifan Yan, Xudong Pan, Mi Zhang, and Min Yang. Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation. In 32nd USENIX Security Sym- posium ( (USENIX Security 23), pages 2347-2364, 2023.
13

---

[53] Zhiguang Yang and Hanzhou Wu. A fingerprint for large language models. arXiv preprint arXiv:2407.01235, 2024.
[54] Ziqing Yang, Yixin Wu, Yun Shen, Wei Dai, Michael Backes, and Yang Zhang. The challenge of identifying the origin of black-box large language models. arXiv preprini arXiv:2503.04332.
2025.
[55] Boyi Zeng. Lizheng Wang, Yuncong Hu, Yi Xu, Chenghu Zhou, Xinbing Wang, Yu Yu, and Zhouhan Lin. Huref: Human-readable fingerprint for large language models. Advances in Neural Information Processing Systems, 37: 26332-126362.. 2024.
[56] Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Marc Ph Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 on Asia conference on computer and communications security, pages 159 172, 2018.
[57] Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. REEF: Representation encoding fingerprints for large language models. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
[58] Ruikai Zhou, Kang Yang. Xiuling Wang, Wendy Hui Wang. and Jun Xu. Revisiting black-box ownership verification for graph neural networks. In 2024 IEEE Symposium Security and Privacy (SP), pages 210-210, 2024.
[59] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint 10X4, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency. research ethics, and societal impact. Do not re- move the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evalu- ation. While "[Yes] is generally preferable to "[No] it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer is often more nuanced, so please just your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
used"). In
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Check- list",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The theoretical guarantee and the proposed algorithm are illustrated in Sec- tion together with discussion of assumptions and limitations. Detailed proofs and algorithmic descriptions can be found in the appendix. Experimental results are illustrated in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
15

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations discussed potential sources of emror of our tester in Sec- tion 3.2. Evaluation of these assumptions in practice are in Section 4.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiscless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our algorithms' theoretical guarantees are derived via standard hypothesis testing or best arm identification results whose proofs are established results. We cite the theoretical results that underpin our tester's guarantee.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
16

---

Answer: [Yes]
Justification: We describe our evaluation setup in Section 4. Further details about the benchmark selection are in Appendix C. We also give algorithms and prompt generation strategies for all of the optimizations in Appendix H. We give details such as the algorithm for the offiine optimization via multi-armed bandits in Appendix I.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language modell, releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will provide the code to train and evaluate the proposed algorithm, which reproduces the experiment results in the paper after the rebuttal process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.o cc/pu Salic/guidee/Codeee
ssi onPolicy) for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. cc http://///gueee/ddddessssisseeee
for more details.
17

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details of our experiments including sampling optimizations, benchmark collection and hyperparameters such and T in Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have conducted repeated experiments to obtain more comprehensive re- sults, such running precision/recall for different significance levels in Figure 9, Ap- pendix G and evaluating the impact of randomness on the tester by using five different randomly sampled sets of prompts in Table 6, Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of eπrors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
18

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Answer: [Ys]]
Justification:
We wrote the details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:// http:////.......ccccpppc///tttta http:////b.////tccccce.eeeeeeell
Answer: [Yes]
Justification:
The research is conducted with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This paper proposes provenance testing for models that can impact taking down vulnerable models or compliance to auditing practices for large language models which is crucial for safety.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
19

---

If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks,
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
This paper cite the original papers such as dataset, model or platforms.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- wewerssithhodde..m age should be provided. For popular datasets, paperswi thcode. com com/datase dataasete has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification:
This paper will release code for running experiments and it is well docu-
mented.
Guidelines:
The answer NA means that the paper does not release new assets.
20

---

Researchers should communicate the details of the dataset/cooeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects Question:
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
21

---

Please refer to our LLM policy http:://coocrrssscccooooorrreess for what should or should not be described.
22

---

Algorithm 2 Model Provenance Tester for Pair (J,g)
Require: Pair (J, g). set of control models C (41 (c1,-.. Jm prompt space Q2 number of prompts
T, significance parameter a, statistical test ZTest.
tid
21 FT
b. Sample T prompts
f T Σ (f(2z) = g(x;))
1
>. Calc sim of f and
for 1 + to m do
14 < + E== ((c((z))) g(z,))
Cale sim of c and
P: ZTest(H, p.,
Obtain p-values
end for
(P(1)...., P(m)) - Sort .o Pm)
Sort p-values
for k t to m do
Q4 f aa
+
Holm-Bonferee adjustment D Not provenance pair
if P(k)
then return FALSE
end if
end for return TRUE
provenance pair
6 Appendix
A
Related Work
We further detail prior work's 'ss required access to the model and any additional knowledge to identify show ownership of model, in particular large language models in Table 3.
White-box setting is the most prevalent in existing works. particular, is assumed that the analysis done by the model owner defender who has access to its model weights. Our approach is fully black-box based and only analyzes the first output token of each query. For instance, in [53], the defender has access their own model's parameters and the last linear layer as fingerprint, and aims to verify ownership through API access to the suspect model. LLMmap considers only different versions of the same model and 5 model variants that have resulted from fine-tuning [34]. LLMmap proposes to train model on a set of queries to detect unusual responses from a model which may come in response to an attacker's 's fingerprinting probing. It uses pre-trained textual embedding model to generate vector representations of the query and responses that they then use to classify and identify models. It also uses heuristics identify the most promising prompts for model identification. Other LLM fingerprinting approaches uses system-level statistics to analyze the fingerprint models such inter-token times when these are available either locally over the network [1].
B Basic Tester
A pseudo-code of the basic tester is given in Algorithm 2. In summary, when the procedure returns rate (FWER) is controlled at the significance level
True, it ensures that the total family-wise of a = 0.05 or lower. This means we can confidently state that the similarity between f and is significantly higher than between and any control model, supporting the existence relationship. Conversely, when the procedure returns False, it indicates that we could not establish this higher similarity with the desired level of statistical significance. This may occur either be- cause there is genuinely no significant similarity indicative of provenance or because the test lacked sufficient power under the given parameters (e.g., sample size or number of prompts) to detect it.
provenance
C Models and Benchmarks
We collect model candidates for provenance pairs from the Hugging Face (HF) platform [21]. Since there is no inherent ground truth to determine whether two models constitute a provenance pair, we employ multiple heuristic approaches. These include analyzing metadata available through the HF API and comparing the weights of downloaded models. We consider the most reliable ground
23

---

Table 3: Comparison of our work to related work based on fingerprinting.
Work
Access Type
Technique
Additional Knowledge
Our
Black-box with only
the
Hypothesis testing with sta- tistical guarantees
No
next token
[34]
Black-box with logits
Proposes to train model on set of queries
Pre-trained embeddings for classifier; Queries manual crafted and domain knowl- edge
[57]
White-box model internal representations
Classifiers trained on repre- sentations
No
[51]
White-box. Models are em- bedded with fingerprint key pairs
Embedding robust finger- prints into the owner model.
Changes in training proce- dure
[1]
System-level statistics such as inter-token times
Deep learning to classify
Network traffic
[4]
Same as LLMmap
LLMmap and other dynamic fingerprinting
Judge model, additional ac- cess to classifier models and manual fingerprinting
[53]
White-box. Model's param- eters and the last linear layer as fingerprint and API ac- cess to the suspect model.
Vector similarity
No
[29] Takes datasets of AI- Linguistic features for simi- lar styles
No
generated texts and classi- fies them in families
[54]
White-box access base model, black-box access to the suspect model
A proactive fine-tuning ap- No proach to modify the output of the model
[28]
Knowledge of the training dataset
Statistical testing and dis- tance to margin
Classification models
[30]
Log-probabilities of the tar- get and random perturba- tions of the input passage another model
Used to decide machine- Additional model generated text
truth to be cases where model uploaders explicitly specify their model as fine-tuned version of an- other model, indicated by the presence "base model finetune: <basemodel name>" keyword in the model description on HF. When this explicit indication is not present, we resort to less reliable methods: we attempt infer parent-chh child relationships through model naming patterns and by ana- lyzing model descriptions on HF. Additionally, we identify potential provenance pairs by measuring the similarity between model weights, assuming that highly similar weights suggest a parent-child relationship. From these models we build two benchmarks.
The first benchmark, called BENCH-A, consists of LLM pairs for model provenance constructed from popular pre-trained models and their tine-tuned derivatives, To build this benchmark, we man- ually selected 10 widely-used pre-trained models (refer Tbl. 4) with between billion and billion parameters (the upper bound was determined by our GPU memory constraints). Among these, we purposefully included four pairs of architecturally similar models from Meta, Microsoft, Google, and AliBaba to evaluate our tester's s ability to distinguish between closely related base mod- els and to have some control models. For each pre-trained model, we then randomly sampled 10 fine-tuned derivatives using the Hugging Face API (i.e. use highly reliable ground truth verification), prioritizing diversity in model creators. This sampling strategy resulted 100 derived models, that constitute AENCH-A.
24

---

Hugging Face Model
params
Table All 10 ppre-trained LLMs from BENCH-A
openai com
124,439,008
(left) and top 10 from BENCH-B (right).
/lystthrAA//yytiiae-000 ni ctr.casstt/iaal.oo77ddddaa
70.426,624
345.000,000
Hugeing Face Model
params
Tecebock/opt-125un
125,239,296
diattilbert/isttllepp2
81.912.576
3,212,449,224
774,030.080
Aerrooott/Pht--3mmiid--------
3,821,079,552
openai-commnn
354,823, 168
nicrosoft/phi-2
2.779,683,8400
Ovea/Qnea2-O.58
goegle/ganma-25
2,506.172.416
Jackfram/llamm-6Be
68, 28.0030,200
goagle/gemaa-2-2
2,614,311,88,
BleutherAA//ggt--ee-1122
125,198,59 198,992
Qven/Oven2-11
1,543.714.344
Dpen/Quen2.1 5-1, In-sstract
1.543.714,304
deapsesk-at httpaaaa///appeeee-ce-ee---- -bb-baaa 3b-hase
1,346,471,99 1,966
L1100,088,384
Table 5: Domain-specific Hugging Face models included in analysis.
Domain
Model
Hogging Face URL
Medical
medical transcriptinn generator
https: http////gggnggccee/e///// caaeeeeccccllllll
http://gaaggacc...o/alaaaaaaaaaa co/alibiddaraaawedialll
Medical
ept2-largo-mddial
bttpa: httpe://hugtnggaac http::///wgggggaoooooooootttttu
Medical
healthbot
ggtge://ugginggfce.coo co
amams00/heealtbboo
finetuned fraud FinguAl-Chat-il
https://uuginggaace wtwp:////////ooeeee/eeoo
finn-nin
Financial
https:
Financial
FinOPT-Lincoll
htap:::/huginngaacccc//o/
Biology
distigpe-finetuned-mmie innctuncdmmiccrioology
ggtp::/auggiggfaee wtt.:/eeccccaceeeeeeeeeeeeeeeeee
Astrology
astroGPT
https: http:://w/gggggaaccccooo////// htt.:///....oetteaaaa/////rrrrr
Game
maglo-the-gatherin gatheringheggg
https: h/tp///ggggggcceeeee/////////eee http://ngfaacccccaaiiiiiiiieeiie http://iinni//aaiiiiaaaeee
Game
chessgn12 medium-
https:
https://huggee bttps://huggingaaace htt.://sww.....e.a///////gg/g/e/
The second benchmark, denoted as BENCH-B, was constructed through more automated and com- prehensive approach. We began by downloading the 1,000 most popular models from Hugging Face with less than IB parameters, ranked by download count. We then filtered out non-English models" and those exhibiting low entropy or high self-perph seff-perplexity, which are indicators of poor training quality or insufficient learning*. This filtering process resulted 608 viable models. To establish ground truth provenance relationships among these models, besides the model owners provided f fine-tune keyword approach, we also used the other less reliable methods. Through this analysis, we identi- fied 57 pre-trained models and established 383 ground-truth model provenance pairs. The remaining 148 models are considered to be independent, having no clear derivation relationship with any other models in analyzed set. Part of models from BENCH-B B given in Table 4.
Examples of Candidates' Domains from Bench-A/B. find examples of candidates for domains such financial, biology and medical among others. See Table 5 for list of examples.
Experimental Setup. We run our model provenance testers on a Linux machine with 64-bit Ubuntu 22.04.3 LTS, 128GB RAM and 2x 24 CPU AMD EPYC77443) @ 1.50GHz and 4x NVIDIA A40 GPUs with 48GB RAM. All experiments are implemented using PyTorch framework [35] and the Hugging Face Transformers library [46].
D Sampling Prompts
To produce prompts for our provenance testers, we use indiscriminately five popular LLMs: gemini-pro-1.1 claude-3. 5-sonnet, momini-flash-1.5. deepseek-chat, and gpt-4o-mini. Each produced prompt is an incomplete sentence containing five to twenty words refer to Table 6 for examples.
Table 6: Examples of prompts.
1 In response to mounting public pressure, the concerned
2 The bright star known as Antares was visible even from
3 The surgeon prepared the instruments for delicate
4 Scholars carefully examined the fragile
5 The phonetics lecturer explained the intricacies of the
DDue to lack of control models them. WWe avoid testing low quality models.
25

---

TUB
BENCH-A
BENCH- B
rocall
recal
0.99
0.83
0.94
0.68
0.98
0.86
0.95
0.67
1.00
0.83
0.94
0.66
Figure 6: Precision and recall of the provenance tester on BENCII-A and BENCII-B with five differ-
of 1,000 prompts.
Model
Model 2
F=1
F=
Kart-elarge
KDt2-neduug mediun
U.64
0.36
0.22
0.16
gpt-2-1arge
0.64
0.38
0.25
0.15
pythia-41Ce-dadupp
pythia-410n
0.37
0.24
0.15
gpt2-mediun
sogatroo-got2-9650
0ven1. 8-0.83
Sailer-0.
0.61
0.35
0.20
0.17
average
0.33
0.13
0.08
Figure 7: Most similar pre-trained models from BENCH-B sorted for 1 (no advanced prompt sampling), and their corresponding values for k = 4, 16, 64.
E The Effectiveness of Rejection sampling
Certain pre-trained models from BENCH-A and BENCH-B exhibit high degree of similarity when comparing their output tokens generated from random prompts. Table presents the top 5 most similar model pairs from BENCH-B, measured by the percentage of matching output tokens when tested on 1, 000 random prompts.
Table 7 demonstrates how the percentage of matching tokens changes with rejection sampling (columns 4, 16, and 64). For example, the in matching output tokens from 64% (k = 1) to merely 16% (k: 64), indicating that rejection sampling significantly reduces token overlap between models. This improvement directly enhances the efficiency of provenance testing by reducing the tester's 's online complexity.
similar pair of models shows reduction
F
Testing with Known Parent
0.0
0.8
Predision
Precision Bench-A Known Parent
Precision Bench-A Known Parent
0.6
Preditoon 0.05
Recall Bench-A Known Parent
Recall Bench-A Known Parent
0.4
Pracsion #01
Recalia 0.01
0.2
Recall Bench-B No Parant
Procision Bench-3 Parent
0.2
Recall Banch-B No Parent
Precision Bench-B No
Recali
R - 0.65
1000
5000
Prompts
Figure 8: Precision/recall the tester on BENCH-A (left) and BENCH-B (right) without known parent (dark blue and dark red), and with a suspected parent (light blue and light red) ).
Figure 9: Precision and recall
of running the model provenance tester on BENCH-A with differ- ent significance levels a.
We evaluate the provenance tester for cases where the parent model is given, addressing whether a pair of models (P, C) constitutes provenance pair.
We construct test pairs from both BENCH-A and BENCH-B benchmarks. From BENCH-A. we take all 100 true pairs (P;, C.) and create 100 false pairs (P,, C;) by selecting one random non-parent P; + P; for each child C, This ensures balanced dataset where random guessing would achieve 50% accuracy. We similarly obtain 766 testing pairs from BENCH-B
Results from testing both benchmarks are shown in Figure As expected, the tester performs better when the suspected parent is known compared to cases with unspecified parents. While re- call remains unchanged, precision reaches 100%. This improvement occurs because false positives now require both that the statistical hypothesis test returns the wrong parent and that this wrong parent matches the suspected parent. The precision with known parents (P (Penownparent) is lower
Unlike BENCH-A, BENCH-B B already contains (child) models that have no known parent among the 57 pre-trained models, and these one of the negative pairs.
26

---

bounded by the precision with unspecified parents (P (Fenpeccifsed) and depends on how we sam- ple the incorrect parents. With uniform random sampling, can be estimated Psnownparent 100 100- Punspecl where n is the number of pre-trained models. For both benchmarks, this yields precision slightly below 100%.
G Precision/Recal for Different a
We also run the model provenance tester for BENCH-A with different significance level a {0.01, 0.05, 0.1}. The results of these experiments are given in Figure 9. As evident from the results, is clear that the significance level does not have any major impact on the efficiency of tester.
H Advanced Sampling of Prompts
We further give the rejection sampling of prompts used to reduce the online complexity of the tester. We present only the sampling procedure, i.c. how set of prompts Z ....... .......y UTp' is produced given the original prompt sampler Ω2 and the two sets of models: the parent candidate set F, and the control set C. The tester that uses the new set is identical to the original tester, with the only difference that samples prompts from the new set. The sampling is given in Algorithm 3.
Algorithm Advanced Prompt Sampling
Require: Parameter k, Candidate set F (fi..... fsh. set of control models = {ci...., Gm prompt space Q2 number of prompts T
Prompts 4- 0
b Initialize empty set of prompts All models
H + C
same
[ + for all i,j [[]]]
D. Counter of times two models produced the same output
token
for +- to T' do
I1...., Ft ~ Ω Score [j] for j + to k do
> Sample i prompts Scores across k prompts b Find score for each
for < to
8
for 1 + to H do
for L2 + to do
old + same|//]|gg]
1-1 new + (omeel/llgl+[(hii (zy (zz)==;;,()) (z₂))
weight. (old-neww)
3 + I(old > new) weight
end for end for Score[j] f-
end for end for Scorel
1 t arg max, Score[j]
b Find largest score
Prompts + Prompts U {ι)} return Prompts
Add that prompt
Reducing Offline Queries with Best Arm Identification
To reduce the offline queries to the parent and control models replace the hypothesis tests with Best Arm Identification (BAI) algorithm (that provides as well theoretical guarantees on confidence). For practical purposes, in implementation given in Algorithm 4 use the BAI proposed in [15].
27

---

Algorithm 4 Tester based on Best Arm Identification
Require: Model g. candidate set F (f1.... f.), set of control models C' {ei,.... Cm }
prompt space S2, number of prompts significance parameter a, maximum average prompts per
model N
M + FUC
P. Set of all models Confidence interval for BAI # same tokens with # queried
(tt, a) = / 1og(112/o) /o)
21
hits mm] - for all m E M
tots m² 0 for all m M
A + .
Active set of models
t
while TRUE do
3 iid Ω
D. Sample prompt
Uy c g(x)
D. Query
for m E A do
D. Update hits/tots query model m
Vm <- m(x)
hits(m] +
1 (Ug Vm)
Update hit counters
tots [m] 1
Update total queries
end for
hits(m) Ubest MaXMEM mxs(m
> Find best μ Confidence radius
+- (t,
for E A do
if Phest
bits then
4 A m
jm too far from best
end if end for
|A| = then
b Only model left
break
end if
Σ ΣMEM [ots/m| N |M] then
Reached max queries
break
end if
t + i f++1
end while
if |A| = and A C F then
b Model needs to be from F
return (TRUE, A)
end if
return FALSE
28


Reviews:
[{"review_id": "GZ2DVcxycn", "reviewer": "Reviewer_JAwj", "summary": "This paper introduces a practical framework for model provenance testing in large language models (LLMs), under the realistic constraint of black-box access. The key insight is that fine-tuned models retain detectable output similarities to their parent models, which can be leveraged via hypothesis testing to establish provenance relationships. The method requires no knowledge of training data or architecture, and is benchmarked on over 600 real-world Hugging Face models (BENCH-A/B), achieving 90–95% precision and 80–90% recall. The authors also propose query-efficient variants using entropy-based prompt selection and bandit algorithms.", "questions": "1.How robust is the proposed method to adversarial fine-tuning or deliberate obfuscation aimed at evading provenance detection?\n2.Can the authors provide a more formal theoretical characterization of their method? Specifically, what are the fundamental conditions under which the test is expected to succeed or fail? A clearer mathematical problem formulation would greatly strengthen the clarity and rigor.\n3.Is there any provable regime (in terms of model divergence, prompt space, or number of queries) where the test is guaranteed to be accurate or, conversely, provably fails?\n4.Could shared pretraining data or overlapping corpora (without actual fine-tuning) cause false positives? How does the method distinguish true derivation from incidental similarity due to data contamination or common training sources?", "limitations": "Incorporating a theoretical analysis of the method's effectiveness would, in my view, substantially strengthen the novelty and contribution of this work.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.This paper is motivated by a timely and practical problem: how to determine whether an LLM is derived from another in realistic post-deployment settings where only API access is available.\n2.The authors conduct extensive empirical validation on two benchmarks with over 600 models, covering a wide range of sizes and domains. Precision and recall results are strong and reported with appropriate statistical care.\n3.Limitations and assumptions are clearly discussed; error sources are empirically analyzed.\n\nWeaknesses\n\n1.This paper assumes that fine-tuned models remain highly similar to parents, which may not hold under heavy domain shifts or adversarial training.\n2.This paper lacks a theoretical guarantee on the effectiveness of the method beyond standard hypothesis testing; success depends on empirical regularities.\n3.The control model selection is handled via simple random inclusion from popular models, which works in aggregate, but may struggle in edge cases (e.g., specialized domains).\n4.The method currently focuses only on next-token prediction and ignores richer output structures, such as full generations or probability distributions. This limits sensitivity, especially for models trained to behave similarly on surface form but differ semantically.", "was_revised": true, "final_justification": "The rebuttal does not address core issues. Adversarial robustness remains untested beyond a non-adaptive threat model. Theoretical characterization is minimal—limited to standard FWER control without conditions or bounds for the core similarity assumption. Empirical scope is narrow, relying solely on next-token prediction despite acknowledged sensitivity loss for richer outputs. I maintain my reject recommendation.", "rebuttal": "We thank all reviewers for their time and constructive feedback. Based on the feedback, we will revise the paper to clarify points raised by the reviewers.\nBelow, we first provide a general response to outline the scope and goals of our work and then respond to each reviewer's specific points in the order we have received them. \n\n\n# General Response\n\nOur work addresses the challenge of model provenance testing, a practical problem formulation that, to the best of our knowledge, has not been systematically analyzed before. We introduce the problem and provide a solution in one of the most practical (and constrained) settings: requiring only black-box API access to models that return just the next predicted token.\n\nOur goal was to propose and empirically validate an initial framework for this new problem. However, we recognize that the problem space is extensive, with many potential variations. A comprehensive analysis of all these variations was beyond the scope of this work, which is the first to establish the viability of the core approach. The reviewers'  questions—for instance, regarding alternative similarity metrics, different derivation techniques, and more complex threat models—highlight several of these valuable and important directions for future investigation. Our approach is extensible and we believe it provides a solid starting point for future work.\n\n\n# Response to Reviewer JAwj\n\n* **On adversarial robustness (Weakness 1, Q1):** Our work considers a non-adaptive adversary. As the first paper to systematically address model provenance with such minimal assumptions, our focus is necessarily on this foundational threat model. While investigating adaptive attacks is an interesting direction for future work, it remains unclear if they could succeed without significantly degrading model performance, as we discuss in our submission (lines 76-80).\n\n* **On theoretical guarantees (Weakness 2, Q2, Q3):** Our method's theoretical guarantee is the formal statistical control of the Family-Wise Error Rate (FWER) via the Holm-Bonferroni procedure [16]. This is a standard and a robust result from statistical testing. Our central assumption—that derivation implies statistically significant similarity—is then validated empirically in Section 4.2, where we show it holds for the vast majority of cases in our large-scale benchmarks. Therefore, our tester’s accuracy is backed by formal statistical guarantees under an assumption that we demonstrate in real-world settings.\n\n* **On control model Selection (Weakness 3):** Our selection of control models was intentionally designed to be simple and unbiased (using all available top pre-trained models) to test the framework's general robustness and avoid introducing selection bias. The high performance on the diverse BENCH-B, which contains a wide variety of models (in total 57), demonstrates this approach works well without needing domain-specific manual curation for control models. Our two benchmarks have 600+ candidates, so manual curation would not have been feasible. Due to the large benchmark size, we expect to extensively cover the fine-tuning setup where a model is trained for specialized domains on custom datasets. For instance, we find examples of candidates for domains such as financial, medical and more (see below for a sample). We have not evaluated other selection strategies, which remains a direction for future work. We plan to release our benchmarks which we hope will create a baseline for future model provenance testing approaches.\nExamples of candidates’ domains from Bench-A/B:\n   - (Medical) https://huggingface.co/alibidaran/medical_transcription_generator  \n   - (Medical) https://huggingface.co/Locutusque/gpt2-large-medical\n   - (Medical) https://huggingface.co/Anjanams04/healthbot\n   - (Law) https://huggingface.co/jslin09/bloom-560m-finetuned-fraud\n   - (Financial) https://huggingface.co/FINGU-AI/FinguAI-Chat-v1\n   - (Financial) https://huggingface.co/MayaPH/FinOPT-Lincoln\n   - (Biology) https://huggingface.co/as-cle-bert/distilgpt2-finetuned-microbiology\n   - (Astrology) https://huggingface.co/stevhliu/astroGPT\n   - (Game) https://huggingface.co/minimaxir/magic-the-gathering \n   - (Game) https://huggingface.co/dakwi/chessgpt2-medium-l\n\n\n* **On using only next-token prediction (Weakness 4):** We focus on the next token as it represents the minimal signal available in our black-box setting. Our results demonstrate that even this minimal signal is sufficient for high-accuracy provenance testing. One can consider stronger signals like n-grams, at the expense of increasing the required sample complexity. For instance, in small-scale, unreported experiments, we observed that the similarity rate is drastically reduced (as the probability of a match on two consecutive tokens is much lower than on one), suggesting a significantly larger number of queries would be needed to obtain statistically significant results.\n\n* **On distinguishing provenance from shared training data (Question 4):** This is a challenge as the models' exact training datasets are unknown. However, our benchmarks implicitly test this scenario. For instance, they include distinct models from the same family (e.g., google/gemma-2b and google/gemma-2-2b as well as a few other pairs) that are expected to have highly similar training datasets. The tester's high precision demonstrates its ability to distinguish the strong statistical signal of fine-tuning from the weaker background similarity arising from a shared datasets.\n\n[16] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70, 1979."}, {"review_id": "HSK9jDgyIR", "reviewer": "Reviewer_ExqM", "summary": "This paper presents a framework for testing the provenance of large language models (LLMs). The motivation for this work stems from rising concerns about issues such as model stealing and backdoor vulnerabilities found in LLMs. The proposed method operates without any assumptions related to the training procedures or data, requiring only black-box access to the models.\n\nThe method is based on the empirical observation that fine-tuned models typically have output distributions more similar to those of their parent models than to unrelated ones. The authors employ statistical hypothesis testing to infer model provenance and validate their hypothesis. The method has been evaluated on a diverse set of LLMs from Hugging Face, ranging from 30 million to 4 billion parameters, achieving over 90% precision and over 80% recall. This represents the first systematic and practical approach to model provenance testing in constrained settings.", "questions": "Can we test with semantic similarity of the outputs a similar approach?", "limitations": "The theoretical assumption that models will behave similar if derived from other is strong, but further testing on specific types of fine-tuning and post-training phases is needed.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The problem has been very clearly defined. It is evident that this is a significant issue in the machine learning community, and the proposed solution addresses it to some extent. Additionally, I appreciate the authors' caution regarding false positive and false negative cases. This acknowledgment should alert the community to the possibility of such occurrences, prompting model owners to take action if they find that other models derived from theirs.\n\nOne of the strengths of this work is framing model provenance testing as a statistical hypothesis testing problem. The extensive evaluation provided also contributes significantly and novelly to the community.\n\nThe related work on watermarking and fingerprinting is accurate and highlights the novelty of the proposed approach. However, I would like to see more related work on the similarities in LLM outputs. While these studies may not focus on model provenance, several have compared LLM behavior across different datasets.\n\nI have only a few minor concerns regarding this work. First, there could be a comparison with other baseline methods. For example, using output similarity (by measuring the similarity of embeddings) could help assess how well the method performs relative to simpler approaches.\n\nAdditionally, the initial hypothesis may have limitations depending on how the model has been fine-tuned. Further studies in the post-training stages could provide the community with more insights about the robustness of these methods.\n\n\nIn Summary, this is a novel and valuable contribution to the community, is well-conducted, and starts an open path for statistically asses model provenance.", "was_revised": true, "final_justification": "While the authors do not definitely solve the challenging problem they are presenting, I think the paper provides a great contribution on the field in that sense and is worth publishing.", "rebuttal": "We thank all reviewers for their time and constructive feedback. Based on the feedback, we will revise the paper to clarify points raised by the reviewers.\nBelow, we first provide a general response to outline the scope and goals of our work and then respond to each reviewer's specific points in the order we have received them. \n\n\n# General Response\n\nOur work addresses the challenge of model provenance testing, a practical problem formulation that, to the best of our knowledge, has not been systematically analyzed before. We introduce the problem and provide a solution in one of the most practical (and constrained) settings: requiring only black-box API access to models that return just the next predicted token.\n\nOur goal was to propose and empirically validate an initial framework for this new problem. However, we recognize that the problem space is extensive, with many potential variations. A comprehensive analysis of all these variations was beyond the scope of this work, which is the first to establish the viability of the core approach. The reviewers'  questions—for instance, regarding alternative similarity metrics, different derivation techniques, and more complex threat models—highlight several of these valuable and important directions for future investigation. Our approach is extensible and we believe it provides a solid starting point for future work.\n\n# Response to Reviewer ExqM\n\nThank you for recognizing the novelty and value of our paper’s contribution.\n\n* **On related work:** As suggested, we can expand Section 2.1 to include recent studies that compare LLM output similarities, which will better contextualize our work.\n\n* **On baselines & semantic similarity (Question 1):** Our statistical framework is metric-agnostic. Using the semantic similarity of full outputs is a natural extension that can be directly integrated into our methodology. We agree this is a promising direction for future work to enhance robustness.\n\n* **On the limitations of the fine-tuning hypothesis:** While our work shows the similarity assumption holds for a wide variety of common fine-tuning methods, testing its boundaries against specific post-training techniques (e.g., distillation, extensive RLHF) is perhaps a next step in adapting to our proposed detection. Our paper provides the foundational methodology to enable such future studies."}, {"review_id": "sIImE45ylz", "reviewer": "Reviewer_ckdW", "summary": "The paper proposes a statistically rigorous test for LLM provenance: that is, given black-box (API) access to a target model and a set of candidate parent models and controls, it determines whether the target model is a fine-tune of one of the candidate parents. The procedure asks whether the first-token agreement rate between the target model and the proposed parent is statistically greater than the agreement rate between the target model and each of a set of control models, over a randomly sampled set of prompts, using a one-sided two-proportion Z-test. Since a test is run against each candidate parent, Bonferroni correction is used to control the FWER. Experiments on multiple benchmarks over more than 600 public models achieve 90–95 % precision and 80–90 % recall with 3,000 prompts. Entropy‑based prompt rejection sampling drops query cost by up to 6x with no reduction in accuracy.", "questions": "- Can this method be extended to consider entire model outputs rather than just the first token? Using only the first token means we're unable to take advantage of differences in style or phrasing between models. \n\n- How do you choose the control models? Are there guidelines for doing so? For example, should controls cover a range of architectures, parameter counts, or training data domains?", "limitations": "Prompts $x_1, \\cdots, x_T$ are shared across models, which creates a statistical dependence between the two proportions (parent-target and control-target) used in each Z-Test. Does this violate the test’s variance calculation?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The test is simple and elegant, requires only black‑box access to models, and is original: it is the first technique that offers formal FWER guarantees for model‐provenance claims. The additional entropy‑based prompt rejection is a novel and interesting idea.\n\n- the authors release two large, well‑curated benchmarks. Their evaluation is rigorous and comprehensive, and well-explained (RQ1 - 4). They run ablations on prompt count and control‑set size / selection.\n\nWeaknesses:\n\n- The test’s effectiveness depends on the choice of control models: if a control model happens to be very similar to the true parent (trained on overlapping data, from the same lineage, etc), the baseline agreement rate rises which could drop the power of the test, and the test may fail.\n\n- Because only the first token is used, attackers could fine‑tune a model to match on that single token while diverging otherwise.", "was_revised": true, "final_justification": "My concerns have been addressed by the rebuttal. Thank you to the authors for taking the time to reply!", "rebuttal": "We thank all reviewers for their time and constructive feedback. Based on the feedback, we will revise the paper to clarify points raised by the reviewers.\nBelow, we first provide a general response to outline the scope and goals of our work and then respond to each reviewer's specific points in the order we have received them. \n\n\n# General Response\n\nOur work addresses the challenge of model provenance testing, a practical problem formulation that, to the best of our knowledge, has not been systematically analyzed before. We introduce the problem and provide a solution in one of the most practical (and constrained) settings: requiring only black-box API access to models that return just the next predicted token.\n\nOur goal was to propose and empirically validate an initial framework for this new problem. However, we recognize that the problem space is extensive, with many potential variations. A comprehensive analysis of all these variations was beyond the scope of this work, which is the first to establish the viability of the core approach. The reviewers'  questions—for instance, regarding alternative similarity metrics, different derivation techniques, and more complex threat models—highlight several of these valuable and important directions for future investigation. Our approach is extensible and we believe it provides a solid starting point for future work.\n\n# Response to Reviews ckdW\n\n* **On extending the similarity metric (Weakness 2, Question 1):** Our framework is metric-agnostic. We chose next-token agreement as it is the simplest available signal, and our results show it is surprisingly effective. The core methodology—comparing a candidate parent's similarity against a baseline of control models using hypothesis testing—can be directly applied to more complex metrics, such as n-gram overlap or semantic similarity of entire outputs, but will change the sample complexity therein.\n\n* **On selecting control models (Weakness 1, Question 2):** We adopted a simple and unbiased selection strategy in our experiments (the control set consists of all available pre-trained models), to show that even without manual curation of control models, the framework can still effectively identify model provenance. Potentially more robust selection would be to use a set of pre-trained models that are similar in domain to the target model. For instance,  if the target is a French-language model for generating Python code, good control models would include other French language models, Python code generation models, and ideally other French Python code generation models.\n\n* **On statistical dependence (Limitation):** The prompts are shared, but they are sampled uncorrelated with (independently of) the model pair being tested. This ensures that the set of prompts does not favor any particular parent or control model, and the independence required for the variance calculations thus holds."}, {"review_id": "gehuFt3ZOw", "reviewer": "Reviewer_CgZ4", "summary": "This paper presents a model provenance testing framework tailored for large language models (LLMs). The core insight is to infer derivational relationships between models by quantifying output discrepancies across different models when presented with identical prompts. The authors construct two test benches of varying scales, collecting divergence data from the outputs of a target model $g$, a base model $f$, and a control model ensemble. Leveraging statistical hypothesis testing, the framework analyzes the significance of $g$ being a derivative of $f$, so as to realize the detection of model provenance. Experiments show that the tester achieves a precision of 90%-95% and a recall of 80%-90% in identifying derived models, proving the feasibility of systematic provenance verification through only API access.", "questions": "1. The sizes the models selected in the paper are all below 4B. Then, is this method applicable to models which sizes are above 4B?\n\n2. Model distillation has become an important means of model derivation. Distillation can lead to a significant compression of the model size. Considering that the method proposed in this paper mainly uses models with similar parameter scales, is this method also applicable in the scenario of model distillation?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. This paper presents a model provenance testing method based on multiple hypothesis test.\n\n2. If the weaknesses mentioned below can be clarified, the method described in this paper can strongly prove that Assumption 1 holds true， and it can also be considered that the Bench constructed in the paper has the potential to become a Benchmark for detecting the derivative relationships of the model.\n\n\n**Weakness**\n\n1. It may be flawed in this paper to solely focus on selecting the **next token** when calculating the similarity between the outputs of two LLMs. For instance, when the **next token** is an adverb, pronoun, or other function word, this approach may compromise the validity of the similarity calculation. The author(s) should provide a clear elaboration on how such scenarios were addressed.\n\n2. The paper neither delineates the construction principles for candidate provenance pairs and control sets nor furnishes a comprehensive inventory of Benches or analyzes the distribution of models within the Bench. A critical oversight arises in the potential bias introduced during control set selection—for example, disproportionately including derivative models of a single pre-trained model. The author(s) should clarify and explain in this aspect.\n\n3. The construction of the benchs is relatively simple. For example, in Bench A, 100% of the models are derived models. If I construct a simple model that returns true every time, I can obtain 100% precision and recall. For Bench B, my model can achieve a precision of nearly 80% and a recall of 100%. It seems that the performance of my model is close to the method proposed in this paper.\n\n4. This paper primarily demonstrates the effectiveness of the detection method in model provenance testing via precision and recall metrics. The underlying concept appears analogous to Instance-based clustering or anomaly detection. If so, the methodology could be strengthened by further detailing how multiple hypothesis testing results guide parameter selection for clustering or anomaly detection algorithms. Alternatively, presenting the clustering outcomes of known model provenance relationships using this method would offer a visual means to validate the approach’s efficacy from a complementary perspective.\n\n**Minors**\n\n1. The two sub-figs in Figure 3 are arranged vertically. The descriptions in the caption is explained in left-right. It is suggested to make adjustments and modifications.", "was_revised": true, "final_justification": "Thanks for the authors' rebuttal. They have done a great job in model provenance, using a novel method. I agree with their viewpoint and will update my score from \"2: borderline reject\" to \"5: accept\".", "rebuttal": "We thank all reviewers for their time and constructive feedback. Based on the feedback, we will revise the paper to clarify points raised by the reviewers.\nBelow, we first provide a general response to outline the scope and goals of our work and then respond to each reviewer's specific points in the order we have received them. \n\n\n# General Response\n\nOur work addresses the challenge of model provenance testing, a practical problem formulation that, to the best of our knowledge, has not been systematically analyzed before. We introduce the problem and provide a solution in one of the most practical (and constrained) settings: requiring only black-box API access to models that return just the next predicted token.\n\nOur goal was to propose and empirically validate an initial framework for this new problem. However, we recognize that the problem space is extensive, with many potential variations. A comprehensive analysis of all these variations was beyond the scope of this work, which is the first to establish the viability of the core approach. The reviewers'  questions—for instance, regarding alternative similarity metrics, different derivation techniques, and more complex threat models—highlight several of these valuable and important directions for future investigation. Our approach is extensible and we believe it provides a solid starting point for future work.\n\n# Response to Reviewer CgZ4\n\n* **On the next-token similarity metric (Weakness 1):** While individual tokens can be noisy, our approach relies on statistical aggregation across thousands of diverse prompts, which reveals a robust signal of provenance.\n\n* **On benchmark construction and control sets (Weakness 2):** We will release the full list of models used in our benchmarks for transparency to show the lack of large bias. Moreover, BENCH-A has perfectly balanced parent sets, as each of the 10 pre-trained parent models has exactly 10 fine-tuned children models, thus ensuring no bias in the parent selection.\n\n* **On the perceived simplicity of the benchmarks (Weakness 3):** This appears to be a misunderstanding of our evaluation. We do not evaluate a simple binary classifier for each pair. Rather, our tester (according to Algorithm 1) must identify one single correct parent from a combined set of candidate parents. If it cannot do that conclusively with statistical error rate set, it will output that there is no parent. Thus the baseline precision for BENCH-A is 1/(10+1) = 9%, and for BENCH-B it is 1/(57+1) = 1.7%, not as high as stated in the review.\n\n* **On the analogy to clustering (Weakness 4):** While conceptually similar to anomaly detection, our method provides a more principled framework. Instead of relying on an arbitrary similarity threshold, we use a dynamic baseline of control models to determine what constitutes a statistically significant similarity. This, combined with the formal control of the Family-Wise Error Rate, is a key advantage and contribution of our work.\nA clustering approach doesn’t provide any such statistical guarantees, and our insights in the problem formulation about using control models is complementary to these other techniques.\n\n* **On applicability to models > 4B (Question 1):** The 4B parameter limit was imposed by our available hardware platform. The statistical principles of our tester are model-size agnostic, and we expect the method to generalize to larger models, as the underlying similarity patterns are unlikely to disappear.\n\n* **On model distillation (Question 2):** We have not evaluated our method against model distillation. As the first paper to formalize provenance testing in this setting, our scope was focused on fine-tuning, a common derivation technique. Investigating other derivation methods like distillation can be a direction for future work to explore."}]

Author rebuttal:

Meta-review: This work proposes a method for testing model provenance based on their token-level output distribution. The main contribution lies in a careful approach to hypothesis testing involving unrelated models to calibrate the alignment score. The resulting technique is able to detect whether one model is derived from another with high precision.

Reviewers generally agreed that this is a relevant problem and that the proposed technique, while straightforward, produces strong results across an extensive evaluation. Prominent concerns include the lack of adversarial robustness testing -- the work only considers non-adversarial settings -- and the relative simplicity of the next-token based analysis, which may perform poorly under more demanding settings (including adversarial settings).

Overall, this work presents a reasonably novel and thorough investigation of an effective approach, while not addressing some of the more technically complex topics in this space. In light of its novelty and to encourage further explorations in this space, we are recommending its acceptance.
Decision: Accept (poster)