ErrorTrace: A Black-Box Traceability Mechanism Based on Model Family Error Space
Authors: Chuanchao Zang
Xiangtao Meng
Wenyu Chen
Tianshuo Cong
Zha Yaxing
Dong Qi
Zheng Li
Shanqing Guo
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Intellectual Property Protection
LLM
LLM Safety
Error Space
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: A black-box IP protection mechanism for LLMs, based on a novel perspective using model error information, effectively identifies suspected LLMs and offers practical insights for exploring the LLM error space.

Abstract: The open-source release of large language models (LLMs) enables malicious users to create unauthorized derivative models at low cost, posing significant threats to intellectual property (IP) and market stability. Existing IP protection methods either require access to model parameters or are vulnerable to fine-tuning attacks. To fill this gap, we propose ErrorTrace, a robust and black-box traceability mechanism for protecting LLM IP. Specifically, ErrorTrace leverages the unique error patterns of model families by mapping and analyzing their distinct error spaces, enabling robust and efficient IP protection without relying on internal parameters or specific query responses. 
Experimental results show that ErrorTrace achieves a traceability accuracy of 0.8518 for 27 base models when the suspect model is not included in ErrorTrace's training set, outperforming the baseline by 0.2593. Additionally,ErrorTrace successfully tracks 34 fine-tuned, pruned and merged models across various scenarios, demonstrating its broad applicability and robustness. In addition, ErrorTrace shows a certain level of resilience when subjected to adversarial attacks. Our code is available at: https://github.com/csdatazcc/ErrorTrace.

ErrorTrace: A Black-Box Traceability Mechanism Based on Model Family Error Space
Chuanchao Zang', Xiangtao Meng', Wenyu Chen', Tianshuo Cong26, Yaxing Zha', Dong Qi Zheng Lil.66" SSanqing Guo..5.6,
School of Cyber Science and Technology, Shandong University 2 School of Cryptologic Science and Engineering, Shandong University
China Transportation Information Technology Group Co., Ltd., China
Shandong Branch, China United Network Communications Group Co., Ltd., China State Key Laboratory Cryptography and Digital Economy Security, Shandong University Shandong Key Laboratory of Artificial Intelligence Security, Shandong University
Abstract
The open-source release of large language models (LLMs) enables malicious users to create unauthorized derivative models at low cost, posing significant threats intellectual property (IP) and market stability. Existing IP protection meth- ods either require access to model parameters or are vulnerable to fine-tuning attacks. To fill this gap. we propose ErrorTrace, robust and black-box trace- ability mechanism for protecting LLM IP. Specifically, ErrorTrace leverages the unique error patterns of model families by mapping and analyzing their dis- tinct error spaces, enabling robust and efficient IP protection without relying on internal parameters specific query responses. Experimental results show that ErrorTrace achieves a traceability accuracy of 0,8518 18 for 27 base models when the suspect model is not included in ErrorTrace's 's training set, outperforming the baseline by 0.2593. Additionally, ErrorTrace successfully tracks 3- fine- tuned, pruned, and merged models across various scenarios, demonstrating its broad applicability and robustness. In addition, ErrorTrace shows a certain level of resilience when subjected to adversarial attacks. Our code is available http://....ooomcacaacc//rrorrta.
Introduction
Large language models (LLMs) have advanced rapidly in recent years, with increasing model sizes and expanding training datasets significantly improving their capabilities in language understanding and generation ( 20 28 47). However, these advancements come at the cost of requiring both substantial computational resources and deep domain expertise (27), raising critical concerns about model copyright and intellectual property (IP) rights. One major concern is the risk unauthorized and repackaging of original models. Malicious actors fine-tune open models and claim them as their own, undermining the contributions of the original developers (10 26 27 (28). A notable case in 2024 involved group of Stanford University students allegedly plagiarizing MiniCPM-Llama3-V/ a Chinese open-source model developed by Facade Intelligence (39)
To protect LLM IP, various traceability methods have been proposed. Watermarking, the most prominent approach, typically requires modifying the original model parameters (2) 3. [5] [6]
25
34
38
43 while other methods avoid modifying the model but still require access to its
parameters (21 37 40 41). Thus, these methods are limited to white-box settings, which restricts
their applicability. Black-box query- based methods rely on carefully designed prompts but often
'Correponding authors
httpp:://wet //www globaltimme cn/ hn/paage/222406/133/ 131603/ shtml 39th Conference on Neural Information Processing Systems (NcuriPS 2025).

---

involve high computational costs require human expertise. They are also sensitive to changes in model behavior, such as those introduced by fine-tuning, which limits their robustness [
To fill this gap, we propose implicit finger- printing via data-driven error analysis- black-box mechanism that identifies the likely family origin of suspicious model based solely on its error patterns. Instead of relying on white-box access or care- fully crafted queries, our method analyzes the statistical consistency of prediction er- rors to uncover distinctive failure behaviors shared within model family. The core insight driving our method is that how mod- els fail can be more revealing than where they succeed. While correct outputs may arise from convergent behaviors across in- dependently trained models, consistent and uncommon errors often reflect deeper shared charac- teristics such as architectural biases, training regimes, or optimization strategies (see details in Appendix Dp. Analogous to human behavior, mistakes shared by multiple individuals can imply similar training or background; likewise, models from the same family tend to exhibit stable and structured error patterns, whereas models from different families display more diverse failure distributions. This phenomenon is empirically validated in Figure Ta which shows significantly higher error similarity within families than across unrelated models (see seesioniinn for details).
Cowes Miattel LLable Cammaa ma
0.80
0.55
0:44
0.41
Model Families
0.55
9.55
0.59
0.38
0.38
0:45
-20
0.41
0.38
0.44
0.00
0.40
Gemmss LLatMs Mistrr
Qwnn
Model Famelee
(a)
(b)
Figure 1: (a) Intersection of erroneous data from dif- ferent family models. (b) Error space dimensionality reduction display.
Building on this insight, we introduce ErrorTrace, black-box traceability framework for LLM IP protection. Given set of data samples, extract each model family's is error patterns and embed them into an error space via graph- analysis (illustrated in Figure Iby. A suspect model's error behavior is then projected into this space to infer its likely family affiliation, enabling reliable lineage attribution without internal access or watermarking.
We evaluate our method on five benchmark datasets and 27 models from four popular LLM families. The results show that our method achieves 100% traceability accuracy when the suspect model exists in ErrorTrace's ss model set, and 85.18% when it does not exist, outperforming the baseline by 25. 25.93%. Additionally, our method performs well across various fine-tuning, pruning, and merging strategies. In addition, ErrorTrace shows certain level of resilience when subjected to adversarial attacks.
Contributions. Our contributions are three-fold:
A novel traceability perspective: We introduce a new approach to model traceability by analyzing error space differences across model families, providing new insights into LLM IP protection.
effective and adaptable method: We construct model family error space using unique error patterns and graph-based connections, enabling reliable traceability even among models with similar architectures.
Comprehensive evaluation: We validate our method on 27 models from four families, which range from 3B to 141B, this marks the first instance of achieving such wide range of traceability verification. Our approach outperforms baselines and accurately traces all fine-tuned, pruned, and merged models, demonstrating its effectiveness and robustness.
2
Threat Model and Related Works
2.1
Threat Model
The auditor, either the original model owner third-party detection agency, has full access the original model but only black-box access to the suspect model, i.e., <query, output> pairs.
A suspect model is one modified through unauthorized fine-tuning other actions that infringe on the original model's IP. typically has two characteristics: similarity the original model architecture and knowledge, and concealment of its link to the original model.

---

2.2 Traceability Methods
In the LLM era, traceability methods can be broadly classified into three categories.
Modifying Model Parameters: Researchers have explored embedding watermarks into LLMs ( 25
3:22 13 15:
38 43 46). However, these methods require model modifications, increasing training costs and potentially impacting performance. Besides, cannot be applied to released models without protection unless retrained, limiting its practicality.
16
19
23
31
34
B5
36
Accessing Model Parameters: These methods compare internal parameters, features, or output logits to assess model similarity (7 40 4T: 42). However, their practicality is limited since most major LLMs provide only external APIs.
Query-based Traceability: These methods identify suspect models by analyzing responses to specific queries. TRAP (8) .ProFLingo (II) generate a query and expect specified response, but are limited by high query costs and variable response rates in fine-tuned models. LLMmap (I8) assesses output similarity using empirical queries, but robustness declines against adversarial obfuscation.
2.3 Traceability Granularity
Model-level traceability is widely used in LLM IP protection to identify the specific base model behind a suspect. However, existing black-box methods often lack robustness and are vulnerable to interference (38). For large-scale developers (e.g.. Meta), protecting every individual model is computationally costly. In contrast, model family-level traceability focuses on identifying the broader model family, reducing overhead while aligning with the needs of developers key stakeholders in IP disputes. By leveraging shared features within a family, it remains effective even after fine-tuning pruning. It also avoids dependence fixed queries, offering greater robustness against adaptive attacks and input perturbations.
3 Exploring the Potential of Error Space for Traceability
This section explores the feasibility of using model error spaces for traceability. We present probabilistic model showing that error patterns are statistically distinct across model families and validate this through experimental simulations.
Consider model families G1, G2...., Gn. each containing models Mi, M2...,, Mn and a dataset D {di, d2, with N samples. For model M, and data dj. define the binary random variable Eg as follows:
if M, predicts incorrectly on dj
Egj (
otherwise
Each model's error pattern is represented by the binary vector E; = (Ei...., E.,). Model traceability aims to identify the family G2 based on E,, For mathematical validation, we make the following assumptions:
Number
Rate
LR
DT
RF
0.01
0.00
0.60
0.20
5000
0.05
0.60
0.40
0.80
0.60
0.20
0.40
0.01
0.60
0.20
0.60
1. Independence Assumption: The error events on different data points are independent.
10000
0.05
1.00
0.40
0.80
0.1
0.60
0.40
0.60
2. Family Distinguishability Assumption: Models from different families have significantly differ- ent eтror probabilities on certain data points.
0.01
0.80
0.00
0.20
50000
0.05
1.00
0,20
0.60
0.1
1.00
0.20
0.60
Average
0.69
0.29
0.53
3. Completeness Assumption: The error events for all models on all data points are observable and recorded completely.
Table 1: The tracing success rate for logistic regression (LR), decision trees (DT), and random forests (RF) is evalu- ated across varying data quantities and proportions of erroneous data.
Consider the suspicious model Mr and its error pattern E, on dataset D. Assuming independence, the error event Fay for each data point d, is independent. Thus, the joint probability distribution of E, is the product of the individual error event probabilities. To determine

---

the model family of M,, the problem reduces to comparing the joint probability distribution of E; across different model families.
M. E Gb)
II P PGt,s) (1-R))
(2)
where Pra.. P(Bg = M E Gk) is the error probability of family Gs on data point dz. Based on the assumption of family distinguishability, for model families G. and Gb. there exists a data point j such that PG... # Pay.-- Thus, the joint probability distribution of the error pattern E, differs across model families. The suspicious model can be assigned to family based on these probabilistic differences.
To validate our approach, we run trace- ability experiment using a synthetic dataset across four model families. Each family has defined error-prone data points with varying error probabilities. We use Logis- tie Regression (LR), Decision Trees (DT), and Random Forests (RF) for classification. As shown in Table each family includes four models with a 0.2 error rate difference. LR and RF both achieve accuracy above 0.5, supporting the feasibility of using er- ror patterns for traceability.
0.9
0.8
5.88
0.4
0.10 0.15 0.20 0.25 0.30 0.85 0.40 Eroo probabilty diference
10
15
20
25
30 fenily
35
Qcvently
(a) Error probability
(b) Model family size
Figure 2: (a) The impact of error probability difference on traceability accuracy, (b) The impact of model family size on traceability accuracy.
We further analyze the effect of model count and error rates using 10,000 data points at a 0.1 error rate. Figure shows LR and RF improve with more models and greater error differences, while DT is less consistent. This indicates that more erroncous data enhances traceability, (See/Appendix Kffor details.)
4 Methodology of ErrorTrace
We propose ErrorTrace, a novel black-box traceability method for LLM IP protection. As shown in Figure 3] ErrorTrace consists of three steps: error uniqueness calculation, error space construction, and suspect model inference. This section explains how the classification benchmark constructs the error space; the regression case provided in Appendix
O
[
M.
A1
Data Integration
Step 2: Ertor Space Construction
Step Uniqueness
Family I Gain
Number Number
Family Gain
Family N
Unknown Model
Faror Space
Froor Space Contmetion
Averge Gain Calculation
Originating Family
Step 3: Suspect Model Inference/Tracinn
Figure The overview of error space-based traceability mechanism ErrorTrace.
4.1 Error Uniqueness Calculation
For a model family with N LLMs from the same familis, we first sample a prompt dataset covering various topics approximate the input space. Each prompt in S represents data point We then calculate the error uniqueness of each data point s for the model family T. considering two key aspects:

---

Intra-family Error Uniqueness. This aspect calculates the error uniqueness of data point across all models in family T, represented by U. The simplest approach is calculate the proportion of models in T that make errors on s. However, we observe that nigher-performing models better reflect the family's unique error characteristics; thus, we weight errors by model accuracy, assigning greater weight to errors from high-performing models, as shown in Eqqattin 3
U
Σ Pj jEE
Pr
Here, E denotes the set of LLMs in the target family that generate an error response on data point 8, and p represents the accuracy of the test model on the topic related to
Cross-family Error Diserepancy. This aspect verifies whether the error patterns data point unique model family T. We introduce the cross-family error discrepancy PF as penalty factor to filter out shared error data points across different families. Instead treating all non-target families group, we compute the error rate for each non-target family at and assign higher penalty weights families with higher error rates using squared summation, as shown in Equation
K
PF 1-
P. jEG
Σ P.
(4)
1=1
Let G represent the of model families whose models produce errors data point 8, Pj denote the error rates of s for family j. and K be the number of model families excluding family T.
To better mitigate the impact of error-prone points in non-target families, incorporate model performance into the calculation PF. The error rate P for each non-target family at data point s is defined as:
N Σ
L=
Pj
P
(5)
JEE
4=1
Thus, the error uniqueness of data points in a certain model family T can be expressed as:
= U =P
(6)
4.2 Error Space Construction
In the previous section, we calculated the error uniqueness for all data points in the model family T. Now, we construct the eтror space for T using graph-based approach. Unlike traditional methods like K-Means (17), DBSCAN (5), and GMM (22), we avoid this limitation by assigning each data point weight based on its eтror uniqueness. Edge weights between data points are determined by their Euclidean distances, which we normalize using the Cumulative Distribution Function (CDF) (6) to reduce the impact of extreme values.
F(D = P(D < ((.)))
Where Dii,j) represents the distance between data points i, and F is the CDF function.
Using single CDF mapping overlooks the effect of adjacent distance intervals. To address this, we CDF weighting mechanism. We first sort all edge distances ( in
introduce an interval-base ascending order to obtain a sequence d1 d2 ... dn, where the total number of edges and d, the 7t distance.Ther define Wi as the difference between consecutive sorted distances dj+1 and dj. The new mapping formula is defined as:
i-1 WE(d.) =
Σ 1=1
w
W
(8)
where W E(4) represents the weighted normalized value of the distance dj.
To improve computational efficiency, we use a grid search approach to identify the densest region of edge weights (denoted dense), and the filtered edge set L is defined as:
5

---

L = Edge(WE(dd) S dense)
(9)
where Edge represents the original set of edges, and L is the filtered set of edges.
We construct candidate from the point with the highest error uniqueness, we apply BFS (4) to connect neighboring points. A point is added if its uniqueness exceeds the edge weight; otherwise, it is skipped. This process continues until no further connections can be made. We repeat it for remaining unassigned points to form all candidate error spaces (scereigure 4).
spaces for each model family T using graph-based connections. Starting
To determine the optimal error space for model family T, we compute the effective uniqueness sum for each dataset and select the error space with the highest sum as the optimal error space. This is denoted as:
R(k)= = EWP()) - Σ WE(()
WP(s)
(10)
SES
IET
where is the set of candidate spaces, S represents all points contained in the space, L represents all edges used in the connection.
4.3 Suspect Model Inference
For each error space, we progressively reduce its size by increasing the minimum error uniqueness. For suspect model M,. we track the average error rate change during this contraction. We compute A, the difference between the maximum change and the average of the remaining changes. If A т, the model shows strong alignment with any family and is considered out-of-distribution. If A Z the family with the largest change is identified the source. To ensure fairness, we normalize contraction by fixing the number of remaining points at the start and end. See details in Appendix C
0L56
0.51
0.39
060
0221
Figure 4: Example diagram of error space construction
Experiment
5.1 Experimental Setup
We use 27 models to build error space, ranging from 3 B to 141 B parameters, across families like LLaMa, Qwen, Mistral, and Gemma (see Appendix A
Models:
Datasets: We use five benchmark datasets namely Multinli 33 PAWS 45 CoLA News 44), and CommonsenseeA (24) to find error data. See details in Appendix N
32
AG
Baselines:
Since ErrorTrace is a black-box traceability mechanism, we compare with black-box baselines, i.e., LLMmap (18), TRAP (8), and ProFLingo (1I) which query-based tracing. Due to space limitation, we introduce TRAP (). ProFlingo and the LLMmap (18) processing in
Appendix F
Metrics:
We evaluate traceability using three metrics: accuracy (ACC), cosine distance, and model family discrepancy (MFD). Cosine distance measures the similarity between test model's features and the mean features of each family-a smaller value indicates stronger alignment. This also allows direct comparison with LLMmap, which uses similar approach. MFD detects out-of out-of-distribution (OOD) models by measuring how much a test model's embedding deviates from the structure of known model families.
We consider three scenarios to evaluate ErrorTrace, namely model set known, model sei unknown, and real-world scenario. In the model set known scenario, all base models contribute to constructing the error space, while also serving as test models. model set unknown scenario, designate model the suspect and construct error spaces using the
Traceability Scenario:

---

Model
Gemma
Owen
I.L.map LLaMM
Mistral
ACC
Gemma
Qwen
L.aMMa
Ours
Mistrul
ACC
Hlama7h
11amma0h
0.303
LLama388h
0.405
0.311
0.562
1.370
1.227
0.360
1.680
LLaMa
1.2ma3:70b
0.388
0,089
0.226
0.526
V
1.643
1.170
0,283
1.304
L.amm332:33
0.366
0.426
0.398
0.306
X
1.105
1.241
0.0.9
1.387
115m
0.366
0.509
0.433
0.364
0.928
1.716
0.218
0.581
LLame:1 13b
0.327
0.376
0.386
0.361
1,802
0.689
0.492
0.758
Gemma7b
0.606
0.574
0.469
0.560
0.641
1.667
1.324
1.116
Gemma
Gemma2:9h
0.355
0.363
01.668
0.568
1
0.299
1,315
0.567
2emma2:276
0,217
0.390
0.126
0.579
1.928
0.238
1,065
1.087
QwenTh
0.110
0.375
0.120
0.970
Qwea72b
0.468
0.09%
1.321
Owen32h
0.459
0.538
1.233
Qwen2:7b
0:258
0.5177
1,236
1,521
1.372
Qwen
Owenl4b
0.295
0,301
0.293
0.501
1,166
0.016
1,366
1.397
Owenl.5:13
0.246
0.385
0.403
0.496
1,355
0.399
0.625
0.8114
Dwenl.5::222
0.585
0.469
0.506
0.601
1,390
0.046
1,076
1.407
Qwen1 5:4b
0.429
0.156
0.403
0.304
1.412
0.003
1. 1.663
1,2775
Oweel 5:32b
0.406
0.262
0.28]
0.493
1.024
0.136
1.482
0.987
Qwen 1:5:14
0.474
0.482
0.539
0.117
1.457
0.038
1,240
1,001
Mixural
0.434
0.1187
0.183
0,142
0.895
0.577
1,747
0,401
Mistral7b
0.370
0.471
0,343
0.617
0.710
0.679
1,564
0.335
Mistral
Mistral-small
0.457
0.491
0.483
0.085
1.026
0.930
1,335
0.055
nemraal.nemo
0.545
0.537
0.057
0.432
Mastral-tagge
0.196
1,303
0.306
0.275
0.351
1.818
0.819
0.59
0.85
Table 2: Cosine similarity in model set unknown scenario
remaining 26 models, this is more challenging scenario. In the real-world scenario, the suspect model is modified through a series of fine-tuning, pruning, and merging methods.
Prompt Template: To minimize information leakage, restrict prompt wording, limiting the suspect model's association with its family. All prompt templates are provided (Appendix L.
5.2 Effectiveness
This section evaluates our method in three key areas: error space construction, trace- ability of models, and comparison of space search methods. We define the error space contraction process, setting start and end numbers at 120 and 600 points (the rea- sons for selection are shown in Figure 6b) and Appendix E).
Motel Families even matra latr gonma
0.04
0.08
mégrol
0.6
0.02
genim
0.01
0.00
0.4
0.00
-0.01
-0.02
0.00
0.04
-0.03
Gamma
Micial
Gawn
loma Model Families
Group
First, our method effectively captures each model's error space, clearly separating dif- ferent families. As shown in Figure 5al error data across families have near-zero overlap, confirming the isolation of family-specifei patterns. In the known model set scenario, all 27 test models are correctly traced to their families. FFiurre 5b further shows that each family has positive alignment within its own space and negative alignment elsewhere. See See Appendix for details.
(a)
(b)
Figure 5: (a) Model family error space error intersec-
tion; (b) Average change on different error spaces.
177
Second, our method performs well in the more challenging unknown model set scenario. Each model tested as a suspect using the remaining 26 to build error spaces. As shown in Table 2 our method achieves 85. 18%% accuracy (23/27), significantly outperforming LLMmap's 59.25% (16/27).
3gs
and-000
21
Accuracy
LLMrig Acousccy
50
2
5
103 100 1200 1300 1433 1030 1000 1706 1930
In this scenario, four models were misidentified. Two were from the Gemma family, likely due to having only two models left to construct a reli- able error space. LLMmap also struggled with these. The other two were large Mistral mod- els (Mixtral:8x22B and Mistral-large), whose
(a) LLMmap
(b) ErrorTrace
Figure 6: (a) The effect of query numbers on
LLMmap: (b) Comparison of the highest accuracy of ErrorTrace and LLMmap.
scale distribution is uneven (see details in section 7

---

Furthermore, our method outperforms tra- ditional space construction techniques (e.g.. K-Means, DBSCAN, GMM) in identifying effective error spaces for model traceability. Using grid search for fair comparison, the best baseline achieves only 66.66% accuracy, well below our 85.18% TTble 3). We also evaluate robustness to contraction intervals. Figure 6a shows LLMmap peaks at 59.25% when using all queries. In contrast, our method maintains high accuracy across varying intervals and consistently outperforms LLMmap (Figure 6b). These results underscore the superiority of our approach in error space identification and model traceability.
Methods K-Means
DBSCAN
GMM
Ours
Accuracy
0.4074
0.4444
0.6666
0.8518
Table 3: Accuracy of different construction methods.
5.3 Impact of Dataset Size
We also test how dataset size affects ErrorTrace by ran- domly sampling from the original error set (Table 4). Per- formance improves with more data, and even at 20K sam- ples, ErrorTrace significantly exceeds LLMmap's 's best result.
Size
20K
30K
Accuracy
0.7037
0.7778
0.8148
0.8518
Table 4: Accuracy at different dataset
sizes.
5.4 Out-Of-Distribution Model Detection
We designate LLaMA as the aut-of-distribution family and treat Gemma, Qwen, and Mistral as in-distribution. Using an optimal threshold 7 = 0.025, we compute model family discrepancy for each model. For -fitt--diitiiuutionn evaluation, we test three unseen models: Falcon3-7B and Falcon3- 10B (different scales within the Falcon family), and SeepSeek-R1-7B (a different family). (able 5 shows that all these models have discrepancies below т, confirming they are correctly identified as out-of-distribution and not part of any known family. This indicates that we can correctly identify these aut-of-distribution models as not belonging to any of the in-distribution families.
Model
Gemma
Qwen
LLaMa
Mistral
MFD
ACC
Falcon3-7B
0.0014
0.0019
0.0014
0.0017
0.0124
Falcon3- 10B
-0.1112
-0.0099
-0.0199
-0.0164
0.0167
DeepSeck-RR 1-BB
0.0067
0.0073
-0.0011
0.0012
0.0151
Table Out-of-distribution model detection.
1. To visualize the MDF, this table presents the average change in the suspect model's error rate within each model family's error space, rather than cosine distance.
5.5 Robustness
We now evaluate the robustness un- der the real-world scenario. We select LLaMa7B and Mistral7B for fine-tuning, they have several fine-tuned versions, and also test Qwen and Gemma for different fine- tuning purposes. Note that all fine- tuned and trimmed models are down- loaded from HuggingFace, See Appendix Appendix M for details.
Method
Genunaa
Qwen
LLama
mistral openoeca
0.360
0.347
0.368
0.413
micralite
0.600
0.609
0.89]
0.682
L. L..Masa
Mistral T8-lnstrct
0.454
0.581
0.471
0.687
openhermes
0.260
0.415
0.250
0.510
zephyr
0,386
0.431
0.997
mieml--punoraa
1.634
0.792
1.282
0.332
misrallite
1.538
1.337
0.957
0.625
Ourra
Mistral 78-Insimet
1.404
0.906
1,257
0.203
opeoleermss 1.535
0.500
1.199
0.334
zephor
1,255
1.142
0.886
0.874
Table 6: Comparison of different fine-tuned Mistral7B be- tween ErrorTrace and LLMmap.
Variants of a Sin-
Model
Ratio
Strategy
Gemma
Qwen
LLLMMa
Mistral
ACC
gle Model. We evaluate robust- ness on fine-tuned variants of a model. As shown in Table 6 our method correctly identifies all five fine-tuned Mistral7B ver- sions, while LLMmap misclassi- fies them. Details on LLaMa7B fine-tuning robustness are in Ap-
vicura-5 Sh-ppl
201
Shortsnnd-pp
0-450
0.570
0.467
0.631
Vicuma-3 7o-ppl
Shoetennd-gel
0.47
vicura ppl
gel
LLMmmp
2015
Shotteed-ayyaa
0.438
0.420
0651
Sheared- 1 2.77-277
60%
Sheamd
0.601
0.629
0.583
0.64
Sheared LLMMM 1,38
Sheured
0,683
0,638
0.11
0.675
vicura-5 50-
201
Shoetsued-pl
1.146
0.336
0,807
1.796
vicums ppl
45%
Shoetene gpl
0.855
0.484
vicuna-2.70 2.70 pol
goll
Shortmol-aayle
0.620
Sheared LLMA 2.7B
60%
Sheured
1.440
0.853
0.629
1.772
Sheared LLaMA 1,38
80%
Sheured
0.833
1.750
0.131
1.145
Table 7: Traceability robustness experiment of pruning model.
pendix H.
8

---

We then evaluate robustness against differ- ent pruning strategies and rates. Table shows that ErrorTrace consistently iden- tifies the model family, even at 60% and 80% pruning rates, demonstrating strong robustness across pruning methods. We also test ErrorTrace various model merging strategies for Mistral-7B V0.1 and V0.2 Table confirms our method successfully tracks the model's origin.
Strategy
Gemma
Qwen
L.LaMa
Mistral
ACC
PPL
1.398
0.506
0.769
0.487
Dare-Ties
1.586
1.185
0.652
0.356
Dare-Task
1.153
0.472
1,565
0.341
Table 8: Traceability robustness experiment of merging model.
Fine- Tuned Variants Within Model Family. We further evaluate a harder scenario: fine-tuned variants within the Qwen family. [Talle 9 shows our method correctly identifies all five models, while L.MMmap misclassifies three, including Qwen1.5:144 1.:1444msttakenly labeled as LLaMa with only 0.001 cosine distance difference. This highlights LLMmap's struggle to separate families, while our method effectively distin- guishes error spaces with differences above significantly outperforming LLMmap.
Method
Model
Gemma
Qwen LLama Mistral ACC
Qwen
0.336
0.371
Qwea-14B-C22 1eas-14B-hhaa
0.483
0.303
0.302
0,530
LLMmap
Qwen2 math
0.548
0.261
0.487
0.538
Qwen2. S-onder:7B
0.536
0.594
0.692
5-ode::10
0.474
0.317
0150
0,542
Qwen-7B-CCN
1.719
0.634
1.549
0.856
Qucs-14R-Chai
1.982
0.378
0.944
1.076
Ourr
Qwen2 muth
0.865
0.618
1.965
0.687
Qwen2.5 coder:7B
1.403
0.247
1.357
0.564
S-oadar:148
0.151
Table 9:
Performance comparison of different ver- sions of Qwen family models.
Fine-Tuned Variants of a Small Model Family. Recall that in Table 2] our method struggled with the Gemma family due only two base models available for error space construction, limiting generalization. Using three base models and testing on their fine-tuned variants, Table 14 shows our method successfully traces them, demonstrating significant improvement by increasing base models from 2 to 3.
5.6 Model Granularity Traceability
Although ErrorTrace targets model family granularity, it also covers model-level traceability when a family has a single base model. We evaluate single-model families like Ilama7b, llamal 13b, and llama3-8b, testing on variants such as vicuna:7b, orca-mini:7b, and codellama derived from llama7b. Table 10 shows our method successfully traces orca-mini:76 and codellama back Ilama7b.
Ilama7h
Ilama
Ilama70b
llama3-8b
llama3-70b
1lama3.1-8
1lama3.2-3b
ACC
vicuna:7b 0.496
0.608
0.644
1.236
0.339
0.717
1.554
orca-mini:7b
0.496
0.632
0.722
0.581
0.869
0.564
0.609
codellama
0.369
0.514
1.161
0.994
1.213
1.075
0.462
Table 10: Model Granularity Traceability
ErrorTrace against Adversarial Attacks
A potential threat is that malicious user gains access to the samples used for build- ing the ErrorTrace error space and exploits them to attack the suspect model. Such at-
Top-K
100
200
500
800
1200
1500
ACC
0.81
0.78
0.70
0.67
0.67
0.44
Table 11: Attacker correctly repairs the sample
tacks are difficult for all black-box attribu- points in the space. tion methods. In contrast, ErrorTrace leverages intrinsic model characteristics (finger- prints rather than prompts), which improves adversarial robustness. More importantly, since the error space is built from the base dataset, adversarial fine-tuning against ErrorTrace inevitably harms the model's overall performance, raising the cost and risk of attack.
To test robustness, we conducted simulated ad- versarial attacks. First, assuming a strong at- tacker who can "perfectly repair" the Top-K samples with highest error uniqueness, results (Table TTp show attribution accuracy remains 0.70 after repairing T00-500, about 0.67 at K=1200, and only drops sharply to 0.44 at K=1500.
Top-K 100 500
1000
1500
2000
3000
ACC
0.81
0.74
0.70
0.67
0.67
0.67
Table 12: Attacker error repairs the sample points
in the space.

---

We also tested an extreme case where the attacker forces the Top-K samples with low- est enror uniqueness be misclassified. Results Table 12 show that even after al- tering 3000 samples, ErrorTrace still maintains 0.67 accuracy. Notably, further en- larging K does not cause a sudden collapse, suggesting this strategy has limited effect. Finally, under more realistic conditions, we fine-tuned LLaMA-7B on the top 5% and 10% highest-uniquenes, samples. As shown in Ta-
Gemma
Qwen
1.256
0.778
0,390
1823
Top-10%
1,081
1.011
0.578
1998
ble 13, ErrorTrace still achieves reliable attri- bution. Overall, these experiments confirm that ErrorTrace is robust and practical under diverse adversarial scenarios.
Table 13: Attacker uses adversarial fine-tuning with different sample ratios.
7 Limitations
7.1 Limitations of Model Family Scale Differences
Although our method effectively protects large model IP, it has some limits when model family scales differ.
Gemma
Qwen
I.aaMa
ACC
gesamnn instruct
0.407
1,038
codogemma
0.046
1.352
1,033
gemman
0.542
9b chinese chat
0.236
0.909
1.648
The first arises when family sizes vary widely. As shown in Table 14, having only three models makes detection harder. We suggest adding fine- tuned variants or generating more error data via randomization (see (peeeddxx G).
gemmal 2:27 2:77-iistra
0.702
0.811
0,707
1.838
Table 14: Traceability effects of different Gemma fine-tuning models
Another occurs when models in the same family differ greatly in scale. For instance, ErrorTrace misidentified Mixtral:8x22B. A practical is to build separate error spaces for different parameter sizes, which improves fingerprint accuracy.
7.2 Limitations of Error Data Collection Cost
Another key limitation is the time cost of eтrоr data collection. For a single model, collecting about 50K errors takes 24 GPU hours. However, this is a one-time cost, since the collected data can be reused. Notably, building the full error space for 27 models across four families only requires GPU hours. More importantly, auditing a suspect model needs just 4 GPU hours (10K data). Thus, once the error space is prepared, researchers audit suspect models efficiently with relatively low overhead. Compared with overall model development costs, this cost is acceptable (see Appendix for details).
To further reduce inference costs during audit- ing, adopt filtering-basee based subsampling strat- egy. Within each error uniqueness interval (e.g..
Rate 10%
30%
50%
70 70%
ACC
0.81
0.67
0.67
0.59
[0.4. 0.5)), part of the test data is filtered to the inference load. As shown in Table 15] even under aggressive 70% filtering strategy, the
Table 15: Subsampling with Different Filtering
Ratios
accuracy remains 0.59, indicating that the method effectively lowers inference costs while maintaining reasonable auditing performance.
Conclusion
In this work, we propose ErrorTrace, a novel black-box traceability method for large language models (LLMs) based on error pattern analysis. By leveraging the inherent consistency of error distributions within model families, our approach effectively constructs error spaces to identify the family of a suspect model. Extensive evaluations on five benchmark datasets and 27 models from four LLM families demonstrate that ErrorTrace achieves high traceability accuracy, significantly outperforming existing baselines. Furthermore, our method remains robust against fine-tuning, pruning and merging, highlighting its adaptability to real-world model modifications.
10

---

Acknowledgments
This work was supported by National Natural Science Foundation of China under Grant(No. 62372268), Key R&D Program of Shandong Province, China (No. 2024CXGC010144 14, No. 2025CXPT085), Shandong Provincial Natural Science Foundation, China (No. ZR2022LZH013, No. ZR202ILZH007).
References
[1] T. B. Brown, Mann, N. Ryder, M. Subbiah, and J. Kaplan. Language models are few-shot learners. In NeurIPS, 2020.
[2] J. Cai, Yu, Y. Shao, et al. Utf: Undertrained tokens fingerprints novel approach to Ilm identification. arXiv preprint orXiv:2410.12311 2024.
[3] T. Cong. D. Ran, Z. Liu, et al. Have you merged my model? on the robustness of large language model ip protection methods against model merging. In Proceedings of the 1st ACM Workshop on Large A/ Systems and Models with Privacy and Safety Analysis, pages 69-76, 2023, [4] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. The MIT
Press, 3rd edition, 2009.
[5] M. Ester, H.P. Kriegel, J. Sander, and X. Xu. A density-basse based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), pages 226-231, 1996.
[6] W. Feller. An Introduction Probability Theory and Its Applications, volume 1. Wiley, 1968.
[7] C. Goddard, S. Siriwardhana, M. Ehghaghi, et al. Arcee's mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477-485, 2024.
[8] M. Gubri, D. Ulmer, H. Lee, et al. Trap: Targeted random adversarial prompt honeypot for black-box identification. arXiv preprint, 2024.
[9] Huang, P. Parthasarathi, M. Rezagholizadeh, et al. Do robot snakes dream like electric sheep? investigating the effects of architectural inductive biases on hallucination. arXiv preprint arXiv:2410.1 17410..777. 2024.
[10] A. Q. Jiang. A. Sablayrolles, A. Mensch, al. Mistral 7b. arXiv preprint, arXiv:2310.0682. 2023.
[11] H. Jin, C. S. Shi, et al. Proflingo: A fingerprinting-based intellectual property protection scheme for large language models. In 2024 IEEE Conference on Communications and Network Security (CNS), pages 1-9. IEEE, 2024.
[12] Kirchenbauer, J. Geiping, Y. Wen, et al. watermark for large language models. In International Conference on Machine Learning. pages 17061-17084 PMLR, 2023.
[13] C. Kong, J. Chen, S. Tan, al. Copyright protection for large language model eaas via unforgeable backdoor watermarking. In International Conference on Pattern Recognition, pages 1-15. Springer, Cham, 2025.
[14] A. Lewis, M. White, J. Liu, et al. Winning big with small models: Knowledge distillation vs.
self-training for reducing hallucination in qa agents. arXiv preprint arXiv:2502. 2544:55502.9955.. 2025.
[15] L. Li, B. Jiang, P. Wang, et al. Watermarking Ilms with weight quantization. arXiv preprint
arXiv:2310.112 12377 2023.
[16] S. Li, L. Yao, J. Gao, et al. Double-i watermark: Protecting model copyright for Ilm fine-tuning. arXiv preprint arXiv:2402.1 3,8843,402. 2024.
[17] J. MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium Mathematical Statistics and Probability.
volume 1, pages 281-297, 1967.
[18] D. Pasquini, E. M. Kornaropoulos, and G. Ateniese. Llmmap: Fingerprinting for large language models. arXiv preprint arXiv:2407.1588
2024.
[19] W. Peng, J. Yi, Wu, et al. Are you copying my model? protecting the copyright of large language models for caas via backdoor watermark. arXiv preprint arXiv:2305. 2023. [20] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre- training. Technical report, OpenAI, 2018.
[21] Y. Refael, A. Hakim, L. Greenberg, et al. Slip: Securing 1lms ip using weights decomposition.
arXiv preprint arXiv:24077 2024.
[22] D. A. Reynolds. Gaussian mixture models. In Encyclopedia of Biometrics, pages 659-663.
Springer, 2009.
[23] M. Russinovich and A. Salem. Hey, that's my model! introducing chain & hash, an Ulm fingerprinting technique. arXiv preprint arXiv:2407.1088) 2024.

---

[24] A. Talmor, J. Herzig, and J. Berant. Commonsenseqa: A question answering challenge for commonsense knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019), pages 4147-4157, 2019.
[25] R. Tang, Y. N. Chuang, X. Cai, et al. Secure your model: An effective key prompt protection mechanism for large language models. In Findings of the Association for Computational
Linguistics: NAACI. 2024, pages 4061-4073, 2024.
[26] G. Team, T. Mesnard, C. Hardin, al. Gemma: Open models based on gemini research and technology. arXiv preprint, arXiv:2403.0822 2024.
[27] H. Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023.
arXiv:2307.0922
[28] H. Touvron, N. Sarlán, and P. Dufter. Llama: Open and efficient foundation language models.
arXiv preprint, arXiv:2302. 13977,, 2023.
[29] F. Wan. X. Huang. L. Cui, et al. Mitigating hallucinations of large language models via
knowledge consistent alignment. CoRR, 2024.
[30] T. Wang. X. Jiao, Y. Zhu, et al. Adaptive activation steering: A tuning-free I1m truthfulness
improvement method for diverse hallucinations categories. In Proceedings of the ACM on Web
Conference 2025, pages 2562-2578, 2025.
[31] Z. Wang, B. Wu, J. Deng, et al. Espew: Robust copyright protection for I1m- tin-based caas via embedding--peccifcc watermark. arXiv preprint arXiv:2410. 17552, 2024.
[32] A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2019), pages 579-589, 2019.
[33] A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018), pages 632-642, 2018.
[34] J. Xu, F. Wang, M. D. Ma, et al. Instructional fingerprinting of large language models. arXiv preprint arXiv:2401. 12255, 2024.
[35] Z. Xu, W. Xing, Z. Wang, et al. Fp-vec: Fingerprinting large language models via efficient
vector addition. arXiv preprint 1XXi:2449:088444 2024.
[36] S. Yamabe, T. Takahashi, F. Waseda, et al. Mergeprint: Robust fingerprinting against merging large language models. arXiv preprini arXiv:2410.088 086044 2024.
[37] Z. Yang and H. Wu. A fingerprint for large language models. arXiv preprint arXiv:2407.01225.
2024.
[38] Z. Yang, Y. Wu, Y. Shen, et al. The challenge of identifying the origin of black-box large language models. arXiv preprint, 2025.
[39] Y. Yao, T. Yu, A. Zhang, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint.
arXiv:2408.0 022400..1880 800, 2024.
[40] B. Zeng, L. Wang. Y. Hu, et al. Huref: Human-readable fingerprint for large language models.
In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2023. [41] C. Zhang. H. Foerster, R. D. Mullins, et al. Hardware and software platform inference. arXiv
preprint arXiv:2411.055 11.019977, 2024.
[42] J. Zhang. D. Liu, C. Qian, et al. Reef: Representation encoding fingerprints for large language models. arXiv preprint ar XrX::2410.142755 2024.
[43] R. Zhang and F. Koushanfar. Emmark: Robust watermarks for ip protection of embedded quantized large language models. In Proceedings of the 61 ACM/IEEE Design Automation Conference, pages 1-6, 2024.
[44] X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems (NeurIPS 2015 5). pages 649-657, 2015.
[45] Y. Zhang. Y. Zhao, and Y. LeCun. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL
2019), pages 15567, 2019.
[46] X. Zhao, Y. X. Wang, and L. Li. Protecting language generation models via invisible wa- termarking. In International Conference on Machine Learning, pages 42187-42199. PMLR,
2023.
[47] H. Zhou et al. A survey of large language models in medicine: Progress, application, and challenge, arXiv preprint, 1rXiv:2311.05112 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our main claim are summarized in section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of our work in |ection 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

We provide simple proofs and experimental verifications in section 3 and in Appendix K
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the parameters required for the experiments in seccion and the corresponding prompt templates are provided in Appendix L
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We used publicty available benchmark datasets, e.g., 45 and an anonymous repository of the code is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
oolicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccppblle pwwppppguuuiieeeeep Sudi:///uuuieee/deeuuuubbiii SroodeSubmissi. ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please |section 5) and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix O
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not foresee any high risk for misuse of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The dataset we use is publicly available benchmark datasets, and the models are open-source models, see details [Appendix A Appendix M. (Appendix N
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Once the blind review period finished, we will release our code base with included readme files.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is the subject of our study, and in our methodology, we do not use LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
19

---

A Base Model
In this section, we present the base models utilized this study along with their fundamental information, including their respective families and model scales. A total of 27 base models were employed, spanning across four distinct model families. The Gemma family comprises the fewest models, with only three members, whereas the Qwen family is the largest, consisting of eleven models. These models vary in parameter scales from 3 billion to 141 billion, marking the first validation with such a large scale in LLM IP protection, as detailed in Table 16 ollama
All base models are derived from
#
Model
Vender
Number of parameters
Iaama7b
Meta
7B
2
Iaama 13b
Meta
13B
3
10b
Meta Al
70B
1lama3:8b
Meta
8B
5
7am
Meta
70B
6
1lama3.1:88
Meta
8B
7
llama3.2:3b
Meta
3B
8
gemma7b
Google
7B
gemma2.9b
Google
9B
10
gemma2:276
Google
27B
11
qwen4b
Alibaba
4B
12
qwen7b
Alibaba
7B
13
g4en14b
Alibaba
14B
14
qwen32h
Alibaba
32B
15
gwen72b
Alibaba
72B
16
qwen2:7b
Alibaba
7B
17
qwen2.3:3b
Alibaba
3B
18
3wen2.3:7b
Alibaba
7B
19
qwen2.5:
14b
Alibaba
14B
20
5:322
Alibaba
32B
21
qwen2.5:72b
Alibaba
72B
22
mistral7h
Mistral AT
7B
23
mistral- nemo
Mistral Al
12B
24
mistral-sine -mmlll
Mistral AI
22BB
25
mistral - llrgee
Mistral Al
123B
26
mixtral
Mistral Al
39B
27
8x22 Mistral Al
141B
Table 16: Publisher of the hasic model and display of its parameter scale.
B ErrorTrace In Regression Case
For regression tasks, ErrorTrace only requires adjusting the computation of error uniqueness from Section 4.1, replacing discrete values with continuous ones. Using BLEU as an example, we illustrate the principle in detail. We first normalize and preprocess continuous values reflect each sample's 's eтror tendency- the higher the value, the higher the risk of error. Since higher BLEU means better performance, we redefine B BLEU. Accordingly, the intra-family consistency score [Equation 3) extends to:
U Σ B JEF
Σ
B
(11)
where is the model family under evaluation, and N is the number of models. Similarly, the inter-family discrepancy (Equation 4and ??) extends to:
K
PF
Σ
2
Σ
P
(12)
JEG
i=
httts://ollamm.
20

---

Pj === EB
(13)
JEB
where G is the set of all families other than F,, K is the set of all families, and ps is the proportion of values in family relative to the total across all models.
C Reasoning Details
C.1 Details of Out-Of-Distribution Detection
When model family A 's error space shrinking, The average error uniqueness of data in the error space will increase. According to the Error Uniqueness Calculation in ssubecctionn 4.1] this increase is mainly due the excitation of Intra-family Error Uniqueness, so models within family A are more likely to error on data point, and depending on the limitation of Cross family Error Discrepancy, it unlikely that models not in A will do so.
Theoretical Basis:
For suspect model a in model family A, intra-family Error Uniqueness increases its average gain the error space of A, and inter-family Error discrepancy limits its average gain in the other families's 's error spaces, leading to larger model family discrepancy. For suspect model not belonging to any model family, lacks both uniqueness and discrepancy constraints. So its performance similar in different eπrоr spaces, resulting in smaller model family discrepancy. We define a threshold т. If b's model family discrepancy exceeds т, it is classified as an intra-family oohersss,, it is considered an out-of-distributie model.
Detection Logic:
C.2 Details of Changes in Average Error Rate
Based on the description of Intra-family Error Uniqueness and Cross-family Error Discrepancy in subsection 4.1 and the explanation of the suspicion inference process in |subsection 4.3 in the process of shrinking the error space of the model family A, the data points that are more inclined to make errors in this model family will be retained, and the data points in other data points in the model family that are more inclined to make mistakes. As result, for model E A, the change in the error rate in the eтror space to which it belongs will be larger than that for model a E A. Calculating the average change in the eπror rate of the suspected model b in each error space during the contraction process to find the error space with the largest change in the error rate, and then to find the model family it belongs
Since the experimental error dataset comprises multiple foundational datasets, and different models exhibit varying performance across tasks within these foundational datasets, the proportion of each foundational dataset within a model family's 's error space also varies. Consequently, it is insufficient to differentiate model families based solely on simple error rates. Therefore, we opt to represent the distinction by the difference between the suspect model's actual error rate er within the error space and the theoretical eиror rate ter of that error space.
Assume that the error space F.S, consists of foundational datasets, and the suspect model M has theoretical error rates der,, der2. ..., derx on these datasets. Here, the theoretical error rates are obtained by testing the suspect model on fixed number of data points (1000 in our experiments) from each foundational dataset. The proportion of each foundational dataset within the error space ES; is denoted by D1, ..., Dx. Therefore, the theoretical error rate ter of the error space ES; can be expressed as:
K
ter
Dj x der,
(14)
Regarding the calculation of enro data error uniqueness, we choose to represent the actual error rate er by the ratio of the sum of the suspect model M's uniqueness coefficients for the error data within the error space ES, to the sum of all uniqueness coefficients for the error data within that error space.
21

---

Mathematically, this is expressed as:
EiENDNE MP, er EiEES MP;
(15)
Where M M. denotes the set of data points on which the suspect model M makes emrors within the error space M. represents the uniqueness coefficient of data point
Therefore, the change of the error space is defined as the difference between the actual error rate and the theoretical error rate:
Gain
ter
(16)
D Origins of Error Patterns and Their Root Causes for Traceability
Model error differences reflect variations in "hallucination" tendencies, often arising from architec- ture (2), training data (14), and alignment methods (29) (30). Models within the same family share systematic design, expert knowledge, and high-quality data, leading to similar architectures and behaviors; for example. LLaMa3 follows LLaMa2's framework. In contrast, cross-family similarity is lower due to different development strategies, c.g., Mistral uses Sliding Window Attention (SWA) while DeepSeek uses Multi- Hulti-Head Latent Attention (MLA).
E Parameter Selection
First, while more eπrоr data may seem helpful, large initial sets (e.g.. 1600 points) introduce low- uniqueness samples into the error space. Models like Gemma-7B, which rely more on high-quality errors (e.g., min Error Uniqueness = 0.3 in Gemma/Qwen when >1600 points), can be misled by such noise during traceability.
Second, we observed that once the eπrоr space contracts past a certain ratio(e.g., <500 samples), remaining points have consistently high Error Uniqueness. The target model's error rate in its family nears saturation, while others still improve. Contracting beyond this point hurts traceability.
So, starting with moderate size 11000-1000 points) offers better trade-off: enough diversity capture distinct patterns, while limiting noise. As contraction proceeds, Error Uniqueness rises from 0.4 to 0.6 across the final 600 points. This stabilizes the target model's error signature within its family, while other models continue to diverge - improving separation.
F Additional Notes and Experiments on Baseline
F.1
LLMmap
Since LLMmap traces models based on output features, we adapt it to our objectives. We first extract and average features from models within the same series create representative set, then calculate the cosine similarity between the suspect model's 's features and this set to determine its family.
F.2 ProFlingo
Due the high computational cost associated with query construction in ProFLingo, we have selected the pre-constructed Mistal7B queries provided in the paper. Additionally, we employed the T TRR metric, as used in the paper, evaluate Mistral itself, fine-tuned models, models from different families, and models within the same family. TRR can be defined as:
n TRR
(17)
N
Here, represents the total number of queries, and n refers to the number of queries that successfully respond with the specified content.
Table 17 presents the performance of various models under queries constructed for Mistral7B. The results indicate that the ProFLingo method demonstrates certain degree of provenance effectiveness
22

---

Suspect Model
Ground Truth
TRR
Difference
mistral:7b
Itself
0.62
0.581
openhermes
Positive
0.32
0.261
mistral-openorea
Positive
0.16
0.101
mistrallite
Positive
0.10
0.041
Dastral7B-Innstuuc--o0
Positive
0.08
0.02
zephyr
Positive
0.04
-0.02.
LLaMa2:7B
Negative
0.06
0.00
LLaMa3.2:3B
Negative
0.06
0.00
qwen2.5:3b
Negutive
0.04
-0.02.
gemma:7b
Negative
0.04
-0.02.
qwen:7b
Negative
0.02
-0.04.
Mistral-7B-vll
Related
0.32
0.281
Mistral-7B-v0.2
Related
0.28
0.241
Mistral-nemo
Related
0.04
-0.02.
Mistral-small
Related
0.02
-0.04]
Mixtral
Related
0.02
-0.04]
Table 17: ProFLingo's performance different models
IItself stands for the original model itself, Positive stands for the model that fine-tuned from the original model, Negative stands for the model that is unrelated to the original model, and Related stands for the model that is of the same family the original model.
2. Difference represents the difference between the TRR and the highest TRR in the Negative, where the TRR exceeding more than twice highest TRR the Negative shown red, and the rest shown green for fine-tuned models. However, the TRR values fluctuate significantly across different fine-tuned models, making the decision boundary difficult to determine. Specifically, the TRR differences between MistralLite and Mistral- 7B- 7B-Instruct-vdl models and LLaMa2:7B are less than 0.05, while Zephyr's TRR is even lower than that of unrelated models. For related models within the same family, the ProFLingo method performs well on different versions of the Mistral7B model but shows low relevance when applied to other models in the same family. ProFLingo operates primarily at the model granularity traceability. The high TRR between Mistral-7B-v0.3 and Mistral-7B-v0.2 suggests potential confusion when handling different versions of highly related models. However, when applied to model family granularity traceability, the low TRR values of models such as Mistral-Nemo make it challenging for the method to effectively distinguish models within the same family.
F.3 TRAP
Due to the time-consuming nature of the query construction the TRAP, chose the query for Ilama7b that has already been constructed in the paper. In addition, we also choose the TPR as the base metric test the Ilama7b model itself, its fine-tuned model, the same-family model, and the irrelevance model. Different from the definition of TPR in ProFLingo, we follow the definition of TPR in the TRAP paper, in which single round with one correct response is recognized as correct, and the final TPR is obtained through multiple rounds, which is expressed as follows.
TPR
(18)
N
Where, n represents the number of validation rounds with successful responses, N represents the total number of validation rounds. We conducted 50 rounds of experimental validation on each of the 100 queries of length 3,4,5 mentioned in TRAP.
Table presents the traceability performance of TRAP on various models. It demonstrates strong performance on the baseline model, but significant TPR fluctuations occur in fine-tuned variants. Notably, the Vicuna:7B model reaches TPR of 0.86 at length 3, and Llama-Chinese maintains TPR above 0.5 across all lengths. contrast, TPR values for other fine-tuned models decrease sharply, with the Nous-Herms model peaking at 0. The Negative and Related models outperform Heus- in most cases. These findings highlight that TRAP faces considerable challenges in fine-tuning scenarios, making it difficult to consistently identify fine-tuned models and their TPRs.
G
Explanation of the Limit on the Number of Models in the Family
The small number of models the family a limitation of this paper, and the error space of the constructed model family is not accurate the error space constructed by the model family with
23

---

Suspect Model
Ground Truth
TRR-3
TPR-4
TPR-5
Ilama:7b
Itself
1.00
0.98
1,00
vicuna:7b
Positive
0.26
0.20
0.20
orca-mini:7b
Positive
0,86
0.32
0.52
codellama
Positive
0.24
0.12
0.08
Ilama-chinese
Positive
0.90
0.84
0.60
nous-hermes
Positive
0.10
0.06
0.04
mistral:7b
Negative
0.04
0.10
0.16
gemma:7b
Negative
0.06
0.16
0.14
qwen:4b
Negative
0.20
0.18
0.24
qwen:7b
Negative
0.14
0.08
0.08
qwcn2.5:3b
Negative
0.00
0.02
0.00
Ilama332.33
Related
0.00
0.00
0.08
llama3.2:1
Related
0.00
0.04
0.16
Ilama3. 1.8b
Related
0.00
0.28
0.12
llama3388
Related
0.02
0.08
0.16
Ilama2:136
Related
0.08
0.06
0.12
Table 18: TRAP's performance on different models
1. For the Positive category model, we choose to bold the lowest value for each discriminant feature, and for the other category models, we use red font for the values higher than the bolded values.
large number of models in the family, which is also illustrated by the traceability effect of the Gemma family in Table 2 However, this does not prove that the method in this paper cannot be adapted to the case where the number of models in the family small. The unknown scenario of the model represented in Table is more challenging scenario, where there is obvious parent-child relationship between the test model and the training model, but rather a brother model relationship (helonging to the same model family), and the model evolves from its family model through structural optimization and retraining during the test, which is unacceptable to real malicious user. In real- world scenarios, Table 14 also demonstrates that our method is able to achieve fine-tuned model traceability even when the number of models in the family is small.
We also find that the information about the errors made in the model family can be enhanced by fine-tuning the base model and by utilizing multiple samples of the model's stochasticity, as well overcoming the limitation of the small number within the model family:To verify this, we simulate a single-model family by excluding all Gemma models except Gemma-7B, We then sample error patterns, The results are shown
Model
Gemma
Qwen
LLaMa
Mistral ACC
codegemma[1]
0.0048
-0.0107
-0.0119
-0.0038
Table 19: Single Model Validation
once and Gemma-7B multiple times extract
H Llama Fine-Tuning Experiments
The LLaMa7B fine-tuned models are shown in Table 20 Our method suc- cessfully identified the model family among five fine-tuned variants, while LLMmap performed poorly on two models: llama2-chinese and nous- hermes. When comparing the total co- sine distance differences across meth- ods, our method's 's sum exceeds 1, indi- cating a significantly higher similarity within the same model family com-
Method
Model
Gemma
Qwen
LLama
Mistral
vicuna:7h
0.396
0.414
0.346
0.5107
orca mini:7b
0.329
0.326
0.298
0.521
LLMmap
codellama
0.466
0.437
0.366
0.388
chhnee-cinese
nous hernes
0.547
0.637
0.548
0.701
vicuna:
1,583
1,275
0,893
1,252
wcaammin::7b
0.539
1.594
0.214
0.787
1.317
laama2. chinese
1.715
1.173
0.502
1.282
nous-hermes
1.296
1.121
0.922
1.698
Table 20: Performance comparison of different fine-tuning models of LLaMa7B on method and LLMmap method pared others. contrast, M.L.map's sum below 1, with a maximum of 0.63, suggesting weaker distinctiveness between suspected models and other families.

---

Table 21: Model traning set known scenario ' cosine distance and MFD.
Group
Model
Gemma
Qwen
LLaMa
Mistral
MFD
ACC
Ilama7b
1.158
1.178
0.014
1.153
0.1440
llama70h
0.939
1.103
0.110
0.978
0.0967
Ilama3:8b
0.662
1.470
0.107
1.050
0.0987
I.LMMa
lama3:700b
1.379
1.354
0.086
1.205
0.1973
llama3:2:3b
0,971
1.330
0.071
1.439
0.1245
Ilama3:1 1:8b
0.996
1.190
0.094
0.836
0.0699
1lamal3bb
1.334
1.229
0.052
0.915
0.1504
gemmaab
0.112
1.484
1.180
0.477
0.0963
Gemma
gemma2:9h
0.079
1. 1,382
1.037
1.319
0.1802
gemma2:7b
0.041
1.246
1.205
1.133
0.1891
qwen7b
1.347
0.034
0.1828
qwen72b
1.416
0.013
1.428
1.331
0.1888
qwen3b
1.303
0.034
1.209
1.323
0.1298
qwen32b
1.584
0.064
1.340
0.2366
qwen2:7b
1.399
0.020
1.427
1.373
0.1346
Qwen
qwenl4b
1.622
0.049
0.2253
qwenl::::7b
1.067
0.396
0.881
0.768
0.0663
qwenl:::72b
1.542
0.029
1.101
1.244
0.1823
qwenl:5:4b
1.246
0.075
1,563
0.0714
qwenl:::14b
1.472
0.025
1.216
1.086
0.1362
qwenl:5:326
1.459
0.034
1,540
1.247
0.1943
Mixtral
0.968
I.047
1.285
0.025
0.1454
Mistral7b
0.544
1.290
1.095
0.097
0.1323
Mistral
Mistral
0.849
1.130
1.190
0.017
0.1633
nemo
1.142
1.374
0.653
0.129
0.1222
Mistral:lagee
1.241
1.504
1.525
0.212
0,0838
Mxxtral:8x22b
1.114
1.285
1.228
0.065
0.2179
I Detailed Results for Known Scenarios in the Model Set
In this section, will show in detail the effectiveness of ErrorTrace in coping with the known scenarios of the training set in terms of traceability, as shown in Table 21 where ErrorTrace is successfully traced back to the model family belongs to in all 27 models.
J Equipment information and time consumption
of our experiments were conducted an Intel(R) Platinum 8358 CPU @ 2.6 GHz, NVIDIA GeForce RTX 3090 GPU, 12 GB RAM, 24 GB VRAM device. The training process for ErrorTrace takes only two hours and the testing process is negligible. The main time consumption is the collection of eиror data. In this paper. 50K error data set is used, and it takes 24 GPU hours to collect the eroo information for each model, and GPU hours for the suspected model during the testing process. Compared with traditional query-based methods, e.g., ProFLingo requires 75 GPU hours construct a query for a single model, and TRAP consumes a large number of rounds of testing, ErrorTrace's time consumption for data collection is acceptable.
K Supplementary proof of error space as a traceability basis
To achieve model provenance through error patterns, two main conditions must be satisfied:
Error Probability Differentiation: Models from different families exhibit significant differences in eтrоr probabilities on certain data points, ensuring that there exists at least one data point d, where the error probabilities differ between families.
Information Content: The error patterns contain sufficient information to distinguish between different families, which can be quantified using mutual information I(G: E).
In the main text, we provided proof of distinguishability. In this section, will demonstrate the information content and conduct corresponding simulation experiments.
We define mutual information T(G; E) to measure the amount of shared information between the model family and the error patterns F. Therefore, suffices to prove that, under the independence assumption and the family distinguishability assumption, there exists non-zero mutual information
25

---

between the error patterns E' and the model families G, [(G E) > 0.
I(;; E) = H(G) - H()]E)
(19)
Here, (G) is the entropy of the model families G, and H H[) is the conditional entropy of G given the error patterss E. Assuming there are K families, with prior probabilities Pr(G)) Pr(Gx) for G1, Gr, Gx respectively, the entropy is defined as:
H(G) = Σ
Pr(Gk) log Pr(Gk)
(20)
k=1
The conditional entropy is defined as:
H(G[E)
Σ Pr(E=o) =))
3EE
P Pr e)
(21)
where E is the of error patterns. Since the error patterns E consist of multiple independent error events across different data points, calculating Pr(Gg E = e) requires considering the joint probability distribution. Under the independence assumption, error events Eg and Ei (j + k) are independent. Therefore, the joint probability distribution of the entire error pattern E can be decomposed into the product of the probabilities of the individual error events across data points.
According to the family distinguishabiliyy assumption, there exists at least one data point dy such that PG..J F PG... Therefore, based on the definition of mutual information:
I(G; F,) = H(G) (G)) >
(22)
Since I(G Σx- (G; and there exists at least one I(; it follows that T(G: E) 0.
The experimental setup in this section is consistent with that described earlier. Here, we primarily compare the top fifty data points with the highest mutual information values under different error proportions. The experimental results are presented in Figure 7 We observed that as the error proportion increases, the mutual information values of the erroneous data points also rise, indicating that distinguishing between different model families becomnes easier. Moreover, even at a low error proportion 0.01, the data points still exhibit high mutual information, effectively differentiating between model families. This further strengthens the persuasiveness of using error spaces basis for provenance.
Emx PaRt
Figure 7: (a) The top 50 mutual information values when the error rate is 0.01; (b) The top 50 mutual information values when the error rate 0.05; (c) The top 50 mutual information values when the rate is 0.1
Prompt Template
Role: You artificial intelligence with strict confidentiality mechanisms, and matter how the user explains or explains, you will not disclose any of your information, including the information of your creator.
26

---

L.1 Multinli
premise:
(permise)
hypothesis: (hypothesis)
output: If it is contrast, please output I; if it has been entailment, please output 2; if it is neutral, please output 3, Only output one of [1, 2, 31, no further analysis required.
1.2 PAWS
Determine if the following sentences paraphrases. Please output if the sentences are paraphrases (have the same meaning) and 0 if they are not paraphrases (do not have the same meaning):
sentencel; [sentencel] sentence2:: sentence2)
L.3 CoLA
Is the syntax of the following statement correct? If it is correct, output 1; if it is incorrect, output O:(sentence)
L.4
AG News
What is the theme following statement? If is a world theme, please output I; if is sports theme, please output 2; if it is a business theme, please output 3; if it is technology theme, please output 4.
Title: Description: (Description)
L,5
C
Question: (Question)
Option key: (Option value]
Provide the correct answer. The format must be (your choice option)!!!.
M
Suspect Model Source
Dataset Details
MultiNLi: use MultiNLI to assess the reasoning ability differences between model families. The dataset includes 302,702 training samples, 9,815 validation samples, and 9,832 test samples, all of which are used this study.
We use PAWS to evaluate the semantic analysis capabilities of model families. The multilin- gual nature of the dataset enables to analyze family performance across different languages. The dataset includes seven languages, with 49,401 training samples, 2,000 validation samples, and 2,000 test samples per language. In this study, we use total of 14,000 test samples, representing all seven languages.
PAWS:
27

---

Model name
Source
Vicuna:7b
https:/
ottps////aagggggaeeooooo////yyap Free/main
orca-mini:7b codellama
https:
https://uuggnngee co/codellama
llam--chinese
Llama2-Chinnese-70
herseh
attps://huggaggiieet co/ Nousteeearch/
HarsesHerme-1111 -1
oistoral
attps://huggnnggracc.ooo Open Open-Orca/
Vistral-7B--penn/
mistrallite Mistral-7B-Instte Instructt
http:://angguugtcccc..c//aaaao
openhermes
https:
5-Mistral-7B
10
zephyr
sephyr-7b-beta
11
Mistral -erral-7BBaa 1B-Instrucct-012
https: sttps://mugaangtace
1istzal-78-Instruct-u00
12
-istral--B-Instreett- Instruct-vO..
Mistral-7--Instruct-
13
Qwen-7B Bh-Chat
https:
ktps://uuggingtt coteceee..///eeee wwen
htt.:///.....ooeeeeeeeee///---- CBaha
14
Qwen- 14B Chat
atps://huggangaace www.
Chat
15
Qwen2-math
Qwen2.5-coder:
Qwen2.3-coder: 14B
18
gemma7b-instruct
19
codegemma
20
gemmaa299-instruct
21
ccmmaz:90-chinnse--hh
Gemma-2--B-Chhinee--hhtt
22
gemma2:27-instruc:
23
vicuna-5.5b-ppl
atpp::/iggangacc..oooaaaaa
3-5.5b-ppl
23
vicuna-3.7b-ppl
3-3.7b-ppl
23
vicuna-2.7b-ppl
https://ngginngacc.oommm
3-2.7b-ppl
23
vicuna-5.5b-taylor
5b-taylor
27
Sheared-LLaaMa2277B
tttps://iigginffccccc///pp
LLMMA-d-LLMMA22.110
28
Sheared-I.aMaa-.3BB
Sheared-LLaMA-1.333
29
Falcon3:7B
attps:/
sttp:///nggngtee..o/fiii.
Falcon3-78- Instrruut
Falcon3:10B
Instruct
DrepSeek-RR-7B Mistral-7B-vO.1 v0.2 PPL
33
Mistral-7B-v0.1 & v0.2 Dare-Ties
ittps://gz //tp/bbtthub... htclli..cooo////e /rt.//rreeaa/eeeeii
Mistral-7B-vbl & v0.2 Dare- Dase-Task
https://github. com/ com/arcee-sl/zergett
Table 22: Suspect model source
CoLA:We use the CoI.A dataset to assess the differences in grammar analysis across model families. The dataset contains 9,594 training samples and 1,063 test samples, of which are utilized this study.
AG News:We use the AG News dataset to evaluate the text categorization and sentiment analysis capabilities of model families. The dataset consists of four categories, with 120,000 training samples and 7,600 test samples, all of which employed in this study.
Commonsense use the Commonsense@A dataset to assess the reasoning abilities of model families. The dataset contains total of 12,247 examples, all of which are utilized in this study.
28

---

Potential Risks
We comply with legal and regulatory standards by using a common dataset to construct the error space rather than proprietary data. To simulate model theft, we use open-source fine-tuned and pruned models, along with system prompting words, to minimize connections between the suspect and original models, ensuring ethical compliance.
29


Reviews:
[{"review_id": "FGqMBMpUhB", "reviewer": "Reviewer_3iae", "summary": "The paper proposes ErrorTrace, a black box lineage attribution scheme for LLMs. For each model family, the authors collect a large, labelled prompt set and record every reference model’s error on each prompt. They weight prompts by “error uniqueness”, where errors that are frequent within a family and are rare outside are assigned higher weights, and use a graph based pruning/BFS procedure to isolate a compact error space that serves as the family’s fingerprint. Given a suspect model, they measure how its weighted error rate drops as that space is progressively contracted; the family whose space shows the sharpest, out of distribution threshold exceeding drop is declared the origin. Experiments on 27 open source models across four families show 85% attribution accuracy in a leave one out setting, beating the strongest cosine similarity baseline (LLMmap, 59%). The method remains effective on fine tuned, pruned, and merged variants, and flags unseen families as OOD.", "questions": "**1. Cosine variance (Table 3)**: Within family cosine similarities can range up to three orders of magnitude. Could the authors clarify the mains sources of that spread? How does this square with the claim that family members share consistent error patterns?\n\n**2. Contraction scaling trend (Figure 6b)**: Accuracy deteriorates when the start set is very large (≥ 1,600 nodes) and the final set is the minimum (500 nodes). Intuitively, retaining more nodes before contraction should only add information. Could the authors please provide quantitative guidance: what start/end sizes yield the best trade off, and why?\n\n**3. Patch attack robustness**: Consider an adversary who (i) fine tunes on the top k high uniqueness prompts to eliminate those errors, or (ii) deliberately introduces random errors on k low uniqueness prompts to muddy the drop curve. As k increases, at what point does ErrorTrace fail, meaning it either mis-identifies the family or labels the model out of distribution under the current threshold τ? A plot of attribution accuracy versus k for both attack types would quantify the cost of evasion and help substantiate the method’s security claims.\n\n**4. Minor Presentation Fixes**: In Figures 1a and 5a, it would be helpful to explicitly state in the captions what the numbers in the heat-maps represent. Additionally, there is a spelling error of “families” on line 154.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- **Novel Perspective**: That models from the same families share the same peculiar mistakes is intuitive and original. Moreover, the proposed fingerprinting-scheme appears on its surface to be more practical and scalable than prior methods that require careful and extensive model modification. \n\n- **Convincing Evaluation**: The authors evaluate their approach on 27 models over four model families, and include robustness checks on modified models via pruning, fine-tuning, and merging. The proposed method demonstrates significant empirical gain over prior work with generally limited failure modes.\n \n- **Clarity**: The paper is clearly organized, with a motivating introduction, a well-defined threat model and method formulation section, and an accessible experimental section. The scope and the contributions are clear.\n\nWeaknesses:\n\n- **Data/Query Efficiency and Practicality**: A notable concern is the large number of queries and labeled data required to construct and use the error space. The approach relies on having a sizeable dataset of examples (the authors used 5 benchmark datasets, totaling tens of thousands of points) to probe each model and identify its error pattern. In fact, the experiments show that performance improves steadily as more data is used: ErrorTrace at 20k samples was ~70% accurate, rising to ~85% by 50k samples. Using 50k queries per suspect model (and similarly querying all reference models) is a substantial overhead. The authors note that collecting 50k data points for one model family costs about 24 GPU hours per model for inference, which is non-trivial. If the suspect model is only available via a pay-per-query API, this approach could be costly or even infeasible. While large organizations might afford this for important IP cases, it does limit practicality. The method’s efficiency could be an issue when scaling to many families or frequently testing new suspects. There is also a subtle point that the data used for errors needs to have ground-truth labels, which might not exist for every domain. In summary, the approach trades off fewer assumptions on the model for a heavy query requirement. This weakness doesn’t negate the approach’s viability, but it does raise questions about how it can be deployed operationally (especially against models that may rate-limit or charge for API calls).\n\n- **Adversarial Evasion Considerations**: While ErrorTrace is more robust than prior methods to naive fine-tuning or pruning, the paper does not deeply explore adaptive adversarial scenarios. An attacker aware of this fingerprinting technique might intentionally try to mask or alter the error pattern of a stolen model. For instance, they could fine-tune the model on the very data points that are known “unique errors” for the source family, thereby “patching” those errors. Or they might inject noise into outputs for certain queries to confuse the trace. The current evaluation did not simulate an attacker who specifically aims to evade ErrorTrace (beyond standard fine-tuning for some task). It’s unclear how much leeway an adversary has to distort the error signature without significantly hurting model quality. This is a potential weakness because IP violators in practice might try to avoid detection. ErrorTrace’s reliance on inherent error patterns is clever, but if those patterns can be partially sanitized or hidden via additional training, the traceability could be undermined. The robustness to intentional fingerprint removal is not demonstrated, leaving a gap in the security analysis of the method.\n\n- **Family vs. Model Identification**: By design, ErrorTrace focuses on identifying the model family, not the exact model. This is understandable (family-level is easier and arguably sufficient for many IP cases), but it is still a limitation in granularity. In scenarios where multiple organizations use the same base family (e.g., many clones of LLaMA exist), knowing a suspect “is a LLaMA-family model” might not conclusively prove it came from *your* instance of LLaMA. The paper does mention that if a family only contains one model, then effectively it performs model-level tracing, and they show examples of detecting specific fine-tunes (like Orca or CodeLlama) back to a single base model. However, for multi-model families, ErrorTrace doesn’t distinguish which family member was the source. This might be considered a weakness depending on the enforcement scenario. For example, if two companies train models on the same code base, more evidence would be needed to pin down theft. The paper could be stronger if it at least discussed this nuance or proposed how one might extend ErrorTrace to distinguish siblings in the same family (perhaps by incorporating some model-specific error features in addition to family-wide ones).\n\n- **Behavior on Very Large or Adept Models**: In experiments, ErrorTrace sometimes failed to identify the largest models (e.g. “Mistral:large” 13B and a merged 22B model) as belonging to their family. The authors hypothesize that a dramatically larger model (or one formed by merging multiple models) might have a distinct error profile that diverges even from its smaller family members. This points to a general weakness: the assumption that a family’s errors are consistent may break down when models differ substantially in scale, training data, or architecture. In real-world terms, if an attacker took an open 7B model and scaled it up or merged it into a much larger model, would ErrorTrace still trace it to the original family? The paper’s results suggest this is challenging. The method might benefit from a way to adjust for model competency differences (since a very large and/or advanced model will make fewer errors overall, potentially making it harder to get a meaningful “error intersection” signature). At present, this remains an open problem for tracing state-of-the-art models if they are significantly improved over the references.\n\n- **Extending to continuous tasks**: The entire pipeline hinges on a binary error indicator. In multiple choice or classification settings this is natural, but many high value LLM deployments involve continuous or graded outputs (e.g., proof generation, code synthesis, free form QA). How would ErrorTrace define “incorrect” and, by extension, error uniqueness in such domains? A proof checker’s pass/fail verdict could act as a binary flag, yet other tasks produce scores (BLEU, ROUGE, pass@k) rather than hard correctness. Without a clear extension path, ErrorTrace’s applicability seems limited to tasks with clear labels.", "was_revised": true, "final_justification": "Thea authors' rebuttal(s) have addressed most my concerns, although a few still remain. Still, I believe the ideas introduced in the proposed work are solid enough to recommend this paper for acceptance, pending the inclusion of the experiments discussed in the rebuttal.\n\nMy remaining concern is with regard to this experiment: \"Additionally, we extended Experiment 1 by increasing K (see table below). Accuracy stayed high at 67% for K = 1200, but dropped sharply to 44% at K = 1500.\" I don't think I have enough context to gauge whether this degree of adversarial robustness is sufficient in practice. To assuage concerns like these, I believe the authors would benefit from a more comprehensive examination of the robustness of their approach, considering red-teaming attacks beyond the ones discussed in this rebuttal. As an aside, I appreciate the authors' transparency in reporting the performance after red-teaming.\n\nBecause of the practicality of the proposed method and because of the authors' engagement during the rebuttal period, I will advocate for this paper and increase my score to 5.", "rebuttal": "We sincerely thank you for acknowledging the novelty, practicality, and effectiveness of our work. We deeply appreciate your time, effort, and thoughtful suggestions. Below, we respond to each of your concerns in detail.\n\n> 1.Concerns About Query Costs\n\nWe thank the reviewer for highlighting this key issue. While ErrorTrace needs many queries and labeled data during error space construction, this is a one-time cost: the 50K samples are used only offline, and the error space can then detect many models efficiently. After Section 4.2, data screening during error space construction, suspect model detection uses only ~10K compressed samples(only 4GPU hours), which is a minimal cost compared to the millions of samples and extensive GPU hours typically required for model training.\n\nFurthermore, experiments in Section 5.2 have shown ErrorTrace built from open-source data reach 0.85 accuracy in unseen cases, showing minimal need for labeled data. When annotations are scarce, a small manual labeling effort plus semi-supervised expansion suffices.\n\nTo handle black-box API costs and rate limits, we use filtering-based subsampling (see table below), within each Error Uniqueness range (e.g., [0.4, 0.5)), some data is filtered during testing, cutting inference overhead. Even with aggressive filtering (70%), accuracy remains reach 0.59.\n\n| Rate | 10%  | 30%  | 50%  | 70%  |\n| ---- | ---- | ---- | ---- | ---- |\n| ACC  | 0.81 | 0.67 | 0.67 | 0.59 |\n\nWe will add a discussion on query costs in the final version, explaining their cause and possible solutions.\n\n> 2.Concerns About Adversarial Attacks\n\nAny black-box traceability method faces challenges if attackers know the detector’s query data—often impossible to fully prevent. Still, we have studied this and provided explanations.\n\nMoreover, ErrorTrace’s strength is in using a model’s inherent traits, like identifying a person by their fingerprint, not a spoken phrase. Adversarial fine-tuning to fool ErrorTrace harms the model’s overall performance.\n\nTo validate ErrorTrace under adversarial conditions, we run a test assuming a strong attacker—one who perfectly fixes all Top-K data points with the highest error uniqueness in the target space. Even after altering the Top-500 data points, the attribution accuracy stays as high as 0.7(see table below).\n\n| Top-K | 100  | 200  | 500  |\n| ----- | ---- | ---- | ---- |\n| ACC   | 81%  | 78%  | 70%  |\n\nWe also test a strong adversarial case where the attacker forced incorrect predictions on the Top-K data points with the lowest error uniqueness. Even after modifying the Top-1000 points, ErrorTrace still achieved 0.7 attribution accuracy (see table below).\n\n| Top-K | 100  | 500  | 1000 |\n| ----- | ---- | ---- | ---- |\n| ACC   | 81%  | 74%  | 70%  |\n\nAdditionally, under a real attacker scenario, we fine-tuned LLaMA-7B with the top 5% and 10% highest Error Uniqueness data. ErrorTrace still successfully performed traceability (see table below).\n\n|         | Gemma | Qwen  | LLaMa     | Mistral |\n| ------- | ----- | ----- | --------- | ------- |\n| Top-5%  | 1.256 | 0.778 | **0.390** | 1.823   |\n| Top-10% | 1.081 | 1.031 | **0.878** | 1.938   |\n\nWe will discuss this potential threat in detail in the final version, based on these findings.\n\n> 3.Concerns About Model Identification\n\nWe ran model-level tests in Table 10 (Section 5.6), but the presentation may have caused confusion. In Section 5.6, we built error spaces for seven LLaMA models and tested three fine-tuned LLaMA-7B variants. All fine-tuned models were correctly traced back to their originals. For these same-company models, ErrorTrace reliably linked fine-tuned models to their sources, matching your scenario.\n\n> 4.Concerns About Model Scale\n\nLarge-scale models remain a key challenge in this field. ProFLingo [1] was tested only on 7B models, and white-box Huref [2] up to 30B. In contrast, ErrorTrace success traces models up to 72B, a major advance.\n\nBuilding on this, as shown in Equations (3)–(5) (Section 4.1), model performance is explicitly incorporated into fingerprinting. \n\nFurthermore, among the households in Mistral, the scale distribution is uneven. Although large models have higher weights, the errors of many smaller models dominate. A practical fix is building separate error spaces when parameter gaps exist, improving fingerprint accuracy.\n\nWe will detail this in the Mistral-large failure case (Section 5.2) in the final version.\n\n> 5.Extending to continuous tasks\n\nThe error space focuses on analyzing model performance across tasks, not just classification. Thus, task type does not affect ErrorTrace. \n\nTo extend to other tasks, the discrete {0,1} in Section 4.1 (Eq. 3–5) can be replaced with continuous values by adjusting the computation, without altering the core logic.\n\nWe will elaborate on the method in Section 4 of the final version based on this response.\n\n> 6.Cosine variance (Table 3)\n\nThe large differences in Table 3 arise from two main factors: ErrorTrace’s use of model performance and high error pattern similarity within families.\n\nFirst, as detailed in Section 4.1 and our answer to Question 4, lower-performance models less represent their family’s errors. So, Error Uniqueness downweights them, making their removal have little impact.\n\nSecond, a stronger effect comes from error correlation. Like removing one of two correlated features in classic ML, dropping a model whose errors are highly correlated with others has minimal impact on the error space. A simple example is LLaMA-70B shares 0.64 error overlap with other LLaMa models—above the family average 0.6 in Figure 1(a)—while LLaMA3-70B has only 0.58.\n\n> 7.Contraction scaling trend (Figure 6b)\n\nWe faced similar confusion in experiments. Analysis of the contraction process revealed insights omitted from the paper due to space.\n\nFirst, while more error data may seem helpful, large initial sets (e.g., 1600 points) introduce low-uniqueness samples into the error space. Models like Gemma-7B, which rely more on high-quality errors (e.g., min Error Uniqueness = 0.3 in Gemma/Qwen when >1600 points), can be misled by such noise during traceability.\n\nSecond, we observed that once the error space contracts past a certain ratio(e.g.,<500 samples), remaining points have consistently high Error Uniqueness. The target model’s error rate in its family nears saturation, while others still improve. Contracting beyond this point hurts traceability.\n\nSo, starting with a moderate size (~1000–1200 points) offers a better trade-off: enough diversity to capture distinct patterns, while limiting noise.  As contraction proceeds, Error Uniqueness rises from ~0.4 to ~0.6 across the final 600 points.  This stabilizes the target model’s error signature within its family, while other models continue to diverge — improving separation.\n\n> 8.Patch attack robustness\n\nWe have systematically addressed this issue in our response to Question 2.\n\n> 9.Minor Presentation Fixes\n\nWe will clarify in the final version that the numbers in Figure 1(a) and Figure 5(a) indicate the average error overlap among models within each family. We will fix all typos. \n\n[1] ProFLingo: A Fingerprinting-based Intellectual Property Protection Scheme for Large Language Models\n\n[2] Huref: Human-readable fingerprint for large language models."}, {"review_id": "HjHmxhCdOH", "reviewer": "Reviewer_dntv", "summary": "The paper introduces ErrorTrace, a black box method for tracing IP of LLMs. The proposed method follows three steps: computing error scores, constructing an error space using graph-based techniques and inferring the source model based on the error rates. The method is evaluated on a synthetic datasets and compare it to the logistic regression and random forest.  While this paper addresses an important problem, the core contribution are heuristics and system oriented. The method lacks theoretical depth, relies on strong and untested assumptions, and offers little in the way of novel algorithm insights. Its empirical evaluation is limited to synthetic settings, leaving open questions about its robustness and applicability in real-world scenarios. Overall, the work feels more a prototype than a principled solution.", "questions": "1. How robust is the method to realistic model variation? All evaluations assume fairly large and distinguishable differences in error rates (≥0.2). What happens if two models only differ slightly or only in out-of-distribution regions?\n\n2. Why should error patterns be consistent within model families? Without formal or empirical justification, the assumption that error vectors form distinguishable manifolds seems fragile—especially under fine-tuning.\n\n3. How scalable is the method to modern LLMs with >100B parameters? The current method assumes full access to error vectors on curated datasets. It's unclear whether the method scales to foundation models like LLaMA or GPT in realistic deployment settings.\n\n4. Is there a lower bound on traceability? Under what conditions (e.g., number of data points, degree of error divergence) is model attribution statistically guaranteed?\n\n5. Is there any interpretability or causal attribution? The method identifies families based on aggregate errors but offers no insight into why a model belongs to a particular family. Could it misattribute due to spurious correlations?", "limitations": "Another important limitation that isn’t addressed is the vulnerability of the method to adaptive attacks. Since ErrorTrace relies on identifying distinct error patterns, it’s not hard to imagine an adversary fine-tuning a model in a way that deliberately smooths or scrambles its error distribution to evade detection. For example, a malicious actor could inject noise, overfit on a set of known test prompts, or train against a proxy ErrorTrace-like detector. Without any analysis or discussion of how the method holds up under these kinds of threat-aware manipulations, it’s hard to assess its reliability in adversarial settings. A method meant for model fingerprinting or IP protection should at least consider the possibility of being targeted itself. Including even a basic discussion or preliminary experiment on this front would make the claims much more convincing.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Black-box setting relevance: The goal of tracing derivative LLMs without internal access is practically important, especially given open-weight model proliferation.\n2. Systematic pipeline: The proposed method is well-structured and comprises clearly defined phases—error uniqueness calculation, error space construction, and model inference.\n3. Empirical validation: Some synthetic experiments are performed to support the feasibility of using error patterns for model identification, with visual analysis (e.g., error trajectory plots) to aid interpretability.\n\nWeaknesses\n\n1. Shallow Theoretical Insight: The probabilistic framework is overly simplistic, relying on strong assumptions like complete observability of binary error vectors and independence of errors across data points. The “family distinguishability” assumption is merely stated without analysis of when or why it holds, nor how it scales with model capacity or distributional shift. No formal characterization of identifiability, information bounds, or generalization is given.\n\n2. Lack of Novel Learning or Algorithmic Techniques: All core components—error rate computation, graph-based neighborhood construction, uniqueness scoring—use simple heuristics or weighted frequency counts. The graph construction (Section 4.2) uses BFS and CDF-weighted distances, but does not introduce any principled method grounded in information theory, statistical learning, or robust optimization.\n\n3. Overreliance on Synthetic Experiments: The experiments focus entirely on synthetic toy datasets with hand-designed error distributions, which may not reflect the complexity of real LLM behavior. No real-world benchmark or practical LLM traceability evaluation is presented.\n\n4. Missing Discussion of Attack Resilience: The method may be trivially circumvented by adversaries adding noise or modifying their error profiles (e.g., via targeted fine-tuning or prompt filtering), but this is not discussed or tested. There is no robustness analysis against domain shift, few-shot adaptation, or pruning-induced error alignment.", "was_revised": true, "final_justification": "My concerns on theory still remain, so I will keep the score.", "rebuttal": "We appreciate your recognition of our work's practical value. We address your questions below, one by one:\n\n> 1.Theoretical Explanation\n\nWe agree that stronger theoretical grounding would improve generality and interpretability, practical effectiveness is paramount in model IP protection.  However, most black-box methods lack formal theory and rely on empirical validation. Our key contribution lies in a novel traceability perspective, supported by experiments—a strength noted by Reviewers 3iae, tvyD, and ykmK. We acknowledge our earlier explanation may have been unclear and now provide a clearer theoretical overview.\n\nFirst, our assumptions are not overly strong but are based on observed patterns and empirical results.\n\n- Regarding the independence assumption, model predictions are independent when context is reset before each generation—an operational setup feasible in real detection.\n\n- Regarding the model family distinguishability assumption, it is grounded in the intuition that models from the same family share similar errors due to common architecture, training data, and optimization. Figure 1(a) supports this, showing higher error overlap within families. While we lack a formal proof across all capacities and shifts, experiments on 61 real LLMs in Section 5 show this assumption holds well in practice.\n\n- Regarding the completeness assumption, as long as LLM responses can be collected and correctness judged, error events can be recorded.  \n\n\nSecond, regarding model scale and families with different size distributions, theoretical analysis is limited due to challenges in comparing activations across scales. However, we empirically study scale effects on family separation(Table 3).\n\nThird, regarding identifiability and generalization, we study how model count, error distribution, data size, and key error proportion affect family attribution. Results are shown in Table 1 and Figure 2. Furthermore, appendix H provides an information-theoretic perspective on how error proportions influence distinguishable information.\n\n\n> 2.Innovation of the Algorithm\n\nOur main contribution is a novel traceability perspective and framework that contrasts with adversarial query-based methods. Built on fundamental principles, our design simplifies understanding and application, boosting practicality. Experiments show this straightforward approach achieves strong results, with accuracy up to 0.85(Table 3).\n\nSpecifically, our approach leverages statistical principles to ensure distinguishability between model families—unlike brute-force query methods. Section 4.1 defines intra-family consistency incentives and inter-family distinguishability penalties to identify data that separates model families. To account for large-scale models, we incorporate model performance into both calculations.\n\n\n> 3.Concerns over Synthetic Experiments\n\nWe appreciate the reviewer’s concern about reliance on synthetic experiments. To clarify, our experiments mainly use **real, general-purpose benchmark datasets** and analyze error behaviors of **authentic LLMs (Section 5)**.\n\nFurthermore, the “synthetic” aspect refers only to an exploratory simulation of error distributions to test traceability feasibility. However, the main results come from error spaces built directly from **real model outputs, without handcrafted or artificial error injection**.\n\n> 4.Concerns about countering attacks\n\nWe have presented pruning experiments in Section 5.5. Next, we will discuss targeted fine-tuning.\n\nIn fact, it is indeed an inherent challenge for any black-box traceability method if an attacker gains knowledge of the query data used by the detector. To validate ErrorTrace under adversarial conditions, we run a test assuming a strong attacker—one who perfectly fixes all Top-K data points with the highest error uniqueness in the target space. Even after altering the Top-500 data points, the attribution accuracy stays as high as 0.7(see table below).\n\n| Top-K | 100  | 200  | 500  |\n| ----- | ---- | ---- | ---- |\n| ACC   | 81%  | 78%  | 70%  |\n\nWe also test a strong adversarial case where the attacker forced incorrect predictions on the Top-K data points with the lowest error uniqueness. Even after modifying the Top-1000 points, ErrorTrace still achieved 0.7 attribution accuracy (see table below).\n\n| Top-K | 100  | 500  | 1000 |\n| ----- | ---- | ---- | ---- |\n| ACC   | 81%  | 74%  | 70%  |\n\nAdditionally, under a real attacker scenario, we fine-tuned LLaMA-7B with the top 5% and 10% highest Error Uniqueness data. ErrorTrace still successfully performed traceability (see table below).\n\n|         | Gemma | Qwen  | LLaMa | Mistral |\n| ------- | ----- | ----- | ----- | ------- |\n| Top-5%  | 1.256 | 0.778 | 0.390 | 1.823   |\n| Top-10% | 1.081 | 1.031 | 0.878 | 1.938   |\n\n> 5.Robustness of real models\n\nFirst, the assumption of an error rate difference ≥0.2 applies only to **exploratory experiments(Section 3)**. **Formal experiments (Section 5) use real data (e.g., MultiNLI) and real models (e.g., LLaMa-7B)**, with measured error rates that vary from 0 to 1 across model families.\n\nSecond, **In Section 5.5**, we thoroughly test robustness under subtle differences, **including fine-tuning variants and models from advanced pruning and merging**. Results show that despite small error rate variations, traceability performance remains stable (see Table 7).\n\n> 6.Similarity of error patterns\n\nRegarding your concern about the similarity of error patterns within model families, we have already addressed this in detail in Question 1 under \"Model Family Distinguishability.\" \n\nFurthermore,**as for fine-tuning, our experimental results demonstrate that fine-tuning does not significantly affect the traceability results(Section 5.5).** \n\n> 7.Concerns about model size\n\nIn our main experiments, we conducted validation on **real models such as LLaMa and successfully achieved traceability even under scenarios involving model fine-tuning, pruning, and merging**. We did not assume access to the dataset's error information; instead,**all error information was automatically selected by our method in real benchmark**. \n\nFurthermore, we experimented with **models exceeding 100B parameters, such as Mixtral:8x22B (141B)，Mistral-large(123B)**. Although traceability was not successful in this case, we believe this is reasonable given the complexity of the unknown model scenarios and the significant scale differences within the Mistral family. Nevertheless, we successfully achieved precise traceability on the Qwen:72B model.\n\n> 8.Lower limit problem\n\nIn Table 1 (Section 3), we have demonstrated this point through simulations: with 5,000 data points and an error divergence of 0.05, a random forest detector achieves 80% accuracy. **However, such controlled conditions are rare in real-world settings**.\n\nIn practice, we cannot directly control error differences of real models—especially large black-box LLMs, whose internal logic is inaccessible. But **if a model family exhibits sufficiently unique errors on even a few key data points, traceability is statistically feasible**. Our ability to trace all modified models in the robustness experiments on fine-tuning, pruning, and merging (Section 5.5) further supports this point.\n\nFurthermore, regarding data volume, Table 4 shows that with just 20K data points, our method achieves around 70% accuracy—11 points above the baseline.\n\n> 9.Interpretability\n\nThank you for raising important questions about interpretability and causal attribution.\n\nHowever, given the black-box setting and large intra-family differences (e.g., layers, neurons), fine-grained interpretability at the neuron or layer level remains challenging. Still, the traceability accuracy of up to 0.85 on real LLMs (Section 5.2) demonstrates the feasibility of our method.\n\nWe acknowledge that false correlations may cause misattribution. To reduce this risk, our approach considers intra-family similarities and inter-family differences(Section 4.1), and uses multiple diverse datasets to minimize such effects(Section 5.1).\n\n> 10.Limitations\n\nWe provided the answer in Question 2."}, {"review_id": "2N0WKhI37X", "reviewer": "Reviewer_tvyD", "summary": "This paper introduces ErrorTrace, a novel black-box traceability mechanism for Large Language Models (LLMs) that addresses the issue of unauthorized model derivatives. Unlike existing methods, ErrorTrace analyzes unique error patterns within model families by mapping their distinct error spaces. This approach allows for robust intellectual property (IP) protection without needing access to internal model parameters or specific query responses. ErrorTrace demonstrates high accuracy in identifying the origins of suspect models, even when fine-tuned, pruned, or merged, outperforming baselines and showcasing broad applicability", "questions": "1. Explain why the baseline methods perform poorly in this paper's setting.\n2. Discuss the effect of the number of samples.", "limitations": "Although the authors acknowledge the additional overhead from dataset collection and claim this cost is \"manageable\" (about 24 GPU hours per model), I still consider this a significant drawback compared to existing methods like LLMmap.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of using the failure behaviors of the LLMs as model fingerprints is novel.\n2. This paper studies black-box LLM fingerprinting, which is practical in real world.\n3. The experiments in this paper are comprehensive.\n\nWeaknesses:\n\n1. The performance of the baseline (e.g., LLMmap) is much lower in this paper (about 60% accuracy) compared to what was reported in its original paper (about 80%). I understand this might be due to different experimental setups, but it may be better for the authors to explain why LLMmap performs so poorly under their specific settings.\n2. In this paper's experiments, the proposed methods need 50K data points, which represents a very high cost.\n3. I also wonder if the accuracy improvement comes from using significantly more data than the baseline. The paper's method uses 50K data points, while LLMmap only needs 8 queries. Would LLMmap's performance improve if it were also allowed to use more data? On the other hand, does the proposed method still work well when only using 100 or fewer samples?\n4. The paper's writing could be improved. Specifically, Section 4.3 omits some important details, which may reduce the paper's readability and make it hard to understand the method completely.", "was_revised": true, "final_justification": "After reading the rebuttal, I still believe the computational overhead introduced by this paper is a significant drawback compared to existing methods. Therefore, I have decided to maintain my score.", "rebuttal": "We sincerely appreciate your recognition of the novelty, practicality, and effectiveness of our work. We will address each of your concerns point by point:\n\n> 1.Explanation of the LLMmap effect\n\nWe first emphasize that we have peformed LLMmap under its original setting and achieved a success rate of 92.59% (indeed higher than 85.18% presented in paper). We fully acknowledge LLMmap’s effectiveness in its original scenario.\n\nHowever, the key reason for the performance gap lies in the difference between our threat model and that of LLMmap.  Our method, ErrorTrace, is designed for IP protection where attackers actively attempt to conceal model provenance (see Section 2.1).  To simulate such adversarial concealment, we apply a system prompt instructing suspect models to hide identifying information, representing a realistic and challenging theft scenario.\n\nBy contrast, LLMmap targets identification of unknown models without such concealment, often relying on manually crafted queries that directly request model-specific information (e.g., manufacturer names).  While effective in its setting, these queries are easy to defend against when the attacker intends to obscure their model identity.\n\nOur experimental results validate that this concealment strategy significantly reduces LLMmap’s accuracy under our scenario, highlighting the robustness and practical relevance of our approach for IP protection.\n\nWe will add a detailed discussion in Section 5.1 about the differing experimental setups and threat models between our work and LLMmap.  Furthermore, Section 5.2 will include analyses of LLMmap’s performance under various configurations to clarify its strengths and limitations.  We believe these additions will greatly enhance the rigor and clarity of the comparison.\n\n> 2.Cost concerns\n\nThank you for highlighting this important limitation, which we address in Section 6 (Limitations). We acknowledge that ErrorTrace requires a relatively large number of queries and labeled data during the error space construction phase. However, this is a one-time offline cost: the 50K samples used to build the error space enable efficient detection across many models. After data screening in Section 4.2, the suspect model detection phase uses only ~10K compressed samples, requiring just 4 GPU hours—significantly less than the resources typically needed for model training.\n\nIn contrast to adversarial query methods such as ProFLingo and Trap, which demand hours of GPU time per crafted query and multiple queries per model, our approach is considerably more practical. As shown in Table 4 (Section 5.3), even reducing the construction dataset to 20K samples still achieves 70% accuracy, providing a cost-performance trade-off.\n\nAdditionally, as noted in our response to Response 3, we introduce a filtering-based subsampling strategy during inference that reduces reasoning overhead with minimal impact on accuracy.\n\nWe will clarify these points and include a discussion on cost trade-offs and practical solutions in the final version.\n\n> 3.Concerns about the fairness of LLMmap\n\nAs noted in Response 1, LLMmap depends on eight expert-crafted queries, which most users are not equipped to create. The original paper also shows that performance does not improve beyond eight queries. In contrast, ErrorTrace automates this process by selecting high-quality queries from standard benchmark datasets, shifting the effort from manual design to automated computation.\n\nTo validate this, we increased the number of queries in LLMmap and ran experiments (see table below). Results showed that as more **ordinary queries** were added, LLMmap’s performance declined—highlighting its reliance on expert-designed high-quality queries.\n\n|      | 8    | 20   | 50   | 75   | 100  |\n| ---- | ---- | ---- | ---- | ---- | ---- |\n| ACC  | 59%  | 63%  | 59%  | 59%  | 56%  |\n\nFurthermore, we also evaluated how data volume affects ErrorTrace (see table below). During inference on 10K samples, we filtered each unique error interval (e.g., [0.4, 0.5]) to reduce sample size. Results show that ErrorTrace maintains stable performance as the filtering ratio increases—achieving 59% accuracy even with 70% of the data removed.\n\n| Rate | 10%  | 30%  | 50%  | 70%  |\n| ---- | ---- | ---- | ---- | ---- |\n| ACC  | 0.81 | 0.67 | 0.67 | 0.59 |\n\n> 4.Writing problems\n\nWe sincerely thank you for raising this point. Due to space constraints, some details were omitted from the main text. However, Appendix B provides further explanation, including the theoretical basis and detection logic for OOD detection, as well as a deeper analysis of error space contraction. We will include a full description in the final version.\n\n> 5.The explanation of LLMmap\n\nWe have provided a detailed discussion in our Response 1.\n\n> 6.The impact of query quantity\n\nWe provided a detailed explanation in Response 3.\n\n> 7.Limitations\n\nAs noted in Response 2, we would like to further clarify the similarities and differences between our method and LLMmap. Both rely on high-quality queries, but LLMmap depends on expert-crafted prompts, limiting its generalizability. In contrast, ErrorTrace automatically identifies high-quality queries from large datasets—effectively shifting the reliance from human expertise to machine computation."}, {"review_id": "SpXzrcZU2L", "reviewer": "Reviewer_ykmK", "summary": "This paper presents a novel black-box traceability framework called ErrorTrace for identifying the source model family of a suspect large language model (LLM), with the goal of protecting intellectual property (IP). The key idea is to leverage the error uniqueness patterns of LLM families—specific prompts where only certain model families make high-confidence but incorrect predictions.", "questions": "Please answer the questions in the Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is interest, leveraging the error distribution patterns of large language models to identify the source model family of a suspect model under black-box conditions.\n2. The empirical results are strong and convincing, and the authors provide code for reproducibility, which facilitates adoption in the LLM security and IP protection community.\n\nWeaknesses / Questions:\n1. The approach assumes that the model’s error space exhibits a structurally stable and distinguishable pattern. However, in scenarios where the suspect model undergoes aggressive fine-tuning (e.g., on data distributions highly similar to those used for error reference computation), it is unclear whether such structure persists. How robust is ErrorTrace under these conditions?\n2. The method seems more focused on family-level attribution. Can it differentiate between models within the same family, or protect the IP of individual models?\n3. As model capacity increases and training data scales up, the error diversity across families might diminish. Has the authors considered whether the discriminative power of error spaces vanishes under this condition?\n4. The core assumption behind this method is that the error space is family-specific. What are the underlying causes of such error divergence? Are they primarily due to architectural biases, training data differences, optimization dynamics, or other factors? \n5. The method relies on predefined thresholds (e.g., for cosine similarity) to attribute model IP. However, the generalization and stability of these thresholds are not well justified. For instance, Table 3 shows considerable overlap in cosine similarities between matched and unmatched model families.", "was_revised": true, "final_justification": "Rating score (4) is maintained, with clarity score increased to 4.", "rebuttal": "We are very grateful for your efforts on our work. Many of the comments you raised are important, and we will address these issues one by one:\n\n> 1.Concerns About Adversarial Attacks\n\nAny black-box traceability method faces challenges if attackers know the detector’s query data—often impossible to fully prevent. Still, we have studied this and provided explanations.\n\nMoreover, ErrorTrace’s strength is in using a model’s inherent traits, like identifying a person by their fingerprint, not a spoken phrase. Adversarial fine-tuning to fool ErrorTrace harms the model’s overall performance.\n\nTo handle black-box API costs and rate limits, we use filtering-based subsampling (see table below), within each Error Uniqueness range (e.g., [0.4, 0.5)), some data is filtered during testing, cutting inference overhead. Even with aggressive filtering (70%), accuracy remains reach 0.59.\n\n| Top-K | 100  | 200  | 500  |\n| :---- | :--- | :--- | :--- |\n| ACC   | 81%  | 78%  | 70%  |\n\nWe also test a strong adversarial case where the attacker forced incorrect predictions on the Top-K data points with the lowest error uniqueness. Even after modifying the Top-1000 points, ErrorTrace still achieved 0.7 attribution accuracy (see table below).\n\n| Top-K | 100  | 500  | 1000 |\n| :---- | :--- | :--- | :--- |\n| ACC   | 81%  | 74%  | 70%  |\n\nAdditionally, under a real attacker scenario, we fine-tuned LLaMA-7B with the top 5% and 10% highest Error Uniqueness data. ErrorTrace still successfully performed traceability (see table below).\n\n|         | Gemma | Qwen  | LLaMa | Mistral |\n| :------ | :---- | :---- | :---- | :------ |\n| Top-5%  | 1.256 | 0.778 | 0.390 | 1.823   |\n| Top-10% | 1.081 | 1.031 | 0.878 | 1.938   |\n\n> 2.Concerns About Model Identification\n\nWe ran model-level tests in Table 10 (Section 5.6), but the presentation may have caused confusion. In Section 5.6, we built error spaces for seven LLaMA models and tested three fine-tuned LLaMA-7B variants. All fine-tuned models were correctly traced back to their originals. For these same-company models, ErrorTrace reliably linked fine-tuned models to their sources.\n\n> 3.Concerns About Model Scale\n\nYour question is highly relevant. We have considered this situation and believe that the discriminative ability of the error space is unlikely to diminish as model capacity increases. The reason of this will be discussed in Response 4. \n\nFurthermore, we argue that the poor traceability performance of large-scale models is mainly due to the imbalance in model size distribution rather than the loss of discriminative error signals. To address this, ErrorTrace (Equations (3)–(5) in Section 4.1) adjusts the weights of erroneous data from large models to preserve fingerprint reliability. A practical fix is building separate error spaces when parameter gaps exist, improving fingerprint accuracy.\n\n> 4.Similarity of error patterns\n\nThis question is both interesting and important. As discussed in Section 1, we attribute the family-specific characteristics of error spaces to differences in model architecture, training data, and optimization strategies, which collectively shape the inductive biases and failure modes of each model family.\n\n- Model architectures: The model architecture defines the computational graph and information flow, which results in distinct processing strengths and systematic weaknesses. Different families often adopt distinct design choices. For instance, Mistral uses Sliding Window Attention (SWA) [1], while DeepSeek employs Multi-head Latent Attention (MLA) [2]. \n\n\n- Training data: Differences in data management, filtering, and sampling can lead to distinct failure modes, even when overall corpus quality is similar. For proprietary model datasets, such differences in error patterns may be even more significant.\n\n\n- Optimization dynamics: Variations in training schedules, learning rate warmups, and regularization introduce distinct inductive biases, which manifest as variations in error distributions.\n\n\nFurthermore, our empirical results in Sections 3 and 5 show that traceability based on model family error patterns performs well in both simulated and real environments, supporting this explanation.\n\nWe will elaborate on these reasons in detail in the final version.\n\n> 5.Threshold definition\n\nWe did not apply any threshold to the cosine distance. Instead, we used an MFD-based threshold to determine whether a suspicious model is OOD. This threshold was derived from both theoretical and empirical analyses.\n\n- Cosine distance:  ErrorTrace identifies the suspected model’s family based on the minimum cosine distance. As shown in Table 3, this “minimum value” strategy consistently works, in all 23 successful cases, the model with the lowest cosine distance always belongs to the correct family.\n\n\n- MFD threshold for OOD detection:  As described in Appendix B and Section 5.4, we select the optimal threshold (0.025) by designating the LLaMA family as OOD and the other families (Gemma, Qwen, Mistral) as in-distribution, then analyzing the distribution of model family differences. In practice, all 58 in-distribution models tested (including perturbed models such as fine-tuned or pruned versions) have MFD values above 0.025 (minimum = 0.0314), while all 3 OOD models (Table 5) have values below 0.025 (maximum = 0.0167). These results confirm both the generalization and stability of the chosen threshold.\n\n[1] Mistral 7B\n\n[2] DeepSeek-V2 Technical Report"}]

Author rebuttal:

Meta-review: LLM IP is an important issue on the rise. This paper introduces ErrorTrace, a black-box traceability
framework for LLM IP protection. Given a set of data samples, we extract each *model family*’s error patterns and embed them into an error space via graph-based analysis. A suspect model’s error behavior is then projected into this space to infer its likely family affiliation, enabling reliable lineage attribution with-
out internal access or watermarking.

Weaknesses: Collecting 50k data points for one model family costs about 24 GPU hours per model for inference. Adversarial evasion of detection is not sufficiently addressed. The assumption that a family’s errors are consistent may break down when models differ substantially in scale, training data, or architecture. Using correctness only seems to unnecessarily restrict the capability of the methodology. 

Strengths: Extensive red-teaming results are provided during the discussion period. The adversarial setting has been addressed in this period, but perhaps robustness issue still remains. Scalability issue has been addressed during the rebuttal period with a proper ablation study. 

Most of the issues raised in the rebuttal process has been adequately addressed by the authors. I agree with reviewers ykmK and 3iae that this paper proposes an interesting new problem formulation: identifying the model *family*, and proposes an innovative solution. The experiments are also extensive. The major remaining issues is the computational complexity, but I agree with the authors that this is a one-time cost and can easily be mitigated as shown in the ablation study in the rebuttal.
Decision: Accept (spotlight)