Can Watermarks be Used to Detect LLM IP Infringement For Free?
Authors: Zhengyue Zhao
Xiaogeng Liu
Somesh Jha
Patrick McDaniel
Bo Li
Chaowei Xiao
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
watermark
model copyright
model infringement detection
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The powerful capabilities of LLMs stem from their rich training data and high-quality labeled datasets, making the training of strong LLMs a resource-intensive process, which elevates the importance of IP protection for such LLMs. Compared to gathering high-quality labeled data, directly sampling outputs from these fully trained LLMs as training data presents a more cost-effective approach. This practice—where a suspect model is fine-tuned using high-quality data derived from these LLMs, thereby gaining capabilities similar to the target model—can be seen as a form of IP infringement against the original LLM. In recent years, LLM watermarks have been proposed and used to detect whether a text is AI-generated. Intuitively, if data sampled from a watermarked LLM is used for training, the resulting model would also be influenced by this watermark. This raises the question: can we directly use such watermarks to detect IP infringement of LLMs? In this paper, we explore the potential of LLM watermarks for detecting model infringement. We find that there are two issues with direct detection: (1) The queries used to sample output from the suspect LLM have a significant impact on detectability. (2) The watermark that is easily learned by LLMs exhibits instability regarding the watermark's hash key during detection. To address these issues, we propose LIDet, a detection method that leverages available anchor LLMs to select suitable queries for sampling from the suspect LLM. Additionally, it adapts the detection threshold to mitigate detection failures caused by different hash keys. To demonstrate the effectiveness of this approach, we construct a challenging model set containing multiple suspect LLMs on which direct detection methods struggle to yield effective results. Our method achieves over 90\% accuracy in distinguishing between infringing and clean models, demonstrating the feasibility of using LLM watermarks to detect LLM IP infringement.

Published as a conference paper at ICLR 2025
CAN WATERMARKS BE USED TO DETECT LARGE LANGUAGE MODEL INTELLECTUAL PROPERTY IN- FRINGEMENT FOR FREE?
Zhengyue Zhao 1 Xiaogeng Liu 2, Somesh Jha 2, Patrick McDaniel ,, Li 2, Chaowei Xiao 1.3
1 University of Wisconsin-Madison UIUC 3 NVIDIA
ABSTRACT
The powerful capabilities of LLMs stem from their rich training data and high- quality labeled datasets, making the training of strong LLMs a resource-intensive process, which elevates the importance of IP protection for such LLMs. Compared gathering high-quality labeled data, directly sampling outputs from these fully trained LLMs as training data presents more cost-effective approach. This prac- Shce-where suspect model is fine-tuned using high-quality data derived from these LLMs, thereby gaining capabilities similar to the target model-can be seen as a form of IP infringement against the original LLM. In recent years, LLM wa- termarks have been proposed and used to detect whether text is Al-generated. Intuitively, if data sampled from a watermarked LLM is used for training, the resulting model would also be influenced by this watermark. This raises the ques- tion: can we directly use such watermarks to detect IP infringement of LLMs? In this paper, explore the potential of LLM watermarks for detecting model infringement. We find that there are two issues with direct detection: (1) The queries used to sample output from the suspect LLM have significant impact on detectability. (2) The watermark that is easily learned by I.LSs exhibits instability regarding the watermark's 's hash key during detection. To address these issues, we propose LIDet, detection method that leverages available anchor LLMs select suitable queries for sampling from the suspect LLM. Additionally, it adapts the de- tection threshold to mitigate detection failures caused by different hash keys. To demonstrate the effectiveness of this approach, we construct a challenging model set containing multiple suspect LLMs on which direct detection methods struggle to yield effective results. Our method achieves over 90% accuracy in distinguish- ing between infringing and clean models, demonstrating the feasibility of using LLM watermarks to detect LLM IP infringemen
INTRODUCTION
With the development of large language models (LLMs) and the increased scale of both model parameters and training data, substantial investments and costs have been introduced for collecting data and training the models (OpenAl 202T Achiam et al. (202)). Although model owners typically offer only public APIs to protect their intellectual property (IP). these models remain vulnerable: stealers can query the model's 's APIs to obtain high high-quality data and finctune their own model with such queried data (Chiang al. 2023 Taori al. (023). As result, stealers obtain models that perform similarly to the source LLMS in general or specialized tasks, thereby stealing the capabilities of the source LLMs and introducing IP Infringement.
To address it, key step is to detect model IP infringement. Despite this problem having been explored in the traditional classification tasks (Tramer al. 2016 Tia et al. 2021 Lukas et al.) 2021 Cos 2022). IP infringement for LLMs faces two challenges. First, LLMs are designed as foundation models for diverse tasks, meaning stealers can sample data from any domain, making difficult to determine the specific data used by the stealer. Second, the stolen data is typically used to fine-tune an existing base model rather than train a model from scratch. This introduces additional complexity in detecting model infringement, as the fine-tuned model retains the characteristics of both the original model and the new data.
https: httpaa////..t.i..o/////egeeeheet http://wwwepppeeeeo.oooooononnnn

---

Published as a conference paper at ICLR 2025
Detector
Infringement Detection
Source LLM
candidace quar
won Arriing wewnett
Materwarking
anchor LLNG
Model Stealing
Stage 1 Quary Selection
API
Stealer
Lote
Math
Narala eipstiight
archor dids [w/o wabermark)
Cide
domain speciic
Stage Adaptive Detaction
Providing
API
sampled watormano
collected dotsset for tuning
Fine-tuning
Base LLM
Suspect LLM
Infriinement
Overview of LLM IP infringement and its detection with decoding-based LLM water- marks. The Stealer aims to steal the source LLM by sampling with its queries and using them to tune its base LLM. While the Detector aims to detect whether suspect LLM tuned with data sam- pled from the source LLM. It can access an API provided by the Stealer but does not know the base LLM and the dataset for tuning. To this end, detection method including selecting proper queries for sampling the suspect LLM and adaptive detection with anchor data is proposed to detect IP in- fringement better.
Figure 1:
An intuitive approach for model IP infringement detection is to inject special distribution into the source model's output, such that a suspect model trained on this data would learn a special distribution, which be used for detection, which is also known as watermarking (He et al,, 2022a Zhao et al. 2022 2023: He et al.) 20226). Recently, watermarking has mainly been applied detect particular text generated by specific LLM by injecting watermarks into the output during inference (Kirchenbauer et al. 2023 Zhao et al. 2024 Kirchenbauer et al. 2024a Kuditipudi et al. 2024]. This watermarking technique works by dividing the vocabulary into "green list" and "red list", and adding bias to the logits of different tokens. This increases the frequency of tokens from the green list in the output text so that the watermarked text can be detected with fixed threshold of z-score from z-test of the green ratio. Since watermarks can change the distribution of output, natural question emerges: "Could such watermarks be used to detect LLM model infringement for free?"
In this paper, we propose a watermark-based LLM model infringement detection algorithm, LIDet (LLM Infringement Detection). Our journey begins by identifying the key challenges of using LLM watermarks for model infringement detection, compared to detecting generated text.
The first challenge is attenuation of watermark detectability caused by domain mismatch between training and detection. As LLMs are foundation models designed for diverse domains, stealers can extract data from arbitrary domains fine-tune their suspect models, such code generation and mathematics. The stealer can avoid being identified by accessing the API of the target model using anonymous accounts. Consequently, the detector is unaware of the specific domain data used by the stealer. Thus, the detector needs to construct detection data. However, if the detection data has large distribution mismatch with the training data, it leads an attenuation the frequency of watermarked tokens in the text, weakening detection capability, as shown in Figure 2a
The second challenge is the guarantee of watermark learnability results in a green ratio mismatch, leading to the failure of watermark detection for some hash keys. Unlike watermarking for detecting generated text, where the watermark is directly injected, watermarking for suspect LLM infringe- ment detection involves the watermark being learned through the training process on unknown sam- pled data. The detectability of the watermark in the suspect model thus highly dependent on its learnability. To ensure the watermark embedded in the source model is also learned by the suspect model, careful design of the watermarking algorithm is essential. While text watermarking typically uses high n- n-gram lengths (e.g., 4) to ensure robustmess against reverse engineering (Kirchenbauer jet al. 2024b), this reduces the learnability the watermark (Gu et al., 2004a),, as also shown by experiments in Figure 2b To address this, intuitive approach improve the learnability
2

---

Published as a conference paper at ICLR 2025
20.
nid
2-score
frequency
15-
2-
10-
1-
5-
same query (i.d query
000 query
3
0:24
0.25
0.26
0.27
queries for sampling suspect LLM
for watermark injection
green ratio
(a) Queries for detection. Same & i.d queries are in-domain while
(h) Hyper-parameter n KGW watermark: the hash for green list is calculated with hash key and previous tokens' ids.
(c) Histogram actual green ratios of KGW watermark with 100 ran- dom hash keys.
OOD queries are out-of-domain queries of queries for tuning.
Figure 2: (a) and (b) show z-scores of text sampled from the suspect LLM tuned with KGW- watermarked data. A higher z-score represents more watermarked green tokens in the suspect LLM,, indicating better Iearnability of watermarks and detectability of suspect LLMs. (c) demonstrates greater n leads the actual green ratio being unstable from the preset green ratio 0.25.
of the text watermark to decrease the n- n-gram length, for example, setting it to 1. However, this introduces another problem: using low n-gram length causes the actual green ratio to be heav- ily influenced by the random hash key, as shown Figure 2c This mismatch between the actual green ratio and the preset green ratio for partitioning the green list makes the z-score-based detec- tion with a fixed threshold fail, which means that even with a valid text watermarking scheme like KGW (Kirchenbauer et al. 2023), the watermark may be detectable with some hash keys but not with others.
To address these challenges, proposed method, LIDet, contains two designs as briefly demon- strated in Figure [ (1) To mitigate the attenuation detectability introduced by mismatched train- ing and detection domains, we apply several anchor LLMs (which be different from the suspect LLM) to help select such queries that are more likely to sample texts containing more diverse and evenly distributed tokens. Compared with those queries that bring texts with highly repetitive or centrally distributed tokens, these queries could get responses containing effectively watermarked tokens with higher probability. (2) To solve the mismatched green ratio brought by watermarks with a low n for better learnability, we replace the fixed threshold of z-score- based detection with an adaptive threshold by estimating an actual green ratio from pre-gene pre-generated anchor data. Meanwhile, we take the cross-model generalization into account during query selection to ensure the green ratio of anchor data generated by anchor LLMs can closer to the natural green ratio un-watermarred text. Therefore, the reference green ratio can help better discriminate between the infringed suspect LM and the clean one.
We conduct experiments marks such as KGW (Kirchenbauer et al. 2023) and Unigram (Zhao et al. 2024), as well as different source LLMs, suspect LLMs, and datasets for tuning suspect LLMs. Results reveal that our proposed detection method increases the reliability of discriminating against unauthorized distilling of source LLMs and further demonstrates successful cases of using LLM watermarks for defending against LLM's model infringement. Specifically, our method achieves the detection accuracy of over 90% in the cross-domain detection on a challenging model set containing suspect LLMs with multiple settings while the vanilla detection struggles to provide effective results.
black-box scenario from the detector's perspective with LLM water-
Our contributions can be summarized as follows:
We explore the feasibility of using existing LLM watermarks detect model infringement and analyze the limitations of vanilla detection methods in this task.
We propose simple yet effective detection method, LIDet, to enhance LLM model in- fringement detection, significantly improving the detectability of model infringement.
Through extensive experiments under realistic threat models, demonstrate that LLM watermarks with our proposed detection methods, can effectively detect model infringe- ment and thus protect the copyright of source L.Mss.
3

---

Published as a conference paper at ICLR 2025
2
BACKGROUND & PRELIMINARY
2.1 LLM WATERMARKS AND DETECTION
KGW. KGW (Kirchenbauer al. 2023) modifies the frequency certain tokens in the generated text by adding bias to the logits of specific tokens during LLM decoding. Specifically, during each decoding step, KGW randomly divides the tokenizer's s vocabulary table into a green list and a red list based on predefined green list ratio and hash key using a hash function applied to the preceding n tokens: frash (E, I-,,,.,.., z-n).
Logits of tokens in the green list are increased by a fixed hyperparameter 6, thus raising the probability that these green tokens will be sampled during decoding. Detection of KGW works by statistically analyzing the proportion of green list tokens in the target text (with [T] tokens in total). After determining the number of green tokens |scl in the target text, a z-test is performed to compute the z-score or p-value to evaluate whether the proportion of green tokens | significantly exceeds the preset green ratio 7o: z ([sc Yo /T)/ / - 7o)[ Typically, if the z-score exceeds predetermined threshold a (such as 4.0), the text is considered to be watermarked.
2024) watermark are similar to KGW. The key difference is that Unigram does not use the preceding tokens to compute the hash function; instead, it divides the green list solely based on the hash key Ç. As a result, the green list of Unigram is static. Compared to KGW (n 2 1), Unigram (n = 0) significantly enhances the robustness and detectability of the text watermark. However, this also increases the influence of the hash key on green tokens, making the actual green ratio deviate from the preset green ratio y.
Unigram. The generation and detection of the Unigram (Zhao et al.
2.2 PROBLEM STATEMENT
The process of LLM model stealing and infringement detection can be divided into the following stages as illustrated in Figure 1 (1) The stealer constructs a query set Q" intended for fine-tuning their base model. By accessing the source LLM's API, they sample the corresponding responses R² from the query set, thus creating dataset D for instruction fine-tuning. (2) The stealer applies the constructed training dataset mD" fine-tune their own base model, resulting @ruspect, They then provide the API of the suspect LLM. (3) The detector accesses the API of the suspect LLM ganspect and uses a series of queries Q" to sample output text R from the suspect LLM's API. (4) The detector analyzes the concatenated output text RD from the suspect LLM @suspect to check for the presence of the source LLM's watermark, thereby determining whether the suspect LLM is fine-tuned using data sampled from the source LLM.
suspect LLM
2.3 THREAT MODEL
We consider two opposing roles in the task: the Detector (the owner of the source LLM and the entity responsible for detecting infringement) and the Stealer (the owner of the suspect LLM).
Stealer. The Stealer is the owner of the suspect LLM, who attempts to steal the capabilities of the source L.LM by fine-tuning their base model using data sampled from the source LLM. The Stealer can only sample data through the source LLM's API but has the flexibility to choose any query for sampling. We consider the realistic scenarios where the Stealer can use different IPs and anonymous accounts to query the source LLM so that the detector can not know what data are queried. In the context of stealing target LLM, we treat this fine-tuning as the last training operation of the suspect LLM. The Stealer would also not try to paraphrase the sampled text considering the paraphrasing will change the stealing target from the source L.M to the paraphrasing model. After fine-tuning, the Stealer provides the API of the suspect I.IM to the public, without revealing the model structure or training data.
Detector. The Detector is the owner of the source LLM and aims to determine whether the suspect LLM has infringed upon the source model. The Detector has full knowledge of the source LLM, including model weights and watermarking configurations. However, during detection, the Detector can only access the suspect LLM's API and is unaware of its model structure or the data used for training. The Detector must rely solcly on the output text generated by the suspect LLM to determine if it has infringed upon the source I.I.M.
4

---

Published as a conference paper at ICLR 2025
Table 1: Detectability of suspect models with different queries. A higher z-score or lower p-value indicates that the suspect LLM is more likely to be watermarked.
Detection
w/ watermark
wio watermark
Watermark
Query
2-scure
p-value
2-score
p-value
Unigram
27.6
I 167
0.731
2.3e-1
Truining Query
KGW
5.61
1,04c-8
0.406
3.4c-1
Unigram
4.9c-134
0.534
3.00-1
Lid Query
KGW
1.5e-7
.0.281
6.1e.1
Unigram
11.2
3.5e:29
-3.67
9.9e-
OOD Query
KGW
1.21
1.1e-1
-1.79
9.6e-1
7.4000
2:000r
7580 10080 35300 1i5ce 2006 GGrrerated
2049 1500
1akce 37500 2009
GGnrerated
(a) Unigram-
Unigram-watermatten watermarked suspect LLM.
(b) KGW-watermarked suspect LLM.
Figure 3: Z-scores of generated text sampled with different out-of-domain query sets.
3 METHODOLOGY
The difference between the directly injected text watermark and the learning-based LLM's model watermark brings two challenges that hurt the effectiveness of detecting infringing suspect LLMs: (1) The scenario where the detector can not get training data of the suspect LLM means there is a domain mismatch between training and detection data, resulting in the attenuation of watermark detectability. (2) The demand for learnable watermarks limits the n of watermarks to small, leading to green ratio mismatch between the actual one and the preset one, which further causes failed detection. We first discuss these two challenges and then introduce our L.Det for reliable detection address them.
3.1 ATTENUATION OF WATERMARKS DETECTABILITY IN DOMAIN MISMATCH
The detection of suspect LLM involves two key steps: sampling text from the suspect LLM for detection, and detecting the watermark in the text. this case, an important consideration is to check if the watermark is detectable under the threat model, i.e. whether watermarks of suspect LLMs can be detected without knowledge of the training data. To answer this question, we conduct a preliminary experiment first fine-tune suspect L.M with coding data (Luo et al. 2024) sampled from the source LLM. Then three different query sets are applied to sample text from the tuned suspect LLM: the same queries for training the suspect model (the first-row of Table []. queries different from training queries but in the same domain (also coding queries, the second-row of Table D.. and the out of domain queries (general queries sampled from Alpaca (Taori et al. 2023).
the third-row of Table
Results from Table reveal that the detectability of LLM watermarks attenuates along with the in- crease of difference between detection queries and training queries. The z-score of the sampled text significantly lower compared to when the detector knows the suspect L.MM's training queries, in- dicating reduced detection capability. Intuitively, this occurs because the watermark in the suspect LLM is learned from the training data, so the closer the sampled text to the training data during detection, the better the detection performance.
Figure 3 further illustrates the detectability of different out-of-domain detection queries. It indicates that though the detected texts are all sampled with general query sets, the detectability of these cases varies. The curve of text sampled with less-detectable queries (e.g. query set of Figure (3b) shows that the z-score sometimes significantly drops while the number of tokens increases, which
5

---

Published as a conference paper at ICLR 2025
Table Learnability of watermarks with different n. A higher z-score or a lower p-value indicates that the watermark is more learnable during the tuning process of the suspect LLM.
w/ watermark
w/o watermark
Watermark
2-score
p-value
2-scere
p-value
Unigram
n=0O
24.6
4,9c-134
0.534
3.0e-1
5.12
1.5c-7
-0.281
6.1e-1
2.11
1.7o-2
-0.196
5.8c-1
KGW
1=4
0.20
4.2o-1
-0.061
5.20-1
0.32
0.29
0.24
0.20
of Notera
Generated
Tokan
(a) Distribution of actual green ratio(b) z-scores with higher green ratio.(c) z-scores with lower green ratio. with y = 0.25 with varies hash keys.
Figure 4: A small can lead to failed detection. (a) small n makes the actual green ratio deviate from the set of 0.25, (b) z-score of unwatermarked text easily surpasses the fixed threshold when the actual green ratio is higher than y (c) z-score of watermarked text is smaller than the fixed threshold when the actual green ratio is lower than y..
demonstrates that some queries make negative contributions to detecting watermarks. In contrast, query sets such as and from VigureBbbsttin demonstrate enough z-scores for watermark detection. We find that those queries causing z-score dropping usually have low diversity responses in tokens, which are short length or have high repetition tokens, thus leading to a bad detection. Therefore, it is crucial to screen out unhelpful queries from an out-of-domain general query set to detect LLM infringement without in-domain queries.
3.2 GREEN RATIO MISMATCH OF LEARNABLE WATERMARKS
The learnability of L.LM watermarks decides whether watermarks of source L.MMs' outputs can be transferred to the suspect L.M during fine-tuning. Obviously, the learnability of watermarks highly related to the pepeer-paramter in KGW. As shown in Table 2 watermark with a greater n results a smaller z-score and higher p-vauue, which means it is less learnable for suspect LLMs. Intuitively, when the watermark is transferred from the source LLM to the suspect LLM, the suspect LLM essentially learns the distribution shift introduced by the n-gram-base watermark. When n increases, the distribution of n- R-grams in the training data becomes difficult for the suspect LLM to sufficiently learn the n-gram watermark with the same amount of training data. This implies that smaller values of n (n 0, infringement, as they make it easier for the watermark to be learned and subsequently detected.
dispersed, making it more
effective for detecting model
However, the learnable watermark with small n leads another problem: the partition of the green list is rather sensitive to the random hash key &. Figure 4a illustrates the distribution of the actual green ratio of unwatermarked texts with random hash keys, which reveals that watermarks with small n can result in mismatch between the actual green ratio and the preset green ratio y during partition. Considering that the calculation of the z-score for detection dependent on the green ratio y, such a mismatch could be disastrous for watermark detection. When the actual green ratio is lower than the set 7. the calculated z-score will be lower than the actual value. In this case, the z-score of text with the watermark might fall below the detection threshold, leading to higher false negative rate (as shown Figure (4c). Conversely, when the actual green ratio higher than the set 7, the calculated z-score will be higher than the actual value. This result in won-watermrrked text having a z-score above the detection threshold (as shown in Figure 4b), thereby increasing the false positive rate.
These results highlight a trade-off between the learnability of LLM watermarks and the stability of the green ratio with respect to the hyper-parameter when detecting model infringement. To

---

Published as a conference paper at ICLR 2025
ensure the watermark's learnability and detectability together, it is crucial replace the original fixed threshold of z-scores with an adaptive threshold which can compensate for the instability of the actual green ratio brought by the small n.
3.3 LIDET: RELIABLE DETECTION FOR LLM INFRINGEMENT
Our proposed LIDet includes two key stages: (1) selecting queries for sampling detection text from the suspect LLM with the help of a set of anchor LLMs, and (2) detecting watermarks from the detection text with adaptive threshold of 7-score. We address the challenge introduced by domain mismatch by selecting queries that can promote responses with diverse and vrequency-balancee kens, regarded as the principle of token entropy. Besides, we mitigate the problem caused by mis- matched green ratios via selecting cross-model generalizable queries and detecting the watermark with an adaptive threshold fill in the z-score gap by estimating the actual green ratio.
to-
Anchoring Treated Query Selection. The queries are selected by two principles: (1) the token entropy their responses, and (2) their cross-model generalizability. To identify the property of each candidate query, we introduce set of anchor LLMs amchhrr) ) Pu first. These anchor LLMs are used help select a proper query set QD from candidates Q (q) 92,...: QK J. By sampling responded text with all K candidate queries from M anchor LLMs, set of anchor responses IR is constructed:
R
0 anchor +Q) (r)
rk
M
Based on the anchor responses, we define the token entropy H of the response for each query qu as the frequency balance degree of contained tokens across all anchor LLMs, shown in equation where т refers to tokens in the corresponding response r and P(r) stands for the frequency of token т. Query with higher H usually can promote a response containing more tokens and the distribution of tokens is more even, making it more likely to include watermarked tokens.
H(gs) -- M
Σ TEPI
P(T) log P(T)
Then we introduce the concept of the cross-model generalizability 1) of queries. Specifically, n of query q is defined as the negative variance of the statistical green ratio 7 of the corresponding response r sampled from all anchor LLMs, demonstrated in equation [3] The green ratio *k of response 7 is calculated by the number of green tokens |&a| and the number of all tokens [rg'. A higher 7) indicates the green ratio of the response sampled from different anchor LLMs is closer, which means that the corresponding query is more likely to produce texts with similar green ratios across various LLMs. Therefore, we could avoid queries that yield significantly different green ratios when sampled from different models by selecting queries with smaller thus providing a better estimation of the actual green ratio of the suspect L.LM the next stage.
(4q)) -Varmm...... (71")
k = 18c |r
(3)
According the token entropy H and cross-model generalizability of all candidate queries, select detection queries following the priority score given in equation Queries with higher token entropy and higher cross-model generalizability are selected first to sample the suspect LLM get detection texts.
H(q}) - min H(q))
n(qx) min 77(k))
v (k)
(4)
max H ((q)) min H (qi)) max ( min 17
Detection with Adaptive Threshold. By the above stage of query selection, the query set QD is built. The texts for watermark detection are sampled from the suspect LLM with @D. Specifically, we sample corresponding responses from the suspect LLM with each query in QD and then concate- nate them together for the later detection process. For z-score-based detection, instead of applying the fixed threshold o, we propose watermark-adapter threshold Reh. correct the mismatch be- tween the actual green ratio and the preset green rate "/o The adaptive threshold works by filling in the gap of z-scores caused by the mismatched actual green ratio. To estimate the actual green ratio, we collect responses sampled from anchor LLMs with Q" as anchor data. The statistical green ratio Pa of the anchor data is treated as approximation of the actual green ratio.
7

---

Published as a conference paper at ICLR 2025
PG 70
(T] +
(5)
Yo %0)
Then the adaptive threshold can be obtained by equation 5 where [2'] is the number of tokens in the text for detection. Figure 4b and Figure 44 demonstrate the curve of adaptive threshold 7th as well as the fixed threshold is clear that while the actual green ratio mismatches with the green ratio for partition, a fails to discriminate the watermarked and clean model but 21h is successful in classifying the two models.
In summary, the reliable detection of the suspect model with LIDet contains the following processes: (1) sample the anchor models with all queries to get corresponding responses; (2) calculate green ratio y and token entropy H of the response of each query (K and anchor model A Mictorr (3) calculate the priority score of each query qu following equation[4 (4) sample text from the suspect LLM with queries of higher priority v and then calculate μlg of equation 3o with anchor data (text sampled from anchor models with the same queries); (5) concatenate the sampled text and then calculate z-score of all sampled text by = = (Iscl - To / - % and calculate 2th following equation Zeh: then the suspect L.LM is considered an infringing model.
If 2 >
Though LIDet requires the detector to sample all anchor LLMs using a query set to obtain the outputs corresponding to all queries, the resulting query subset and green ratio are applicable to any suspect LLM. As long as the watermark configuration of the source LLM is determined, there is no need to re-filter the query subset when detecting new suspect LLMs.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Models & Datasets. We conduct experiments with multiple models and datasets for a more com- prehensive study the detectability of LLM watermarks in model infringement scenarios. Specif-
ically, we use Allama-2-chat-77 (Touvron et al. (023) and Llama-3-Instruct-8B (Metal 2024) as
source LLMs, and Bloom-7b (Le Scao al. (203)) and Mistral--nnttrut-76 (Jiang et al. 2023) as
the base models for the suspect LLMs. These models differ in structure and vocabulary size. We also serve Llama-2-chat-
as anchor models of LIDet. For the queries used to sample fine-tuning data, we select two common domains: code generation (Evol- Instrnnt-touooe (Luo) jet al. (024)) and math problems (GSM8k (Cobbe et al., (02T)). These domains are widely used and differ from general-purpose tasks, helping to assess detection capabilities with black-box access to data. During detection, we use queries from Alpaca dataset (Taori et al.) 2023) to sample text from the suspect LLMs, simulating the scenario where the training data is unknown.
and -lama-3-Instuuce
Construction of Model Set. Detecting LLM model infringement is a binary classification prob- lem, so using only the z-score metric might not be sufficient reflect the effectiveness of detection. To address this, we construct a model set containing positive samples (trained with watermarked data) and negative samples (trained with un- wattrrmmmrrked data). Specifically, train 320 suspect LLMs based on different configurations, including source LLMs (Llama-2, Llama-3), suspect base models (Bloom, Mistral), training queries (code, math), and watermark methods (KGW, Unigram, no watermark) with 10 random hash keys. Among these. 160 are positive samples (w/ watermark) and 160 are negative samples (w/o watermark). Each positive sample could correspond negative sample with the same source LLM, suspect base model, training query, and hash key.
Baseline & Metries. We use the vanilla detection method directly as a baseline. Specifically, vanilla detection applies fixed z-score for detection and sample texts from the suspect LLM with randomly selected queries. The metrics include commonly used measures in binary classification tasks: TPR (True Positive Rate), TNR (True Negative Rate), and ACC (Accuracy). Additionally, we propose metric called Detection Successful Rate (DSR). For a set of samples in the model set (containing positive sample and its corresponding negative sample), if both samples are successfully detected, the sample set is considered a successful detection. DSR is the proportion of all successfully detected sample sets out of all sample sets in the model set.
Details of Watermarking, Training & Detection. Watermarks contains Unigram (n = 0) and KGW (n 1) with both 7 0.25 and 5 3.0. Training data of suspect models are sampled from source models with 5k queries in the code or math dataset. Suspect models are tuned with LoRA (Hu al. 2021), with batch size of 32, epochs of 4, and constant learning rate of 10-1 For
8

---

Published as a conference paper at ICLR 2025
Table 3: Detection results of all suspect models in the model set with different source models and base models of suspect LLMs. Queries for detection are all sampled or selected from Alpaca, while the training data of suspect LLMs are sampled from coding or math domain.
Suspeat I.M
Watermark
Source LLM
Method
Bloom
Mistral
avg.
ACC
TPR
TNR
ACC
DSR
ACC
DSR
Baseline
0.60
0.70
0.65
0.30
0.55
0.70
0.625
0.25
0.638
0.275
Llama2
LIDet
0.95
0.90
0.925
0.90
1.0
0.95
0.975
0.95
0.95
0.925
Unigram
Baseline
0.90
0.35
0.625
0.25
0.85
0.30
0.575
0.15
0.60
0.20
Llama3
LiDet
1.0
0.90
0.95
0.90
1.0
1.0
1.0
1.0
0.975
0.95
Baseline
0.55
0.70
0.625
0.25
0.40
0.70
0.55
0.10
0.588
0.175
Llama2
L.Det
0.85
0.875
0.80
0.85
0.85
0.85
0.75
0.863
0.775
KGW
0.675
0.35
Llama3
0,90
Baseline
0.65
0.30
0.60
0.20
0.625
0.25
LIDet
0.913
0.85
0.931
0.90
0.922
0.875
Table Detection results of all suspect models in the model with different queries for detection.
Query for Detection
for Training
Alpuca
Math
TPR
ACC
DSR
TPR
TPR
Baneline
Unigran
Code
0.80
0.50
0.65
0.30
0:975
0.65
0.813
0.625
Math
0.65
0.525
0.588
0.175
0.90
0.75
0.825
0.65
KGW
Math
0.45
0.75
0.50
0.20
0.925
0.675
0.50
0.60
Unigram
Code
1.0
0.925
0.963
0.925
1.0
1.0
1.0
1.0
0.975
Code
KGW
Math
0.875
0.90
0.888
0.825
1.0
0.95
0.975
0.95
detection, we sample text from suspect models until the total number of sampled tokens achieves 20k. The threshold of z-test detection is set to 4.0 for Unigram and 2.0 for KGW.
4.2 RESULTS
We demonstrate the main detection results of suspect LLMs in the model set in two dimensions. First, evaluate the detection results from dimension of different source LLMs and suspect LLMs, as shown Table Then, we assess the detection from a dimension of different training queries and contrast them with in-domain queries for detection, shown in Table
Table 3 presents the results of watermark detection on different source and suspect LLMs. Our method detect LLMSs containing watermarks with ACC of 92.2% and DSR of 87.5% average. contrast, vanilla detection almost fails to identify watermarked models, with a DSR of only 25%. Compared KGW, Unigram demonstrates better detection performance with about 10% higher DSR. Intuitively, the static green list in Unigram helps the suspect LLM learn the token distribution of the watermark more effectively during training. Generally, our method remarkably improves the detection performance of LLM infringement, demonstrating that LLM watermarks can be applied for LLM's IP protection even in a challenging scenario.
4able/shoows the impact of different training and detection queries on the detection of suspect mod- els. The column corresponding to Alpaca represents the out-on-domain detection scenario, while the columns for Code and Math represent the in-domain detection scenario for suspect models training with Code and Math data respectively. For the out-of-domain scenario, our method shows remark- able results where DSR for Unigram surpasses 90% and for KGW surpasses 80% across two training domains. For the in-domain scenario, our method achieves 100% DSR for Unigram and over 95% DSR for KGW watermark. In contrast, vanilla detection only has rather low ACC and DSR, in- dicating that the green ratio mismatch can still significantly impact watermark detection even in in-domain scenario.

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on the modification of threshold and the selection of detection queries.
4.3 ABLATION STUDY
We conduct ablation studies to illustrate the ef- fectiveness of each part of LIDet. Results are shown in Table
Metric
Watermark
Method
ACC
DSR
LiDet
0.956
0.938
Adaptive Threshold. Results demonstrate that after replacing the adaptive threshold with the original fixed threshold, the ACC and DSR are significantly dropped to smaller than 60% and 25% respectively, showing that the adaptive threshold is the most critical design.
adaptive threshold (Axed)
0.583
0.163
query selection (random)
0.938
0.875
Unigram
query selection (1) oaly)
0,963
0.925
query selection (Hl only)
0.944
0.913
query selection (1) reverse)
0.90
0.80
query selection roverse)
0.869
0.738
LIDet
0.881
0.813
adaptive threshold (fixed)
0.60
0.238
Query Selection. For the impact of query se- lection, we replace the strategy with random, higher cross-model generalizability (n only), higher token entropy (H only), lower cross- model generalizability (1) reverse), and lower token entropy (H reverse) respectively. Results
query selection (random)
0.856
0.725
KGW
query selection (1) only)
0.875
0.80
query selection (H only)
0.888
0.788
query selection (1) reverse)
0.794
0.613
query selection (H reverse)
0.788
0.575
indicate that both principles of cross-model generalizability and token entropy are important for selecting proper queries for detection. Especially, the DSR of reversed selection (n reverse and H reverse) drops over 20% in the worst case, which demonstrates that inappropriately selected queries may significantly reduce the detectability.
5 RELATED WORKS
Model Infringement. Model infringement which is also known model imitation or model ex- traction, aims to steal a target model through its API by training with sampled data (Tramèr et al.] 2016 Orekondy al., 2019 Wallace et al. 2020 The process of model infringement is similar to knowledge distillation (Hinton et al. 2015 Gu et al. 2024b) but only has black-box access to the target model. Both classification models and generative models are vulnerable to such stealing (He) et al. 2021 Krishna et al. 2020 Szyller et al. 2021). Recently, tuning LLMs with data sampled from strong models such as GPT4 has become usual way to improve the ability of open-source LLMs (Taori et al. 2023 Chiang et al. 2023 Luo et al. 2024).
Detection of Model Infringement. The most common way to detect model infringement is the model watermark (Uchida et al.] [017]] [Zhang et al. 2018). For language models, watermarks are usually added to the outputted texts by injecting special words or linguistic features (Cos) 2022 He et al. 2022al Zhao al. 2022 He et al., 2022b). Especially, Zhho et al. (023)) applied bias- based logit level watermark for generative language models, but it requires white-box access to the suspect model to detect special signals from the probability vector. Recently, [ander et al. (2024) studied the radioactivity of LLM watermarks, providing innovative perspectives to detect LLM wa- termarks from the suspect model tuned with watermarked texts. While it studies an extensive range of training and watermarking configurations, some key variables of detecting LLM infringement such as source/suspeet models, hash keys of watermarks, and training/detection queries together with caused problems are not studied yet. Besides, it also assumes access to the same domain (at least i.i.d) of training data, which may not align with the practical setting. Therefore, whether wa- termarks can be used for detecting LLM infringement under a practical threat model has not been completely solved. We fill these gaps in this paper and provide targeted method for this task to demonstrate the success of detection in more challenging scenario.
6 CONCLUSION
In this paper, we explored the possibility of using existing LLM Watermarks to defend against Model IP infringement, thereby protecting the copyright of LLM models. Based on the fundamental differences between watermarking for models and watermarking for text, we propose LIDet for detecting watermarks from the perspectives of watermark detectability and learnabllity under black- box conditions. The evidence shows that even in scenarios where the target model and training set are unknown, our method can still determine the existence LLM infringement with a high degree of accuracy.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is partially supported by U.S. Department of Homeland Security under Grant Award Number I7STQAC00001-07-000 and A12050 program at Schmidt Sciences. Patrick is partially sup- ported by the National Science Foundation under Grant No. CNS-2343611. Bo Li is partially supported by No. 1910100, No. 2046726, NSF AI Institute ACTION No. 2S-2222,77, DARPA TIAMAT No. 80321, the National Aeronautics and Space Administration (NASA) under grant No. 80NSSC20M0229, ARI. Grant 1991NNF23-2-0137 Alfred P. Sloan Fellowship, the research grant from eBay, Al Safety Fund, Virtue AI and Schmidt Sciences. Somesh Jha is partially supported by DARPA under agreement number 885000, NSF CCF-FMITF-18839788 and ONR N00014-21-1- 2492.
DISCLAIMER
The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security, the National Science Foundation, DARPA, NASA and Schmidt Sciences.
REFERENCES
Cosine model watermarking against ensemble distillation. 36:9512-9520, Jun. 2022. doi: 10. wtt9:/aai.v669..21112. URL https:://jjssaaaa index. http://mneexppppAAA///tttttooo
view/21184
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
-3-00-3-3000 30 -0-3000--000 -000viicaa///
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14.1668,, 2021. URL
https::/arre
org/ abs 2210. 14168
Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of water- marks for language models. In The Twelfth International Conference on Learning Representa- fions, 2024a. httpttttts://peeeeeeee
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. ICLR, 2024b. URL
5hCof7IBZZ
Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model extraction and adversarial trans- ferability, your BERT vulnerable! Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tus Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 202/ Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 2006-2011 Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v/20011 m0t.::5/111222mmmmmmmmmmaa -naccl-main. URL https://acl www.ellanthoolosoo www.oooo..ooog/o/oo naacl main. 161
Xuanli He, Qiongkai Xu, Lingjuan Lyu, Fangzhao Wu, and Chenguang Wang. Protecting intellec- tual property of language generation apis with lexical watermark. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 107661 2022a.

---

Published as a conference paper at ICLR 2025
Xuanli He, Qiongkai Xu, Yi Zeng, Lingjuan Lyu, Fangzhao Wu, Jiwei Li, and Ruoxi Jia. CATER: Intellectual property protection on text generation APIs via conditional watermarks. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. URL htttps://openret ew. net. fort//foumm? i= L7P3TVsOUXY,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distillling the knowledge in neural network, 2015. URL https: org/
02531
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learming Representations, 2021.
Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, and Nicolas Papernot. En- tangled watermarks as a defense against model extraction. In 30th USENIX Security Sympo- sium (USENIX Security 21), pp. 1937-1954. USENIX Association, August 2021. ISBN 978-1- 939133-24-3, URL https: www. usenix wttp:///weeeeecooeeeee/eee////ee http///ffoerreeeiooocccceeeeeee/ presentation Jia
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference Machine Learning, pp-
17061-17084. PMLR, 2023.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. In The Twelfth International Conference Learning Repre- sentations, 2024a. URL httlt: http:::///peeeeeeesseee net
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of water- marks for large language models. In The Twelfth International Conference on Learning Repre- sentations, 2024b. URL http:: /oppnr/v//peeevviiiie
net wott/oorumndd-DJIDDD....
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer. Thieves on sesame street! model extraction of bast- apis. In International Conference on Learning Representations, 2020. URL (tRL:ttss::/ /ttttpps://oprnrevieewe net/ www.mmiddddaaao 5 BYNRRFDD
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL btttgs://openneviie..rr wet.rreeiieeeneeeeeeeme
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. 2023.
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by con- ferrable adversarial examples. In International Conference on Learning Representations, 2021.
URL WRKLhttps:///peeeeee
net.
1H1
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing
Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024. URL
https::/cpenrev/ wet.:t/eoooooooonnnnnnsoseee... Li-
net. forum
Meta. Introducing meta llaaa 3: The most capable openly available 1lm to date, April 2024. URI.
https://ai.mes awt.aaa.....aam.........a....... com haamm//oog//metee 11ama- 3/
OpenAl. Chatgpt: lcrge-scale generative model for open-domain chat. httts:
com/openai/gpl 2021.
12

---

Published as a conference paper at ICLR 2025
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive. In Thirly-eighth Conference on Neural Information Process- ing Systems, 2024. URL htt.lttttpaaaxttttt org/ abs abs/24/2/222222 14904
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. Dawn: Dynamic adversarial watermarking of neural networks. MM '21, pp. 4417-4425, New York, NY. USA, 2021, Associ- ation for Computing Machinery. ISBN 978 1781455366 doi: 10.1145/347005534455599 URL
sattps://dee org/10 wttp://0.1111//// 114. 34 4485554555459
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https://githi: com tttt//tttttuute 1ab/ saab/stanfooe stanford_alpace 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307, 092888, 2023.
Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, and Thomas Ristenpart. Stealing ma- chine learning models via prediction APIs. In 25th USENIX Security Symposium (USENIX Security 16), pp. 601-618, Austin, TX, August 2016. USENIX Association. ISBN 978-1-
931971-32-4. URL http::ttt//// www uswnix technical-sse sess ons ceerssionnnseerees wttps///ppeeetteeoottotttteeeeee
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference
Multimedia Retrieval, ICMR '17, pp. 269-277, New York. NY. USA, 2017. Association for Computing Machinery. ISBN 9781450347013. doi: 10.1145/3088971.307889
URL
https:://di..or www.ddi....g/g/oo 1tt.////0/0111111311131.7111 3078977
Eric Wallace, Mitchell Stern, and Dawn Song. Imitation attacks and defenses for black-box ma- chine translation systems. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference (EMNLP), pp. 5531 Online, November 2020. Association for Computational Linguis- tics. doi: 100118653/1/2220.cmm mappp 446. URL baattpss/aalanntcclooooooo 2otpaa/tttooooyyoo//200....
Empirical Methods in Natural Language Processing
em. monlp-main.. 446
Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Ph. Stoecklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In Pro- ceedings of the 2018 Asia Conference Computer and Communicattonns Security, ASIACCS '18. pp. 159-172, New York. NY, USA, 2018. Association for Computing Machinery. ISBN 9781450355766. doi: 10.1145/3196444.116655 3196494.3196550
URL httpp:://cc..orree
1145/
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Eestillation-resistaa watermarking for model protec- tion in NLP. In Findings of the Association for Computational Linguisties: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL
org/ 2022. windings-emr.lsp 370
Xuandong Zhao, Yu- Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference Machine Learning, pp. 42187-42199. PMLR, 2023.
Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang. Provable robust water- marking for Al-generated text. In The Twelfth International Conference on Learning Representa- tions, 2024. URL http::t//////c httttss:/oppeneevieee net forum? in- Swd.d
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
In this work, we mainly focus on the factor of LLM infringement detection of source LLMs, suspect LLMs, and domains of training/detection queries. Other factors such as training configurations (e.g. data size, learning rate, and adapters) and watermarking configurations (e.g. more kinds of watermarks and watermark strength 6) have not been exploited. Considering that related works such as Gu et al. (2024a) and Sander et al. (2024) have studied the impact of these factors on the hard- label distillation of LLMs, the influence of these factors is clear enough. For instance, if the Stealer tunes their models with full-paran full-parameter training instead of LoRA, with more sampled data, the successful rate of detection will certainly increase.
Besides, we do not consider the robustness of watermark detection when facing paraphrase attacks, mixed-data tuning, or further training in this paper. In the case of "model stealing", the Stealer aims to targetedly imitate the source LLM while also improving their own models. Anyway, it is an important concern to detect model infringement with higher robustness and we leave this as the future work.
B DETAILS OF EXPERIMENTS
B. EXPERIMENTS IN SECTION 3.1
In section 3.1 we evaluate the detectability of suspect LLMs when the queries for detection and queries for training are different. Specifically, we first sample responses from the source LLM, Llama-2-7b-chat, watermark-injecto Instruct-Code. Then train the Bloom-7b on the sampled data, resulting in suspect LLM. Then we respectively use (1) the same queries from the training data, (2) other queries from Evol- Instructt Code, and (3) the Alpaca dataset to sample responses from the suspect LLM. The training configu- ration is the same as 4.1 We collect the responses and concatenate them together for each condition until the total length reaches 20k. Then these texts are used for z-texting and the z-score and p- p-value are obtained. In contrast, suspect LLM without watermarks is trained from the un-watermarked sampled data from 1aama-2-bb-hha with the same training queries. For results out-of-domain detection, we sample queries from the Alpaca dataset with different seeds.
by Unigram or KGW (n=l), with 5k queries from Evol-
B.2 EXPERIMENTS IN SECTION 3.2
control the training data and detection data i.i.d domain, i.e. both training data and detection data are sampled with queries from lono---nstuuttoode but are not the same, to better demonstrate the impact of n. For the experiments in 4al we randomly choose 100 different hash keys, and then follow the partition of the green list in KGW to calculate the actual green ratio in 100 sampled responses from Alpaca's s queries with Llama-2-7b-chr the detection texts are sampled from Alpaca, to demonstrate the results of detection in mismatched green ratio scenario.
In section 3.2
For the experiments in 4b and 4c
B.3 DETAILS OF CONSTRUCTION OF MODEL SET
The model set in 4.1 is constructed by training the base LLM with data sampled from the source LLM with queries. Specifically, we first sample two source LLM: Llama-2-7b-chat and Llama-3- 8b-Instruct with queries sampled from Evol--nstrrct--oode and GSM8k respectively. For positive samples, we add KGW and Unigram with 10 random hash keys when processing the data sampling. For negative samples, we do not add any watermark to the response. Then we obtain 2 2 x 2 10 80 (source LLM,, dataset, watermarks, and hash keys) watermarked datasets and datasets. Next, we apply the datasets to train suspect models, where the base LLMSs contain Bloom- 7b and Mistral-7b-Instrucct Then the model set of positive samples contains 80 x 2 = 160 (2 base suspect LLM) models. To align the size with it, we individually train 20 models on the clean dataset for these base models and result in x 2 x 20 = 160 negative samples. Thus, the total size of the model set is 320 with 160 positive samples (infringing suspect LLM) and 160 negative samples (clean LLM). Further more, each positive sample is correspond to a negative sample with the same source model, data domain and suspect model, assess the set-wise DSR metric in experiments.
2 4 clean
14

---

Published as a conference paper at ICLR 2025
C DISCUSSION
C.1 GREEN RATTO MISMATCH
C.I.1 GREEN RATIO ACROSS HASH KEYS
To show the influence of different hash keys, demonstrate examples of the actual green ratio of watermark detection calculated by 100 responses sampled from negative sample in the model set. Figure demonstrates the Unigram partition and Figure presents the KGW (n=l) partition. It has shown that the actual green ratio varies across all 10 hash keys used to build the model set. In other words, the problem of mismatched green ratio is very common in the model set. Besides, calculate the green ratio from the anchor data generated with the same queries sampled from -lama-2-7b--le As shown in the figures, the green ratio of anchor data is much more close to the actual green ratio.
0.350
preset actual
0.325
0.300
anchor
0.275
0.250-
0.225
0.202
5
10
Different Hash Keys
Figure 5: Green ratio of presented nto the actual ratio from an un un-watermarked suspect LLM, and the ratio of anchor data with different hash keys for Unigram.
preset
anchor
0.25
0.22
3
10
Different Hosh Keys
Figure 6: Green ratio of presented Yoo the actual ratio from an un- un-watermarked suspect LLM, and ratio anchor data with different hash keys for KGW.
C.2 IMPACT OF GREEN RATIO MISMATCH ON DETECTION
Figure shows the DSR of suspect LLMs under different hash keys for Unigram watermarks. The results indicate that the vanilla detection method heavily influenced by the hash key and struggles to detect watermarks in the suspect LLM across all these hash keys, aligned with the discussion in Section 3.2 In contrast, our method demonstrates more stable detection performance across different hash keys and is significantly better at detecting watermarks in the suspect L.I.M
15

---

Published as a conference paper at ICLR 2025
1.00
ours Vanilla
0.75
0,50
0.25
0.00
Key 1
Key
Key 3
Key
Key
Figure 7: DSR of detection of suspect LLMs when the hash key for the Unigram watermark is different.
anchor LLM
What new technology is currenily being developed impeove the healtheure system?
resporoe
As a holpful resistant, I'm glad you asked! There are several new technologios currenily being developed to ingrove the healtheare system Here some of the most promising ones: Artifcaall Intelligence (AT) Al his alrendy made significnnt inreads varions inductries, and healtheare exception. Al algorithms
help doctors andlyze medical images, diagnose disesses, and develop perscualized plans. For instance, IBM's Watson Onaology an Al-powened platfoem that helps oncologist identify the trestment options cancer patients based on their individual characterssiiss and medical history. 2. Telemedicine: With the rise telemodicine, patients can now concalt doctors ressotely through video conferencing other digital chanelss This can be especially useful for people living in rural nreas helps reduce hospital readmission
patienss conditions trackers heooming incressingly popular people sock to momior their health outside of traditional clinical settings These devices can track vinil signs, such as heaet nile and blood presuure, and provide real-time foodbock to help patients maraage chronic canditions like dishetes hypertension. 4, Blockchain: the potential revolutionize healthence by securing dections and hospitals can securely share pulient informatioon without compromising privacy or security. This can help prevent medical errors and enlunce the overall quality of care. 5D printing 3D printig is being used to create eustomized prostherics, implonts, and even ongans. This technology has the posentaa to transfiom organ transpiattaiion and copair surgery, making possibbl to create hespoke inglants that fir putient's unique instomy.
those with mability issues. Telemedicine closely after they leave hospital. Wearthic devices: Wearable devices smortwatches and
by enabling doctors
Blockchnin tochnology
data assuring intepiiy. Ry using blockchiin.
editing The emerpence gene editing like CRISPR raised possibility treating genetic disorders their sounce. By odhing genes cornect mutations, researchers hope develop treatments disenses sickle call anemia and eystic libuusis. Robotics: Robots are being used to assist with..
loken eatrups
5.432
query
Provide answer to equation.
respumee
Of coumel answer to equation
token entrupy
2.707
Table 6: An example of queries with a high and low token entropy of the anchor LLM.
C.2.1 TOKEN ENTROPY
The token entropy H is influenced by the length of the generated responses as well as the diversity of tokens, as shown in examples in Tan Table If a response contains more number of tokens, the token entropy is higher. Intuitively, these texts are more helpful for detecting watermarks learned by the suspect LLM considering that they more likely to contain the watermarked tokens when the detection and training data are mismatched that the token entropy obtained from the anchor model (Llama-2-7b--d) (Mistral-7bb
domain. Table ( and Table 7 also demonstrate
and the suspect model is rather close (due to the general ability of LLMs), indicating the feasibility of applying token entropy calculate from anchor LLMs to approximate the possible responses from the suspect LLM.
D VISUALIZATION CASES OF DETECTION
Figure shows visualization cases of detecting suspect LLMs in the model set. These figures in- dicate that even though there is gap between 7-scores of positive samples (w/ watermark) and negative samples (w/ watermark), the fixed threshold can not distinguish them properly. The main reason is that the mismatched green ratio makes z-scores of positive samples or negative samples deviate from the expected value. In contrast, the adaptive threshold successfully discriminates them in these cases because takes the mismatched green ratio into account by estimating the actual ratio with anchor data.
16

---

Published as a conference paper at ICLR 2025
Table 7: An example of queries with a high and low token entropy of the suspect L.L.M.
suspect LLM
What new technology is currenily being developed imgeuve the healtheure system?
resporoe
Contiilly! There are several energing technologies that have the polemial to significantly impuct the healtheare industry- 1. Telemodicine and Vinual Care. With the rise of remote work and social distancing messures, telemedicine has hecomee incressingly popular as a way provide medical services remotel. Vistual crre platfomss allow putients to consult with
reorive disgnoses, trealments, and follow-up care via video confermee phone calls, This can save time and resourees while essuring that patients receive high-qualit care. 2. Al and Muchine Leuming: Artifccial intellignnce (AT) and machine leaming algorihmms are being develoyed to help disgnose discases, develop penomilized treaiment plans, and even predict puliens outcomes. These tools analyze lange amounts of data from electonic health identify putterns and
reconls, laborstcry tests and other
doctoon and potential complications early on and appropritt preventative messures. Wearable Devices Wesrable devices such as snartwaches. finess tackers, and sensors are bacoming more suphistizated and integrated into healthcare systems. These devices can monikor vital signs, tack plysical setivity levels, and even detect early warning signs of chreeic conditionn like dishetes heart disease. Ry provictigg real- time data, devices enable healthcare providers to intervene cutlier and prevent complication. technology has revolutionieed the field of genetios, allowing researchers to make precise changes to DNA sequences. This technology has enamous potostial in healtherre, porticuluely in the areas of genttic disorders and cancer therapy. It could pocemially zure genttic diseaves, inberited cundition, and revers aging. 5. Bluckchain Technology: Diockchii technology is being explored for its putental applications healihccure
predictions They
ideetify
Gene Editing: CRISPR gene editing
platform for sharing sensitive medical information, roducing fraud, improving incroperability berween different hesttheaee systems could lead to better cocrdination hetworn dection, hospitals and insurers, resulti ng in mure effisiens and effective care delivery. 6. Nanoocchnology: Nanotechnolugy refers to the munipuhation of materials the nunoscale level (less thin 100 sunometers). Iis applications in healthetre include drip delivery sydems (hat target specific oells tisones bicwensors desect discases monikor medication adherence, even prosthetic limbs that can mimic natural mov novemmnss. These innovations revolutionie how we approach various health challenges, including cancer, disbetes...
secure,
entropy
5.434
query
Provide answer equation 7+6
respumse
Of coune!! The sum of and 6 is: 13
token entrupy
2.686
Tarc Srashll
500 110M s0le H0W 1158 Nlm
200 188 108: 1900 1150 DIN 1991 1987
100 5m0 100 1960
1908 1960 1500 1908
Taken (a)
(b)
(c)
Mapio Treduu
Generated Tiken
GGreerated Bisen
Generated Taken
(d)
(I)
Majio Trnduc
17d0 GGreeraled aldd Iocon
Geverated Taken
Getic ubod Takeo
g
(h)
Figure& 8: Cases of detection with the adaptive threshold.
17


Reviews:
[{"review_id": "JoQG7YF1lR", "reviewer": "Reviewer_RZ9s", "summary": "The authors propose LIDet, a novel method designed to detect whether an LLM has been fine-tuned using the outputs of another LLM, potentially causing IP infringement. The approach assumes that the source LLM’s generated outputs are watermarked and that a suspect model is fine-tuned on them.  Based on this and assuming that the suspect model has learned this watermarked distribution of data, the method uses a set of anchor LLMs to select appropriate queries for sampling from the suspect model. It then detects watermarks in the suspect's generated text using an adaptive threshold based on z-scores. The authors evaluate the method with different models and datasets, proposing an analysis and ablation study of LIDet based on different watermarking methods.", "questions": "•\tDo you think LIDet performs better than [1]? Could you provide a comparison of the two methods?\n\n•\tShould the set of anchor models always contain the same source model architecture?\n\n•\tDo you think the suspect model's dimension (number of parameters) could affect the performance of the method?\n\n•\tDo you think there is a correlation between the detectability of the watermarks and the domain of the datasets considered in the queries used to train the suspect model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tThe authors propose an evaluation of the method with data from different domains.\n\n•\tThe authors evaluate the performance of the detection method with different watermarking techniques.\n\n•\tThe authors propose a clear and precise problem statement and threat model.", "weaknesses": "•\t The paper lacks in the comparison of LIDet with other methods in the state of the art.\n\n•\tThe Results section (4.2) does not provide sufficient details on the performance of the different anchor models considered in LIDet.\n\n•\tThe authors do not propose an evaluation of the method using different combinations of original and suspect models. \n\n\n**Comments.**\n\n1.\tIn the related works, the authors cite the work in [1] which proposes another method to detect LLM watermarks from the suspect model tuned with watermarked texts. The authors did not propose a comparison of LIDet with this method, even though it attempts to solve the same type of problem. Although the threat model is slightly different, the author could have proposed a comparison with [1] in Table 3 when the Query for Detection follows the same distribution as the Query for Training, i.e., i.i.d. detection queries as considered in [1].\n\n2.\tIn the first step of the detection method, the authors propose to sample a set anchor LLMs $\\theta_{\\rm anchor}^M$ used to help select the proper queries to sample the suspect LLM. As presented in the paper, the set of anchor models presents the same model architectures as the source model, however, there is no discussion on how to select the right set of anchor models, nor is there any insight into which anchor model contributes the most to the selection of the final set of queries. The paper does not present any ablation on the number of anchor models or on the type of model architectures.\n\n3.\tIn addition to that, the authors considered only the scenario where Llama2 and Llama3 models are used as source and anchor models, but not as suspect models. There is no analysis of LIDet's performance when the suspect and source models are derived from the same base model. Similarly, the authors did not provide any analysis of the method using different model architectures as source models, e.g. Mistral.\n\n\n\n**References**\n\n[1] Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze and Teddy Furon, Watermarking Makes Language Models Radioactive, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Toxbz3fCyI", "reviewer": "Reviewer_gu9D", "summary": "This paper studies the problem of Model IP infringement for LLMs. The authors propose to use different anchor LLMs to select query samples that are more likely to contain more diverse and evenly distributed tokens, which are useful for later detection. Furthermore, for the detection phase, it uses dynamic thresholding for the z-score-based detection methods. The empirical results show the significant detection performance compared to the baselines.", "questions": "As the paper is mostly an empirical paper, I would suggest making the evaluations more comprehensive. The detailed suggestions for comprehensive evaluations are listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The structure of the paper is good that, it first empirically identifies the vulnerabilities of existing watermarking scheme for text or models.\n2. The empirical performance is strong.", "weaknesses": "1. The evaluation settings rather limited and only contains two domains and also did not consider the fact that, if the target domain is a mixture of multiple domains (e.g., math and code). The paper can benefit from a more comprehensive evaluations that mixes multiple domains. \n2. The threat model is still unclear to me. Even though the different types of downstream applications exist, the the source model trainer should still have some general understanding about the high-level domains that are involved in the training set. Therefore, even for the vanilla approach, generating some possible responses from each of these high-level domains is a possible (and potentially stronger) baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qi4N1fZu1L", "reviewer": "Reviewer_aqr8", "summary": "To distinguish the AI-generated texts from human-produced ones, watermarking is added over generated texts, by using green-and-red list and adding bias to the logits of different tokens. In this paper, the authors exploit this text watermark to detect whether a new model $M_a$ is finetuned over the texts generated from a well-trained model $M_b$. Ideally, if $M_a$ is totally trained over the texts generated from $M_b$, the outputs of $M_a$ satisfy the same bias as $M_b$, so that the detector can detect the output of $M_a$ to determine whether the IP of $M_b$ is violated or not. However, in the realistic situation, there exist the domain mismatch and green ratio mismatch between $M_a$ and $M_b$, which leads to the failure of the detection. The authors proposed a new method called LIDet to detect whether $M_a$ is finetuned over the $M_b$-generated data.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tClear definition of threat model to show the capability of stealer and detector.\n2.\tScrutiny the reasons on the attenuation of watermark detection: domain mismatch and green ratio mismatch.", "weaknesses": "Lack of adaptive attack: The authors only assume that the stealer will directly finetune $M_a$ over the queried data, and fail to exploit the adaptive attack. For instance, if the stealer knows the queried data including the watermark, he/she could modify the queried texts to remove the watermark. By this way, the finetuned model $M_a$ may not learn the bias. From another perspective, the stealer can even finetune the $M_a$ without the data from $M_b$, but mimic the same bias as $M_b$ to increase the FPR of LIDet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MD8wBpbBPT", "reviewer": "Reviewer_azDs", "summary": "This paper investigates using watermarks in large language models (LLMs) to detect intellectual property (IP) infringement. It identifies two key issues with direct watermark detection: the impact of sampling queries on detectability and the instability of watermark hash keys. To address these, the authors propose LIDet, a method that selects suitable sampling queries and adapts detection thresholds. Experimental results show LIDet achieves over 90% accuracy in distinguishing between infringing and clean models, demonstrating its effectiveness for detecting LLM IP infringement.", "questions": "(1) Please evaluate the number of queries required for effective detection and present the results obtained using varying numbers of queries.\n\n(2) It is recommended to conduct experiments using models other than Mistral-Instruct-7b as the base model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with clear logic and organization that makes it easy to follow.\n2. The experimental results appear promising, demonstrating the proposed method's effectiveness in distinguishing between infringing and clean models.", "weaknesses": "1. Evaluation of Time/Computational Costs: The experimental section lacks an assessment of the time and computational costs involved. It should evaluate the number of queries required for effective detection, as this is crucial for understanding the practicality of the proposed method.\n2. Transferability of the Approach: The paper does not adequately address how the proposed method, LIDet, can be applied to other models. Clarifying the conditions under which the method retains its effectiveness across different architectures would strengthen the contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBJMhibYPG", "reviewer": "Reviewer_z7oZ", "summary": "This paper proposes using watermarking to detect IP infringement in large language models (LLMs). The proposed method, LIDet, improves detection accuracy by addressing domain mismatch and watermark stability through adaptive thresholding and optimized query selection based on token entropy and generalizability. Experiments show that LIDet achieves high reliability in distinguishing between infringing and non-infringing models, enhancing LLM protection.", "questions": "1. If two models are trained on overlapping open-source datasets, will the false positive rate significantly increase? Shared data could lead to misclassification of non-infringing models as infringing ones.\n2. How does watermarking affect the model’s performance—does it enhance or degrade output quality? It’s essential to evaluate whether watermarks could unintentionally reduce the overall effectiveness of the model's generation capabilities.\n3. How can candidate queries effectively ensure that the infringing model has learned specific watermark phrases? If the infringing model has not learned the intended watermarked phrases, can these queries reliably retrieve them?\n4. How can we ensure that the responses retrieved through queries are indeed related to the watermark? Some models may unintentionally produce watermarked phrases simply through regular training, which could lead to inadvertent misclassifications.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach in this paper leverages watermarking as a novel method to detect IP infringement in large language models (LLMs), providing a robust layer of protection against unauthorized replication. The proposed detection method, LIDet, tackles key challenges such as domain mismatch and stability issues through adaptive thresholding and a targeted query selection strategy. By focusing on token entropy and cross-model generalizability, the proposed query strategy ensures more reliable detection, reducing false positives and achieving high accuracy across various settings.", "weaknesses": "The paper could benefit from further exploration of the impact of watermarking on overall model performance, including metrics such as perplexity, BLEU scores, or human evaluations to assess any trade-offs in output quality and diversity. Additionally, while candidate queries are selected for high token entropy and cross-model generalizability to ensure that infringing models retain watermark patterns, the detection reliability may vary if infringing models do not fully absorb specific watermarked phrases. Testing detection on models trained with varying amounts of watermarked data could clarify this absorption. Lastly, the “anchor LLM” method aims to reduce false positives by selecting queries with favorable green ratios and entropy, but there is still a risk that unwatermarked models could unintentionally exceed detection thresholds. Analyzing the false positive rate across a broader set of non-infringing models under various conditions would provide a more comprehensive understanding of the method’s specificity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)