<figure><img src="image_1.png" /></figure>

HAL
open science

# Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

Yanming Li, Seifeddine Ghozzi, Cédric Eichler, Nicolas Anciaux, Alexandra

Bensamoun, Lorena Gonzalez Manzano

► To cite this version:

Yanming Li, Seifeddine Ghozzi, Cédric Eichler, Nicolas Anciaux, Alexandra Bensamoun, et al. Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique. 2025. [hal-05293957](https://hal HAL Id: hal-05293957)

Preprint submitted on 6 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Technique

Yanming Li

Inria, INSA CVL, Univ. Paris Saclay
Palaiseau, France
yanming.li@inria.fr

Seifeddine Ghozzi

ENSTA Paris
Palaiseau, France
seifeddine.ghozzi@ensta.fr

Cédric Eichler

INSA CVL, UO, Inria
Bourges, France
cedric.eichler@inria.fr

Nicolas Anciaux

Inria, INSA CVL, Univ. Paris Saclay
Palaiseau, France
nicolas.anciaux@inria.fr

Alexandra Bensamoun

Univ. Paris Saclay
Saclay, France
alexandra.bensamoun@universite-paris-saclay.fr

Lorena Gonzalez Manzano

Universidad Carlos III de Madrid
Madrid, Spain
lgmanzan@inf.uc3m.es

**Abstract**—We address the problem of auditing whether sensitive or copyrighted texts were used to fine-tune large language models (LLMs) under black-box access. Prior signals—verbatim regurgitation and membership inference—are unreliable at the level of individual documents or require altering the visible text. We introduce a text-preserving watermarking framework that embeds sequences of invisible Unicode characters into documents. Each watermark is split into a cue (embedded in odd chunks) and a reply (embedded in even chunks). At audit time, we submit prompts that contain only the cue; the presence of the corresponding reply in the model’s output provides evidence of memorization consistent with training on the marked text. To obtain sound decisions, we compare the score of the published watermark against a held-out set of counterfactual watermarks and apply a ranking test with a provable false-positive-rate bound. The design is (i) minimally invasive (no visible text changes), (ii) scalable to many users and documents via a large watermark space and multi-watermark attribution, and (iii) robust to common passive transformations. We evaluate on open-weight LLMs and multiple text domains, analyzing regurgitation dynamics, sensitivity to training set size, and interference under multiple concurrent watermarks. Our results demonstrate reliable post-hoc provenance signals with bounded FPR under black-box access. We experimentally observe a failure rate of less than 0.1% when detecting a *reply* after fine-tuning with 50 marked documents. Conversely, no spurious *reply* was recovered in over 18,000 challenges, corresponding to a 100% TPR@0% FPR. Moreover, detection rates remain relatively stable as the dataset size increases, maintaining a per-document detection rate above 45% even when the marked collection accounts for less than 0.33% of the fine-tuning data.

**Index Terms**—Watermarking, LLM, invisible unicode

## I. INTRODUCTION

Large-scale AI models are trained on vast, diverse and reliable datasets, but the exact composition of these datasets is generally undisclosed. This lack of transparency makes it difficult to determine whether personal data (e.g., CVs on

professional platforms, personal blog posts or forum discussions) or works protected by copyright (e.g., news articles, books, poetry) are included in training. The opacity of training data has implications for privacy, intellectual property, and data governance. It has also led to increased attention from regulators and civil society, as well as to the emergence of initiatives aimed at documenting or monitoring potential data usage, such as AI Watchdog<fnref n="1" />. This opacity not only raises technical and societal concerns, but also creates major legal uncertainty.

**Legal context.** The need for technological support is more evident than the law’s inability to ensure legal certainty. While there is indeed an exception at European level for text and data mining that suspends the application of copyright and related rights, which could perhaps benefit model training, this is subject to the dual condition that the AI provider has lawful access to the content and that the rights holder has not exercised his opt-out right. However, the opacity surrounding these activities makes it impossible to know whether the conditions are being met. Therefore, the AI Act<fnref n="2" /> imposed a transparency obligation on AI providers, which consists of developing and making available to the public a “sufficiently detailed summary” of the content used, for which a model was provided by the AI Office<fnref n="3" /> However, it is not certain that the required level will enable rights holders to enforce their rights. The situation is no more satisfactory in the United States, where some 50 lawsuits are pending on this issue and where the fair use exception is very uncertain. As proof, Anthropic agreed to settle a lawsuit by offering to pay $1.5 billion in compensation for the unauthorized use of 500,000 books<fnref n="4" />. These uncertainties illustrate that legal instruments alone are unlikely to provide sufficient protection or enforceability, reinforcing the need for complementary technical solutions.

This research was supported by grants PID2023-150310OB-100 (MORE4AIO) funded by MCIU/AEI / 10.13039/501100011033 / FEDER, UE, DATAIA Convergence Institute as part of the “Programme d’Investissement d’Avenir” (ANR-17-CONV-0003) and IPoP PEPR (ANR-22-PECY-0002”) operated by Inria.

<footnote n="1">¹See the author search engine provided by The Atlantic.</footnote>

<footnote n="2">²See the AI Act enacted in June 2024 (link).</footnote>

<footnote n="3">³See the AI Office enacted in July 2025 (link).</footnote>

<footnote n="4">⁴See for example the news article by Reuters (link).</footnote>
**Objective.** An important objective for trustworthy AI is to enable citizens, authors, and rights-holding organizations to verify whether their data have been used in the training or fine-tuning of large generative AI systems. This goal is relevant for general-purpose LLMs (e.g., GPT or LeChat) and is especially critical for domain-specific chatbots obtained through fine-tuning, such as customer-service agents deployed by banks or commercial platforms. In this work, we address the need for auditing techniques by designing and evaluating a minimally invasive marking framework that supports reliable post-hoc detection of fine-tuning data usage.

**Limits of existing solutions.** Existing technical approaches do not provide reliable non-invasive detection of unauthorized use of data in LLM training. First, prior work has shown that training data can sometimes reappear verbatim in model outputs (regurgitation) [1], but such occurrences are rare, cannot be targeted toward specific documents, and are not legally robust<fnref n="5" />. Second, membership inference attacks (MIAs), initially developed to determine whether specific data points were part of a model’s training set [2] and more recently adapted to LLMs [3], remain limited in practice [4], [5]. Even with access to model weights [6], MIAs perform poorly on individual documents due to bias and weak statistical signals, and they scale only to large collections [7] under restrictive “laboratory conditions” (e.g., when training/test splits are revealed, as in Pythia). Recent analyses further suggest the theoretical impossibility of obtaining statistically reliable evidence for copyright claims through MIAs in post-hoc settings [8]. Recently, several copyright protection techniques for textual documents have been proposed for LLM training data [9], but these techniques are generally very invasive. For example, they often require inserting long, unnatural “traps” into the original text [10], which makes them difficult to apply to sensitive content such as journalistic articles, books, or poetry.

**Our approach.** We introduce a text-preserving watermarking framework that leverages invisible Unicode sequences as canaries for fine-tuning data provenance auditing. Each watermark is split into a *cue* (embedded in odd chunks of a document) and a *reply* (embedded in even chunks). At audit time, we issue a black-box challenge that presents the *cue*; successful regurgitation of the *reply* in the model’s output indicates memorization consistent with training on the marked document. To provide sound evidence with a bounded false-positive rate (FPR), we adopt a ranking test against a reserved set of random counterfactual canaries. This yields a post-hoc membership decision with provable FPR control. Our design emphasizes (i) minimal invasiveness (no visible text change), (ii) scalability to many users via a large watermark space and multi-watermarks attribution, and (iii) robustness to common passive transformations (copy–paste and basic format conversions).

**Contributions and result summary.**

* We present a minimally invasive, text-preserving watermarking framework for provenance auditing after LLM fine-tuning, based on invisible Unicode canaries with *cue*–*reply* structure and black-box detection.

* We provide (i) a ranking-based verification procedure using reserved random watermarks, yielding membership decisions with a *provable* bound on the false-positive rate, and (ii) a large watermark space with uniqueness constraints, and an attribution protocol that supports many users and multiple concurrent watermarks without degrading detection.

* We evaluate across multiple open LLMs and text domains (blogs, poetry), studying (i) memorization/regurgitation vs. repetitions, (ii) sensitivity to training set size, and (iii) multi-watermark interference. We release code, watermark generation and detection utilities, and evaluation scripts to facilitate reproducibility.

* We experimentally approximate a failure rate of less than 0.1% when detecting a *reply* after fine-tuning with 50 marked documents. Conversely, in over 18 000 challenges, presenting a *cue* absent from the fine-tuning dataset yielded no spurious *reply* recoveries, corresponding to a 100% TPR @ 0% FPR. Moreover, detection rates remain relatively stable as the dataset size increases, maintaining a per-document detection rate above 45% even when the marked collection accounts for less than 0.33% of the fine-tuning data. Similarly, the average success rates are generally only mildly affected by an increase in the number of unique watermarks. In only one of our four settings did the detection rate decrease significantly, from 95% to 72%, when the number of unique watermarks was tripled.

**Outline.** The rest of the paper is organized as follows. Section II formalizes the data provenance auditing problem and the main requirements. Section III presents our proposal, including watermark properties, generation, embedding, detection, and the FPR-bounded test. Section IV reports the experimental evaluation. We position our work within prior related works in Section VI. Section V discusses limitations and avenues for strengthening robustness. We conclude in Section VII.

## II. PROBLEM STATEMENT

The dataset copyright auditing problem can be formally defined as follows. The notations used in the paper are summarized in table I.

**Data provenance auditing (DPA) problem.** Let $\mathcal{D}$ denote a protected dataset (e.g., a collection of news articles, poems, personal documents), and let $\mathcal{L}$ denote a suspicious model potentially trained or fine-tuned on a dataset including $\mathcal{D}$. A data provenance auditing procedure $\mathcal{P}$ takes as input both $\mathcal{D}$ and $\mathcal{L}$ and outputs a binary decision: $\mathcal{P}(\mathcal{D}, \mathcal{L}) \rightarrow \{0, 1\}$. If $\mathcal{P}$ returns 1 the procedure claims that $\mathcal{D}$ (or part of it) was used in the training or fine-tuning of $\mathcal{L}$; otherwise, it returns 0.

<footnote n="5">⁵See Nicholas Carlini’s 2025 blog post, which explicitly advises against using these results for copyright litigation (link).</footnote>
<table><caption>TABLE I: Notations</caption><tr><td>D = {D}</td><td>A sensitive collection of documents</td></tr><tr><td>D = (d<sub>i</sub>)<sub>i=1..Δ</sub></td><td>A document of Δ words</td></tr><tr><td>L</td><td>A suspicious model</td></tr><tr><td>P</td><td>A data provenance auditing procedure</td></tr><tr><td>A</td><td>The alphabet of invisible characters</td></tr><tr><td>s ∈ A<sup>m</sup></td><td>A syllable of m invisible character</td></tr><tr><td>W</td><td>The domain of watermarks</td></tr><tr><td>W<sub>K</sub> ⊂ W</td><td>Set of K candidate watermarks provided upon request</td></tr><tr><td>w = (s<sub>i</sub>)<sub>i=1..n</sub></td><td>A watermark of n syllables</td></tr><tr><td>wc = (s<sub>i</sub>)<sub>s<sub>i</sub>∈w, i≤j</sub></td><td>The “cue” of w made of its first j syllables</td></tr><tr><td>wr = (s<sub>i</sub>)<sub>s<sub>i</sub>∈w, i>j</sub></td><td>the “reply” of w made of its last n-j syllables</td></tr><tr><td>step</td><td>The “step” in number of words used in the marking scheme</td></tr><tr><td>D = (D<sub>i</sub>)<sub>i=1..[Δ/δ]</sub></td><td>D is a sequence of contiguous chunks of δ words</td></tr><tr><td>{D<sup>wc</sup><sub>2i+1</sub>}</td><td>Odd chunks of D<sup>w</sup> watermarked with syllables of cue</td></tr><tr><td>{D<sup>wr</sup><sub>2i</sub>}</td><td>Even chunks of D<sup>w</sup> watermarked with syllables of reply and the cr last syllables of cue</td></tr><tr><td>D<sup>w</sup> = mark(D, w)</td><td>D chunks marked by scheme mark using w</td></tr><tr><td>verif()</td><td>A verification procedure</td></tr><tr><td>ID<sup>w</sup><sub>i</sub></td><td>Challenge constructed from D<sup>w</sup><sub>2i-1</sub></td></tr><tr><td>λ</td><td>Number of repetitions of each ID<sup>w</sup><sub>i</sub> during verif</td></tr></table>

We consider the case of protection of sensitive documents (because of copyright, privacy...) where initial sensitive documents can be marked beforehand, to then be detected. In addition, the auditing procedure must operate under black-box conditions, as proprietary LLMs often restrict access to their internal states and token probabilities, particularly in fine-tuned chatbots, which typically expose only their output text. This leads to a DPA in two steps: (1) a marking scheme called *mark* which embeds marks in documents, and (2) a verification procedure called *verif*, which prompts the suspicious model and makes a decision from the resulting responses. This leads to two layers of requirements.

**Data marking requirements.** A marking scheme *mark* embeds imperceptible marks into documents $D \in D$ before possible training or fine-tuning. It must satisfy the following requirements:

* **Text authenticity.** The marking scheme ensures that a marked document *mark(D)* must leave the visible form of *D* unchanged, ensuring that the author's original words remain fully intact (e.g., in poems or literary works the exact form of the text is crucial).

* **Scalability.** The marking scheme must support the assignment of many distinct watermarks across users and documents, enabling multi-watermark attribution without collisions or interference.

* **Robustness.** The marking scheme must ensure that marked texts remain effective despite common transformations, such as copy-paste or format conversions between (e.g., PDF and plain text) and, to the extent possible, active manipulations (e.g., text normalization or automated cleaning).

**Verification procedure requirements.** Given a suspicious model *L* and a set of marked sensitive documents *mark(D)*, the verification procedure *verif* must satisfy:

* **Soundness.** The auditing procedure must guarantee that if it outputs *verif/mark(D, L)* = 1, then this decision is correct except with a provably small probability. In particular, the probability of erroneously flagging data that was not used (*false positive*) must be strictly bounded. Soundness ensures that positive findings can be relied upon as credible proof, e.g., in legal or forensic contexts.

* **Completeness.** The auditing procedure must guarantee that if a dataset (or document) was indeed used in training, then it is detected, i.e., *verif/mark(D, L)* = 1 except with a small probability. In other words, the probability of failing to identify genuinely used data (*false negative*) must remain small. In the post-hoc, black-box setting considered here we do not claim that a general-purpose theoretical proof bounding this “small” probability is attainable. Instead, completeness should be demonstrated through rigorous empirical evaluation across representative models, datasets and reporting observed false-negative rates, ensuring that actual training data cannot systematically evade detection.

Relying on invisible Unicode sequences rather than visible edits (e.g., synonym or explicit “copyright traps” [10]) yields a methodological advantage. Because invisible marks do not change the visible bytes of a document, they remove the semantic, stylistic and editorial constraints that govern visible modifications. This freedom permits the insertion of short non-printing “syllables” at arbitrary intra- or inter-word positions and the construction of structured *cue-reply* patterns that are independent of surface language. Practically, these properties<fnref n="6" /> (i) increase the usable watermark channel capacity and lower collision risk across many users, (ii) enable focused memorization signals during fine-tuning without altering the published text, and (iii) simplify black-box detection by allowing searches for non-linguistic sequences instead of brittle semantic matches.

While it changes the design space, a central difficulty in this setting lies in controlling false positives while still detecting genuine use. A naïve regurgitation-based test may mistakenly flag a document as training data simply because the model outputs a sequence that coincidentally matches a suspected watermark, leading to an erroneous claim of infringement. To address this, we employ a ranking-based testing strategy inspired from [11]: the suspicious watermark is evaluated against a held-out set of random counterfactual watermarks. This approach makes it possible to bound the false positive rate (FPR) by design. For instance, if *k* top-ranked watermarks are accepted among *K* candidates, the FPR is provably bounded by *k/K* [8]. These formal guarantees are essential to ensure that positive detections can be trusted as credible evidence.

<footnote n="6">⁶The advantages of using invisible marks are also counterbalanced by vulnerabilities to adversarial manipulation. Addressing these vulnerabilities is left for future work (see Section V).</footnote>
Obviously, properties must also be evaluated empirically to assess them and demonstrate their practical interest.

### III. PROVENANCE AUDITING THROUGH NON-VISIBLE WATERMARKS

Our design is driven by the requirements of Section II: (i) Text authenticity: marks must not modify visible text; (ii) Scalability: support a very large watermark space and multi-user attribution; (iii) Robustness: survive copy-paste and common format conversions; (iv) Black-box operability: verification uses only text/prompt I/O; (v) Soundness: membership claims have a provable false-positive bound; and (vi) Completeness: when the data is used, it is detected with high probability. The subsections below make these links explicit.

The overall proposal is sketched in Figure 1. We first introduce our marking scheme which includes watermark generation, attribution, and embedding. Then we present the verification procedure to make the membership decision. We end the section with a discussion and limitations.

#### A. Watermark generation

Watermarks serving as canaries are sequences built from an alphabet $\mathcal{A}$ composed exclusively of invisible characters, ensuring text-preservation. Because isolated invisible characters may resemble noise, we group them into syllables to improve memorization. A syllable is an ordered list of $m$ characters of $\mathcal{A}$, denoted $s \in \mathcal{A}^m$. A watermark $w$ is hence an ordered list of $n$ syllable, $w = (s_i)_{i=1..n}$, $s \in \mathcal{A}^m$.

**Text authenticity.** Because the alphabet $\mathcal{A}$ contains only non-rendering code points (non-visible Unicode), insertion leaves the sequence of visible code points unchanged; thus the author's visible text is byte-for-byte preserved.

Each watermark consists of a cue and a reply, denoted $wc$ and $wr$ respectively, with:

$$wc = (s_i)_{s_i \in w, i \le j} \quad \text{and} \quad wr = (s_i)_{s_i \in w, i > j}$$

The cue is embedded in a prompt and serves as the stimulus for the appearance of the reply. When prompting a model, the presence of the corresponding reply in its output is an indication that it was trained on a document containing the watermark.

We note $W$ the domain of watermarks. To minimize false positives, we aim to reduce the likelihood that a reply associated with a dataset is observed by a model not trained on it. To this end, we impose the following two constraints on $W$:

1) Each cue corresponds to a unique reply and vice versa: $\forall (w_1, w_2) \in W^2$, we have:

$$(wc_1 = wc_2 \lor wr_1 = wr_2) \Leftrightarrow w_1 = w_2$$

2) No cue is contained in a reply and vice versa:

$$\forall (w_1, w_2) \in W^2, \neg (wc_1 \sqsubseteq (wr_2)) \land \neg (wr_1 \sqsubseteq (wc_2))$$

where $x \sqsubseteq y$ denote that the ordered list $x$ appears as a contiguous subsequence of the ordered list $y$.

Because of condition (1), the number of distinct watermarks equals the minimum of the number of distinct cue and reply. Condition (2) is generally related to cross-bifix-free codes, which prevent a direct computation of the total number of admissible watermarks $|W|$. In the special case where $j = n-j$, i.e. when cues and replies have the same length, (2) reduces to an inequality. Since each cue or reply is a sequence of $j$ syllables, each syllable consisting of $m$ letters from $\mathcal{A}$, the space of all possible sequences can be divided evenly into the sets of admissible cue and reply. Consequently, when $j = n-j$, the total number of admissible watermarks is:

$$|W| = \frac{|\mathcal{A}|^{m-j}}{2}$$

**Scalability.** These two constraints imply disjoint (cue, reply) pairs and exclude overlaps between cue and reply in different pairs, yielding a space $|W|$ that scales exponentially. In practice, with moderate $m$ and $j$, this supports millions to billions of watermarks usable for publication, enabling large-scale multi-user issuance without collisions.

#### B. Watermark Attribution

We assume the presence of a trusted entity responsible for generating and assigning watermarks while enforcing the constraints defined in Sec. III-A. This entity guarantees the uniqueness of each watermark by maintaining a record of all previously communicated watermarks, denoted $W_{com}$.

Whenever a user requests a new watermark, the trusted entity samples a set $W_K$ uniformly at random from $W \setminus (W_{com})$, updates its record $W_{com} \leftarrow W_{com} \cup W_K$, and returns $W_K$ to the user. In particular,

$W_K$ serves as a candidate set of watermarks for the user: they select one uniformly at random from $W_K$ for actual use and retain the remaining watermarks as counterfactuals for proof purposes (see below).

#### C. Watermark Embedding

A document $D$ is split by spaces and treated as an ordered list of words, $(d_i)_{i=1..\Delta}$ (e.g., "Hello world." is considered two words: "Hello" and "world.").

Because the cue is used to construct prompts in which no reply should appear, the two are embedded in separate chunks of the document. A first approach would be to simply split the document in two. However, this creates problems, especially for large documents. First, the portion containing the reply may be too large to fit within the audited model's context window while still leaving space for a response. Second, that would lead to large excerpts of the documents (e.g., entire chapters) on which a model might have been trained on that contain either the reply or the cue but not both.

Hence, to ensure that significant samples of the document necessarily contains both the cue and the reply, and to account for context window, we chunk $D$ into $\lceil \Delta/\delta \rceil$ sub-documents $(D_i)_{i=1..\lceil \Delta/\delta \rceil}$ containing $\delta$ words (except for the last).

Alternatively, a chunk is embedded with either (most of) the cue, $(s_i)_{s_i \in w, i \le j}$, or $(s_i)_{s_i \in w, i \ge j}$ (i.e. the cr last syllables of
<figure><img src="image_3.png" /><figcaption>Fig. 1: **Overview of the proposal.** The main logical steps are indicated by black circles numbered 1 to 6. ①: A new sensitive dataset is produced. ②: A new batch of *K* (e.g., 100) watermarks, each consisting of invisible Unicode characters, is selected. One of them is randomly chosen (here *w*₁) for publication, while the other *K*−1 are reserved for the final ranking to ensure statistically grounded decisions. ③: The dataset is watermarked *K* times, once with each watermark. ④: The publishable dataset is the one marked with *w*₁. ⑤: A suspicious chatbot is based on a model fine-tuned on a dataset potentially containing sensitive documents (or parts of them). ⑥: This model is probed via black-box access: each odd (green) chunk of a marked document (with watermarks *w*₁ to *w*ₖ) is submitted, and the outputs are analyzed to detect the presence of the corresponding watermark’s “reply” sequence of syllables (represented by the blue dots and present in even chunks only). Finally, for each watermark and each document marked with it, the number of expected occurrences with a “reply” component of the watermark in the output is counted. The watermarks are then ranked by frequency. If the publication watermark ranks high enough (above a fixed threshold *k*), this indicates that sensitive data was included in the model’s fine-tuning set.</figcaption></figure>

the *cue* and the *reply*. Algorithm 1 describes the partitioning and the alternative embedding.

Sub-documents with odd indexes, or *cue* chunks, are embedded with the first *j* − *cr* syllables of the *cue* and denoted {*D*<sub>2*i*+1</sub><sup>wc</sup>}.
Sub-documents with even indexes, embedded with (*s*<sub>i</sub>)*s*<sub>i</sub>∈*w*,*i*>(*j*−*cr*) (i.e., the last *cr* syllables of the *cue* and the *reply*), are called *reply* chunks and denoted {*D*<sub>2*i*</sub><sup>wr</sup>}.
This ensures that *cue* chunks can be used in a prompt without containing the *reply*.

The function *chunk_mark* embeds the relevant syllables within a subset of the document. The first syllable is inserted after the first word, and the remaining ones are inserted every *step* word. The *j* − *cr* or *n* − *j* + *cr* syllables are injected cyclically throughout the list. When the end of this list is reached, a syllable is not necessarily inserted even if the last one was injected exactly *step* words before. Rather, if the final repetition is incomplete, the remaining syllables are concatenated at the end to ensure full representation. The resulting watermarked chunk consists of an ordered sequence of syllables (each containing *m* invisible characters) interleaved with the original words, which remain unchanged and in their original order.

Appendix A provides an analytical expression for the num-
ber of watermark repetitions in a document marked with this
strategy.

D. Verification: Watermark Detection

We denote by $\mathcal{L}(I)$ the output generated by a LLM $\mathcal{L}$ when prompted with input $I$. This output is treated as an ordered list of words and invisible characters. Unlike the embedding stage, these extracted invisible characters are not assumed to be in a syllable of size $m$.

1) *Single document:* For a watermarked document $D^w$, the verification consists of constructing $\lfloor \frac{[\Delta/\delta]}{2} \rfloor$ inputs $(ID_i^w)_{i=1..\lfloor \frac{[\Delta/\delta]}{2} \rfloor}$ made of:

* $D_{2(i-1)+1}^{wc}$, containing the repeated $j-cr$ syllables of the *cue wc*, followed by

* the first cr·(1+step) elements of $D_{2i}^{wr}$ (i.e. the first step·cr words of $D_{2i}^{wr}$ embedded with the last cr syllables of the *cue wc*). This completes the partial *cue* of $D_{2(i-1)+1}^{wc}$ and serves as a stimulus for the appearance of the *reply wr*.

We then query the model $\lambda$ times with each $ID_i^w$. The chunk-level verification, described in Algorithm 2 outputs one if and only if the *reply wr* is detected in $\mathcal{L}(ID_i^w)$ in at least one of the $\lambda$ repetitions. Detection is conducted by filtering the model's output, removing all visible characters and concatenating only the invisible ones in their original order into $o_{invis}$. If any subsequence of $o_{invis}$ matches *wr*, the *reply* has been detected.

Given a document $D^w$, the overall verification score is
obtained by summing the results of each of the $\lfloor \frac{[\Delta/\delta]}{2} \rfloor$ chunk
**Algorithm 1** *mark*, embed Watermark *w* in *D*

Input: $D = (d_i)_{i=1..\lceil\Delta/\delta\rceil}$, $\delta$, cr, step, j, w

Output: $D^w$, *D* watermarked with *w*

1: $D^w \leftarrow \text{empty list}$

2: $x \leftarrow 1 \{\text{Start of chunk}\}$

3: **while** $x + \delta < \Delta$ **do**

4: &nbsp;&nbsp;&nbsp;&nbsp;`y &lt;&lt; x + δ - 1`

5: $D^{wc} \leftarrow \text{chunk\_mark}((d_i)_{i=x..y}, (s_i)_{s_i \in w, i \le j-cr})$

6: Concatenate $D^w$ and $D^{wc}$

7: $x \leftarrow x + \lambda$

8: $y \leftarrow \min(x + \delta - 1, \Delta)$

9: $D^{wr} \leftarrow \text{chunk\_mark}((d_i)_{i=x..y}, (s_i)_{s_i \in w, i > j-cr})$

10: Concatenate $D^w$ and $D^{wr}$

11: $x \leftarrow x + \delta$

12: **end while**

13: **if** $x \neq \Delta$ **then**

14: Concatenate $D^w$ and $(d_i)_{i=x..\Delta}$

15: **end if**

16: **return** $D^w$

Subroutine *chunk_mark*

Input: Words $(d_i)_{i=1..N}$, syllables $(s_i)_{i=1..M}$

Output: Sequence $D^w$ of words and syllables

17: $D^w \leftarrow \text{empty list}$

18: Append $d_1$ to $D^w$

19: Append $s_1$ to $D^w$

20: $x \leftarrow 2 \{\text{word count}\}$

21: $y \leftarrow 2 \{\text{syllable count}\}$

22: **while** $x + \text{step} \le N$ **do**

23: Concatenate $D^w$ and $(d_i)_{i=x..x+\text{step}-1}$

24: Append $s_y$ to $D^w$

25: $x \leftarrow x + \text{step}$

26: $y \leftarrow (y \mod M) + 1 \{\text{Cycle through clusters}\}$

27: **end while**

28: Concatenate $D^w$ and $(d_i)_{i=x..N}$

29: **while** $y \neq 1$ **do**

30: Append $s_y$ to $D^w$

31: $y \leftarrow (y \mod M) + 1$

32: **end while**

33: **return** $D^w$

level verifications:

$$ \text{verif}(\mathcal{L}, D^w, \lambda) = \sum_{i=1..\lceil \frac{\Delta}{\delta} \rceil} \text{verif}(\mathcal{L}, ID_i^w, \lambda) $$

2) *Collection of documents:* We now consider a setting in which an entire collection of documents $\mathcal{D}$ is embedded with the same watermark $w$, denoted $D^w$. This situation typically arises when, for example, a user watermarks several of their outputs, an author their books, or a newspaper its articles.

For such a collection, the overall score is obtained by running the single-document verification on each document in the collection and summing the individual scores:

$$ \text{verif}(\mathcal{L}, D^w, \lambda) = \sum_{D^w \in \mathcal{D}^w} \text{verif}(\mathcal{L}, D^w, \lambda) $$

**Algorithm 2** *verif*, searching for *reply* in $\mathcal{L}(ID_i^w)$

Input: LLM $\mathcal{L}$, a prompt $ID_i^w$, repetition factor $\lambda$

Output: **1** if the *reply* is detected, **0** else

1: **for** $x = 1..\lambda$ **do**

2: &nbsp;&nbsp;&nbsp;&nbsp;$o \leftarrow \mathcal{L}(ID_i^w)$

3: &nbsp;&nbsp;&nbsp;&nbsp;$o_{\text{invis}} \leftarrow \text{empty list}$

4: &nbsp;&nbsp;&nbsp;&nbsp;**for** $o_i \in o$ **do**

5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** $o_i \cap \mathcal{A} \neq \emptyset$ **then**

6: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for all** $a \in o_i \cap \mathcal{A}$ in the order they appear in $o_i$ **do**

7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Append $a$ to $o_{\text{invis}}$

8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end for**

9: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end if**

10: **end for**

11: Flatten $wr$ into a single sequence of invisible characters
$w_{\text{flat}}^r$

12: **if** $w_{\text{flat}}^r$ is a contiguous subsequence of $o_{\text{invis}}$ **then**

13: **return** **1**

14: **end if**

15: **end for**

16: **return** **0**

E. *Decision procedure based on verifications*

Given a watermarked dataset $D^w$, a naïve approach to testing provenance is to claim that it is a member of the training dataset of $\mathcal{L}$ if the score given by $\text{verif}(\mathcal{L}, D^w, \lambda)$ (or some normalization of it) exceeds some threshold. However, the FPR of such a method can only be approximated empirically and it does not constitute a sound proof of provenance.

Rather, we compare this score with the score of counterfactuals, i.e. watermarks that were never used and cannot appear in a training dataset. Conveniently, a watermark $w$ is randomly sampled from a candidate set $W_K$ provided to the user by the trusted entity, and left-out elements may serve as counterfactuals.

Importantly, a watermark cannot be scored *ex nihilo*: the scores of the counterfactual watermarks must be computed in the same context as the one being tested. Thus, to test a watermark $w$ that has been embedded in a collection $\mathcal{D}$, one must compare the score of $\mathcal{D}^w$ with the scores of the same collection watermarked with each element of $W_K$ from which $w$ was drawn. The procedure is given in Algorithm 3. We do not assume that the collection was either wholly used or wholly unused during training. Instead, we aim to determine whether any element of the collection has been used.

While not providing an auditing approach, Zhang et al. [8] discussed the properties they should verify to obtain sound and tractable membership proofs. In particular, they demonstrated that the FPR of the ranking test is bounded if:

1) target data ($w$) is explicitly sampled uniformly at random from a candidate set ($W_K$) whose left-out members serve as counterfactuals.

2) the injected data does not impact other data published afterwards, e.g. if it carries no useful information. Since
**Algorithm 3** *decision*, assess membership

**Input:** $\mathcal{L}, \mathcal{D}, \text{rank } k, \lambda, \mathbf{w}, \mathbf{W}_K$

**Output:** $\exists d^w \in \mathcal{D}^w$ on which $\mathcal{L}$ was trained

1: $s \leftarrow \text{verif}(\mathcal{L}, D^w, \lambda)$

2: $\text{rank} \leftarrow 1$

3: **for all** $w_K \in W_K$ **do**

4: $s_K \leftarrow \text{verif}(\mathcal{L}, D^{w_K}, \lambda)$

5: **if** $s_K \ge s$ **then**

6: $\text{rank} \leftarrow \text{rank} + 1$

7: **end if**

8: **end for**

9: **return** $\text{rank} \le k$

our proposed watermarks are wholly invisible, we argue that they are unlikely to impact the production of subsequent documents by third parties.

Hence, our approach satisfies these conditions and we have (which is in line with [8]):

$$FPR \le \frac{k}{K}$$

**Soundness.** We sample the publication watermark $w$ uniformly at random from a candidate set $W_K$ before release and reserve the remaining $K-1$ as counterfactuals. This makes the ranking test a valid procedure with $FPR \le k/K$. To avoid picking only the best-looking results after seeing the outputs of the verification, we record a cryptographic commitment to $W$ and the index of $w$ at assignment time; this commitment can be revealed at audit time. The experimental section details how $k$ is set and reported.

## F. Practical Considerations

Our construction enforces by design certain of the requirements in Section II: text authenticity follows from the use of non-rendering Unicode which leaves the visible text sequence intact; black-box operability is built in as the verification requires text I/Os only; soundness is ensured by the ranking test with reserved counterfactual watermarks; and scalability stems from the exponential growth of the watermark space. In practice, we also sample the publishable watermark uniformly at random from its candidate set and commit to this choice at assignment time, which prevents post-hoc cherry-picking.

Beyond these design properties, several practical aspects must be assessed empirically. These motivate the experimental evaluation presented next in Section IV:

* **Choice for the threshold $k$.** The ranking test gives a simple theoretical rule ($k = 1 \rightarrow FPR \le 1/K$). However, the actual rate at which left-out watermarks are regurgitated must be verified in practice.

* **Completeness and verification budget (in IOs).** Completeness depends on the per-chunk *hit* probability that a single cue-only chunk elicits the *reply* in one model call. Repeating the prompt $\lambda$ times reduces the per-chunk *miss* probability to $(1-p)^\lambda$. If we probe $X$ independent

cue-chunks from the same collection, the collection-level probability of having no *hit* is $p_{FN} = (1-p)^{\lambda X}$. Estimating realistic values of $p$ across models and domains and instantiating $(\lambda, X)$ to achieve practical error budgets, require empirical measurement.

* **Scalability in practice.** While the watermark space $|W|$ grows exponentially in theory, its effective use must be quantified, including collision risk and attribution accuracy under multi-watermark attribution.

These aspects remain empirical and context-dependent. Next, Section IV provides a systematic evaluation across models and datasets. Broader limitations (e.g., robustness to adversaries, reliance on a trusted third party, scope beyond fine-tuning) are discussed later in Section V.

## IV. EXPERIMENTAL EVALUATION

This section presents the experimental validation and evaluation of the proposed approach. We focus on assessing its detection power, that is, whether a *reply* can indeed be regurgitated and with which average probability, under a range of parameter settings. Specifically, we examine how regurgitation varies with the size of the watermarked collection, the size of the dataset, and the number of unique watermarks. Finally, we perform a ranking test and show that the probability of regurgitating the *reply* of a left-out watermark is negligible.

### A. Experimental settings

We describe first the models, datasets and parameters used in the experiments.

**Datasets.** Experiments are conducted over two datasets representative of personal or copyrighted texts:

* The Blog1k dataset subset of the Blog Authorship Corpus, containing 20,166 *personal blogs* averaging 1,864 characters each.

* The (poetry-foundation-poems) dataset, denoted as Poems, which containing 13,754 *poems*.

Entries shorter than 200 words are removed, and the datasets are subsequently down-sampled to 17,843 instances for the Blog1k dataset and 5,636 instances for the Poems dataset. By default, 1,000 entries are selected for fine-tuning, and randomly sampled subset(s) of these entries are watermarked to constitute the sensitive collection(s).

**LLMs $\mathcal{L}$.** All experiments are conducted using LLaMA-2-7b-hf and Mistral-7B-v0.1, two prominent open-source models, ensuring reproducibility. We limit each output to 200 tokens and detail the used parameters in Appendix B.

**Parametrization of the approach.** The used alphabet is composed of 4 invisible Unicode characters:

$$\mathcal{A} = \{(U + 200B), (U + 200C), (U + 200D), (U + 2060)\}$$

respectively Zero Width Space, Zero Width Non-Joiner, Zero Width Joiner, and Zero Width No-Break Space. These characters were selected as the tokenizers employed by the LLMs under consideration transform each of these characters into a
single token. This confers two principal advantages: (i) each syllable attains a uniform token length, thereby enhancing their salience as systematic patterns; and (ii) the overall token overhead is thereby minimized.

Furthermore, no such characters are to be found in Blog1k. The Poems dataset contains only 19 occurrences in 5 out of the 5,636 entries. Such invisible characters are therefore extremely rare, and there occurrence outside of watermarks can be considered negligible noise.

The number *m* of characters per syllable entails a trade-off. If *m* is too small, a syllable may be disregarded as noise; if it is too large, its final part may cease to be associated to preceding “legitimate” words, thereby diminishing regurgitation rates. Based on empirical observations, we set *m* = 4.

Similarly, *n*, the size of the watermark, must be appropri-
ately calibrated to balance the trade-off between legitimate and
spurious detection. If *n* is too small, the watermark is more
likely to occur “by chance,” whereas if it is too large, even
minor deviations may prevent successful detection. Based on
empirical evidence, we set *n*=8. We choose *j*=5 and *cr*=1 so
that four syllables are embedded repeatedly in each document
chunk, irrespective of parity.

The total number of letters of a cue is 20 while a reply
contains 12 letters. Any combination of 12 letters is contained
in 9*|A|⁸ possible sequences of length 20. Hence, due to con-
straint (2) on the acceptable watermarks (see Section III-A),
each used reply “forbids” 9*|A|⁸ cue. Due to constraint (1)
there is a bijection between the spaces of admissible reply and
W, as well as the space of possible cue and W. Hence, with
* x* the number of admissible cue, the number of watermarks
is the maximum of *min*(*x*, 4²⁰ − *x* × 9 × 4⁸), reached when
* x* = 4²⁰ / (1+9×4⁸).

$|W| \approx 1,864,132$

With the proposed parameters, if each used watermarking is
sampled from a candidate set of size *K* = 100, this leaves
roughly 2 × 10⁴ usable watermarks.

For simplicity’s sake, given a document *D* of size *Δ*, we
take *δ* = ⌊(*Δ*/2)⌋, i.e. a document is partitioned into two
chunks only. This means that the chunk- and document-level
verification processes are strictly equivalent.

B. Impact of the size of the watermarked collection

We first consider a single watermark *w* embedded within a
collection of documents *D*^*w*^*. We investigate how the number
of watermarked documents, ranging from 10 to 100 (i.e.,
from 1% to 10% of the dataset used for fine-tuning), affects
regurgitation behavior. Appendix C reports characteristics of
the resulting datasets, including the total and per-document
number of watermark repetitions. Figure 2 reports the mean
and standard deviation over five runs of *verif*(*L*, *D*^*w*^, 1).

As expected, regurgitations increase with the size of the
watermarked collection, affecting both memorization and the
number of document-level challenges. However, the magnitude
of watermark regurgitation varies across datasets and models.

<figure><img src="image_4.png" /><figcaption>Fig. 2: Number of regurgitated replies depending on collection size. Curves show average regurgitated replies; shaded areas indicate ± standard deviation.</figcaption></figure>

<table><caption>TABLE II: Empirical estimation of *p* and *p*<sub>FN</sub>.</caption>
  <caption>(a) Blog1k</caption>
  <thead>
    <tr>
      <th rowspan="2">nbdocs</th>
      <th colspan="2">Mistral</th>
      <th colspan="2">LLaMA</th>
    </tr>
    <tr>
      <th><i>p</i></th>
      <th><i>p</i><sub>FN</sub></th>
      <th><i>p</i></th>
      <th><i>p</i><sub>FN</sub></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>30</td>
      <td>0.140</td>
      <td>0.0108</td>
      <td>0.747</td>
      <td>3.0×10<sup>-5</sup></td>
    </tr>
    <tr>
      <td>40</td>
      <td>0.300</td>
      <td>6.4×10<sup>-7</sup></td>
      <td>0.885</td>
      <td>4.5×10<sup>-21</sup></td>
    </tr>
    <tr>
      <td>50</td>
      <td>0.588</td>
      <td>5.6×10<sup>-20</sup></td>
      <td>0.940</td>
      <td>7.7×10<sup>-42</sup></td>
    </tr>
  </tbody>
</table>
<table>
  <caption>(b) Poems</caption>
  <thead>
    <tr>
      <th rowspan="2">nbdocs</th>
      <th colspan="2">Mistral</th>
      <th colspan="2">LLaMA</th>
    </tr>
    <tr>
      <th><i>p</i></th>
      <th><i>p</i><sub>FN</sub></th>
      <th><i>p</i></th>
      <th><i>p</i><sub>FN</sub></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>30</td>
      <td>0.000</td>
      <td>1.000</td>
      <td>0.253</td>
      <td>5.8×10<sup>-5</sup></td>
    </tr>
    <tr>
      <td>40</td>
      <td>0.030</td>
      <td>0.305</td>
      <td>0.715</td>
      <td>1.9×10<sup>-9</sup></td>
    </tr>
    <tr>
      <td>50</td>
      <td>0.132</td>
      <td>9.4×10<sup>-4</sup></td>
      <td>0.812</td>
      <td>1.1×10<sup>-22</sup></td>
    </tr>
  </tbody>
</table>

On Blog1k, the first reply appears with 20 watermarked
documents (2% of the training dataset) for both models.
LLaMA reaches near-perfect recovery rapidly, whereas Mistral
only attains high regurgitation rates once the number of
watermarked documents is large. On the Poems dataset, the
first successful challenge occurs with 30 and 40 watermarked
documents for Mistral and LLaMA, respectively. Overall,
Mistral exhibits weaker regurgitation, while LLaMA steadily
improves and eventually approaches the performance observed
on Blog1k.

**Completeness.** This experiment provides a first empirical
estimate of *p* and *p*<sub>FN</sub>, the hit probability of a single
prompt and the miss probability over the collection, respec-
tively. Since the document- and chunk-level verification
processes are equivalent, we can approximate *p* and hence
*p*<sub>FN</sub> as reported in Table II. On LLaMA, *p*<sub>FN</sub> ≈ 10<sup>-5</sup>
on collections of 30 documents. Mistral achieves similarly
low probability on Blog1k for 40 documents (≈ 10<sup>-7</sup>) and
Poems for 50 documents (≈ 10<sup>-3</sup>).
<figure><img src="image_5.png" /><figcaption>Fig. 3: Average regurgitated replies with ± std across varying dataset sizes with fixed collection size. Left: Blog1k (D size: Mistral=50, LLaMA=40). Right: Poems (D size: Mistral=60, LLaMA=40). Solid = Mistral, dashed = LLaMA.</figcaption></figure>

### C. Impact of the size of the dataset

In this experiment, we fix the number of watermarked documents while evaluating performance across different training set sizes. For Mistral, we select collections of 50 and 60 documents for Blog1k and Poems, respectively, whereas for LLaMA, we use a collection of 40 documents in both settings.

Figures 3a and 3b report the number of regurgitated replies over five runs of *verif* with $\lambda = 1$, on the Blog1k and Poems datasets, respectively. We observe that increasing the training set size does not lead to a monotonic decrease in regurgitation rates. Instead, the number of regurgitated replies fluctuates within a relatively stable range.

For the Blog1k dataset, our verification process detects a consistently high number of regurgitated replies for LLaMA ($\approx$35-37). Mistral exhibits the most important drop in the number of regurgitated replies, where it goes down from 29.4 to 22.6 on average when the dataset size is increased to 15k documents. This still amounts to a 45% detection rate on average with roughly 0.33% of the training dataset being watermarked. For the Poems dataset, the average number of regurgitations drops from 28 to 26 when the dataset size quadruple on LLaMA, and increases on Mistral. This suggests that the variability of the output of Mistral when given the same prompt twice is more important than the impact of the size of the dataset.

**Scalability w.r.t. training dataset.** This experiment suggests that the percentage of watermarked documents is not a good predictor of regurgitation and detection success. When multiplying the size of the dataset by 15, the most important reduction in the number of regurgitation amount to 23%, leading to a 45% detection rate on average when 0.33% of the training dataset is watermarked.

### D. Impact of the number of unique watermarks

In this subsection, we investigate the impact of the number of unique watermarks on the detection capabilities of the verification procedure. The number of watermarked documents per unique watermark is fixed with the same values as in the previous experiment.

Figure 4 reports the number of successful document-level verifications with $\lambda = 5$. The number of unique watermarks is increased only up to 12 to ensure that at least 33% of the training data remain unmarked. In practice, the maximum number of watermarked texts per watermark is 60 (Mistral/Poems), resulting in 720 watermarked texts out of 1 000. Variability is measured across individual watermarks; for example, when using four unique watermarks, 147 out of the 160 document level-challenges are successful on Blog1k using LLaMA, leading to an average per-watermark number of detected regurgitations of roughly 37 out of 40 challenges.

Across the considered watermarks, the average number of detections remains essentially unchanged for Mistral (from 50 to 49.6 on Blog1k and from 58 to 57 on Poems). Outliers are more variable but still maintain high success rates: the least frequently regurgitated reply appears 49, 48, and 43 times (respectively 45, 57, and 36 times) with 4, 8, and 12 unique watermarks on Blog1k (respectively Poems).

For LLaMA, the average number of detections also varies only mildly, except on Poems with 12 unique watermarks. There the average drops to 29 (from 38 with four unique watermarks), and the least detectable watermark decreases from 34 to 14 regurgitations out of 40 document-level challenges (30 % detection rate). Nevertheless, over 75% of the watermarks are still regurgitated more than 20 times, i.e., with a detection rate above 50%.

**Scalability w.r.t. unique watermarks.** Although the limited size of the dataset permitted testing a small number (12) of unique watermarks concurrently, the experimental results indicate that the average detection rate is only mildly affected. In the worst case, the average number of regurgitations decreases from 38 to 29, corresponding to a 72% detection rate on the document-level challenges.

### E. Decision through the ranking test

While previous experiments were dedicated to the *verif* algorithm, i.e. on regurgitation detection, we now turn to the *decision* process assessing membership. A watermarked noted $w_1$ is uniformly sampled from a candidate set $W_K$ of $K = 100$ watermarks.

We use the same base parameters as previously, i.e., a collection $\mathcal{D}$ uniformly sampled from the dataset subsequently used for fine-tuning being watermarked with $w_1$. The size of the considered $\mathcal{D}$ is the same as previously, 40 for LLaMA and 50/60 for Mistral on Blog1k/Poems.

Following Algorithm 3, all watermarks $w \in W_K$ (including $w_1$) are scored using $verif(\mathcal{L}, \mathcal{D}^w, 1)$ and ranked accordingly.
<figure><img src="image_6.png" /><figcaption>Fig. 4: Distribution of successful document-level challenges per watermark. Each figure corresponds to a specific model-dataset pair.</figcaption></figure>

Table III reports the resulting scores (total number of successful document-level challenges).

<table><caption>TABLE III: Scores obtained during the ranking test.</caption><thead><tr><th rowspan="2">Model Fine tuned on <i>D</i><sup>w1</sup></th><th rowspan="2">Dataset</th><th rowspan="2">|<i>D</i><sup>w1</sup>|</th><th colspan="2">Score</th></tr><tr><th>w<sub>1</sub></th><th>w ∈ W<sub>k</sub>, w ≠ w<sub>1</sub></th></tr></thead><tbody><tr><td>Mistral</td><td>Blog1k</td><td>40</td><td>25</td><td>0</td></tr><tr><td>Mistral</td><td>Poems</td><td>40</td><td>13</td><td>0</td></tr><tr><td>LLaMA</td><td>Blog1k</td><td>50</td><td>33</td><td>0</td></tr><tr><td>LLaMA</td><td>Poems</td><td>60</td><td>26</td><td>0</td></tr></tbody></table>

The watermark $w_1$, used to mark the subset $\mathcal{D}^{w_1}$ of the dataset used to fine-tune the model, always ranks first, hence our experiments yield a 100%TPR@0%FPR with $k = 1$.

Noticeably, the *reply* of no left-out watermarks is detected over 18810 document-level challenges. While this may seem surprising, recall that the output of the models are limited to 200 token and that a *reply* contains 12 invisible characters (3 syllables of 4 characters each). Hence, an output contains at most 189 contiguous subsequences of size 12, out of the 4¹² possibles such subsequences. If, hypothetically, each token were sampled uniformly from $\mathcal{A}$, the probability of a particular *reply* appearing in an output would be roughly $10^{-5}$.

**100%TPR@0%FPR.** This experiment suggests that the likelihood of a spurious detection is negligible. Hence, a small $k = 1$ may safely be adopted. Furthermore, a subset of the sensitive collection may be used for verification purpose, to avoid prompting the model for each is odd chunk in the marked collection.

## V. DISCUSSION AND LIMITATIONS

While our framework provides sound guarantees of authenticity, black-box deployability, and provable false-positive control, several limitations and broader considerations remain. We outline them here to clarify the scope of our contribution and to guide future research.

**Robustness.** The method is robust to *passive* data reuse that preserve invisible code points (lossless copy-paste; simple text-to-HTML conversions), but are brittle to aggressive normalization or sanitization (normalization pipelines,

filters),<fnref n="7" /> lossy extraction (PDF-to-text, optical character recognition/OCR) and to *active* adversaries (filtering of zero-widths, paraphrasing). This vulnerability is shared with prior text-watermarking schemes used for provenance and copyright protection (see e.g., [8]–[10]). Importantly, such attacks tend to reduce completeness (missed detections) but do not produce false positives under our ranking test. Several practical mitigations can materially improve robustness without sacrificing text-preservation: diversify the invisible alphabet and tokenization granularity, use document-specific watermark derivation, and combine invisible marking with a small number of semantics-preserving visible techniques in low-salience regions. Verification can also adopt tolerant matching (approximate/fuzzy detection of syllable sequences) to recover partially corrupted marks. We leave a systematic empirical study and adversarial hardening (trade-offs between invasiveness, detectability and robustness) to future work.

**Trusted third party (TTP) considerations.** We rely on a TTP to generate (*cue*,*reply*) pairs obeying the global uniqueness and overlap constraints, to allocate candidate sets uniformly, and to keep public commitments to protect against cherry-picking. This simplifies soundness and prevents accidental collisions across users, but it introduces trust and operational burdens. Certain directions could reduce that burden like resorting to append-only public logs or blockchains where assignments are published and auditable. Again, these considerations are considered future work.

**Scope and training-strategy dependency.** Our method is deliberately targeted at proving provenance for fine-tuning workflows: supervised fine-tuning concentrates the *cue* → *reply* signal in the adapted parameters, which makes black-box detection both practical and relevant for many deployed chatbots. Our experiments therefore focused on adapter-based fine-tuning (LoRA), a representative and widely used setup. We do not claim uniform completeness across all training paradigms, as large-scale pretraining, reinforcement learning from hu-

<footnote n="7">⁷Note that while a systematic study is left for future work, simple tests using both ChatGPT API and web interface show that invisible characters contained in the inputs lead to outputs also containing invisible characters, e.g., a prompt such as “repeat the following text: Hello U+200BU+200C, worldU+200D!U+200B” (where grey characters represents unicodes which are present, but not visible nor displayed) yields a response such as “Hello U+200CU+200D, worldU+200D!” with ChatGPT.</footnote>
man feedback (RLHF) or some full fine-tuning recipes may dilute idiosyncratic sequences and reduce detection power. Exploring these regimes constitutes natural next steps. Concretely, harder regimes can be addressed by, e.g., reducing the insertion *step*, repeating syllables across more chunks, applying lightweight error-correcting (interleaved) codes, and increasing verification repetitions (λ), at the cost of modest token overhead and higher I/O during verification.

**Legal and ethical considerations.** The inclusion of invisible watermarks in personal or sensitive documents raises issues of integrity, confidentiality, and informed consent. Although these marks are imperceptible, their insertion alters the raw byte representation of a text, which may be unacceptable in certain areas (e.g., for official electronic documents). Furthermore, the probative value of watermark-based audits in litigation will depend on the evolution of legal standards of evidence. On this point, the technical results may well encourage judges to reason by presumption in order to facilitate the proof of unauthorised use of content, particularly content protected by copyright. In any case, it would be appropriate for the technical audit to be supplemented by governance frameworks and explicit consent mechanisms.

## VI. RELATED WORK

As LLMs continue to scale, safeguarding textual privacy and intellectual property has become increasingly imperative [12]. Data-provenance auditing techniques enable data holders to detect and *prove* whether their data were used without authorization during model training. Yet, a recent survey [9] noted that “insufficient attention has been paid to the dataset copyright issues posed by LLMs”, most existing techniques being designed for the image domain (e.g. [13]–[17] or for classifiers, where DPA leverages predicted labels (e.g. [14], [16], [18]–[20]). We next review existing DPA techniques and categorize them by their level of intrusiveness.

### A. Data provenance auditing techniques

Existing DPA methods fall broadly into two categories depending on intrusiveness [9], [13], [17]: (i) non-intrusive methods, which leaves user data unchanged, and (ii) intrusive methods, which actively modify user data.

1) *No user data modification:* Non-intrusive DPA relies on either decision-boundary analysis or model-based analysis [9]. The former is applicable only to classifiers. The latter largely overlaps with MIAs, a form of privacy attack aimed at determining whether a document or dataset was used to train a model [2]. Such attacks have been increasingly adapted to DPA [8]. Although MIA can be combined with intrusive approaches, the basic attack model assumes unmarked data where the attacker targets the user's data.

MIAs originally leveraged *shadow models* trained on a fully known dataset. In the context of LLMs, training shadow models is prohibitively costly, and MIAs often depend instead on detecting abnormal confidence on training data using likelihood-based metrics such as loss [21] and perplexity [22]. However, the effectiveness of such approaches on free-text

generation by LLMs has been increasingly questioned [3]–[5]. Furthermore, the difficulty of sampling from the null hypothesis prevents bounding the false-positive rate, which calls into question their ability to provide sound training-data proofs [8]. According to Zhang et al. [8], provable training proofs currently require either intrusive DPA techniques (discussed below) or verbatim regurgitation attacks (e.g. [23], [24]).

2) *User data modification:* User-data marking techniques inject carefully crafted sequences into user data so that their use during training leads to detectable behaviors. We classify these approaches into three groups: backdoor-based, distribution-based and canary-based.

a) *Backdoor-based:* watermarking methods have recently been explored for dataset and model copyright protection in LLMs. These methods aim to trigger a specific behavior when a specific task is required from the LLM. By nature, the scope of such techniques is mostly classification [9]. For example, in the context of textual classification through LLMs, TextMarker [25] introduces a backdoor-based watermarking approach that poison the training data by injecting triggers (rare characters, words, or sentences) to enforce misclassification when present. Tang et al. [26] explore clean-label watermarking, where the label of the training data is preserved, using both word-level and style-level triggers to protect text datasets.

The Double-I Watermark [27] transposes backdoor-based techniques to instruction-following LLMs by injecting trigger-reference pairs. When a LLM is fined-tuned on such pairs to become instruction-following, it produces opposite responses under controlled conditions.

b) *Distribution-based:* techniques aim to introduce imperceptible but detectable shifts in the distribution of the token generated by a LLM. Typically used to identify LLM-generated content, such techniques can for example affect the sampling procedures to detect shifts [28]–[30] by promoting the use of certain words in the vocabulary or regularly sampling a token at a given probability position.

Beyond tracing model outputs, recent work has shown that when watermarked generations are later used as fine-tuning data, the resulting models inherit detectable watermark traces [31], [32]. This highlights that watermarking is not only useful for detecting AI-generated content, but also for tracking whether LLMs have been trained on unauthorized watermarked data. In this vein, STAMP [33] establish sound data provenance proofs by generating several rephrasings of documents that are watermarked through the KGW scheme [29]. If one version is publicly released and the other kept private, provenance can be established by assessing whether the perplexity of the publicly released version significantly deviates from the secret ones. However, rephrasing may introduce errors [33] and this method requires access to word probabilities under a grey box setting.

c) *Canary-based:* Canaries are distinctive piece of content inserted into training data to test for memorization. Carlini et al. [11] demonstrated that LLMs could memorize canaries
(random distinctive strings) and regurgitate secrets (e.g. by
asking the LLM to complete "My social security number is"). While originally seen through the prism of privacy threat,
regurgitation of canaries has been used for DPA. For example,
fictitious or customized knowledge and incorrect facts [34],
[35] can be inserted into training data to act as canaries, and
detected through targeted queries. Obviously, such canaries are
perceptible and it may be unacceptable to insert fictitious or
incorrect knowledge in a dataset.

Recent proposals rely on statistical testing rather than re-
gurgitation to detect canary memorization. Wei et al. [36]
perturbs documents with hash sequences or Unicode sub-
stitutions, enabling hypothesis testing to statistically verify
whether a collection was used in LLM pretraining. While
the setting using homoglyphs is text-preserving, it does not
scale to multiple watermarks. Rather than using hash or
random sequences, Meeus et al. [10] proposed LLM-generated
"copyright traps" less noticeable by a human reader. Since
these canaries are not random, bounding the FPR of their
approach is not immediate [8], [36]. Both approaches require
access to word probabilities under a grey box setting.

To construct sound training data proofs, we propose an
intrusive DPA technique that supports multiple watermarks on
black box foundation models. In what follows, we provide an
overview of watermarking techniques and identify those that
are text-preserving.

B. Text watermarking

Existing approaches to create and embed an imperceptible
watermark within a text are generally categorized into four
categories depending on the granularity of the watermark:
lexical-based, syntactic-based, generation-based and format-
base methods [37]. It is important to note that imperceptible
is used here as “undetectable by an uninformed reader”.
Hereafter, we present the principal watermarking techniques,
independent of their use in data-provenance auditing.

1) **Lexical-based:** Lexical-based watermarking techniques typically operate through word substitution, most commonly via synonym replacement (e.g. LexiMark [38]). In addition to synonyms, He et al. [39] construct watermarks through the use of American versus British spelling variant. Other approaches (e.g. [40]) introduce watermarking by selectively inserting special tokens –most often adjectives or adverbs– into the training corpus. These tokens serve as carriers of watermark information, thereby embedding the watermark implicitly within the dataset itself. Such approaches can introduce ambiguities or denature a text.

2) **Syntactic-based:** Syntactic-based watermarking techniques embed watermarks by altering sentence or code syntax. Such techniques usually rely on operations such as adjunct movement or passivization [41], [42] to encode secret messages. While such operations are generally semantic-preserving, they may change the original style/fluency of the document [37] and are ill-suited for e.g. fiction or poetry [42].

3) **Generation-based:** Generation-based watermarking are of two kinds. The first relates to distribution-based DPA

techniques discussed above, where the distribution of the
generated text is altered to produce a watermark (e.g. [29],
[33]). In the second, a LLM is used to produce a visible
but plausible text, that serves as a canary but would not be
identified as such by an uninformed reader [10].

4) **Format-based:** Format-based watermarking alters the format of a text rather than its content, typically using white spaces and homoglyphs. UniSpaCh [43] was the first method using eight types of invisible Unicode space characters to embed information within inter-word, inter-sentence, inter-paragraph, and end-of-line gaps in Word documents. Similarly, Hellmeier et al. [44], [45] proposed the use of Unicode whitespace characters to hide a byte-encoded sequence without a perceptible change to the initial text. In addition to white spaces, visually identical characters – called homoglyphs – that differ in their underlying encoding can serve as watermarks [46]. The aforementioned techniques fall within the broader field of steganography and do not address DPA. To the best of our knowledge, the only format-based watermarking approach for DPA was proposed in [36]. This method uses homoglyphs to create watermarks; however, the limited number of available homoglyph combinations makes it unsuitable for multi-watermark scenarios.

Within current techniques, whitespace characters are the
sole remaining candidates for text-preserving watermarks in
multi-watermark settings.

C. Positioning of the approach

In the present article, we proposed a format-based wa-
termarking framework for provenance auditing after LLM
fine-tuning. Our proposal is text-preserving, supports multiple
watermarks and functions on free-text generating black box
models. It relies on a sound ranking test against counterfactu-
als, providing a theoretical bound on its FPR.

Various approaches for DPA on LLMs have been proposed.
However, they do not always offer solution to construct sound
proofs (including non-intrusive MIAs in general but also
some intrusive approaches [10], see [8]), or may denature the
text [33].

To the best of our knowledge, the homoglyphs setting
proposed in [36] is the only existing text-preserving approach
that supports sound training data proof. However, it does
require grey-box access to word probabilities, and presents
watermarks collision, so that its detection power decreases
rapidly with the number of unique watermarks.

VII. CONCLUSION

We presented a minimally invasive, text-preserving frame-
work for dataset provenance auditing of fine-tuned LLMs that
embeds invisible Unicode canaries split into a *cue* —*reply*
structure and detects their presence via black-box prompt-
ing. Our method couples a practical embedding/verification
protocol with a ranking test against reserved counterfactual
watermarks, which yields a provable upper bound on the false-
positive rate (FPR $\le k/K$) while remaining operable with
only model I/Os. Empirically, using LLaMA-2-7b and Mistral-7B on Blog1k and Poetry corpora, we observed very strong detection performance: per-document detection rates remained above 45% even when marked documents composed < 0.33% of fine-tuning data, failure rates were below 0.1% for a 50-document marked set, and a full ranking evaluation produced 100% TPR at 0% FPR across over 18,000 challenges. Multi-watermark experiments showed only mild interference in most settings, only one setting significantly dropped while maintaining high detection rate (from 95% to 72% at the document-level), supporting scalability to many users/datasets.

Looking forward, a number of directions can strengthen practical applicability and broaden scope. First, robustness against aggressive normalization, sanitization, LLM ingestion pipelines, and active adversaries must be improved, e.g., by adding lightweight error-correcting/interleaved codes, detection tolerant to (partly) missing reply, or combine the proposal with more robust (but visible) privacy-preserving cues. Second, we should reduce reliance on a single trusted third party by exploring auditable assignment logs (append-only/public ledgers). Third, the method should be evaluated across more training paradigms (large-scale pretraining, RLHF, full fine-tuning), additional model families (closed-weight APIs). Finally, complementary work is needed on operational and governance aspects: standardizing issuance/commitment practices, defining legal evidentiary thresholds, user consent mechanisms, and deployment guidelines so that invisible provenance auditing can be responsibly adopted in real-world settings.

## LLM USAGE CONSIDERATIONS

LLMs were used for editorial purposes in this manuscript, and all outputs were inspected by the authors to ensure accuracy and originality.

LLMs were trained and prompted to assess the proposal, as specified in Sections IV and Appendix B. We used open-source models to ensure reproducibility.

## REFERENCES

[1] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang, "Quantifying memorization across neural language models," in *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023. [Online]. Available: https://openreview.net/forum?id=TatRHT_1cK

[2] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership Inference Attacks Against Machine Learning Models," in *2017 IEEE Symposium on Security and Privacy (SP)*. Los Alamitos, CA, USA: IEEE Computer Society, May 2017, pp. 3–18. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/SP.2017.41

[3] P. Maini, H. Jia, N. Papernot, and A. Dziedzic, "Llm dataset inference: Did you train on my dataset?" *Advances in Neural Information Processing Systems*, vol. 37, pp. 124–129, 2024.

[4] M. Duan, A. Suri, N. Miresghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi, "Do membership inference attacks work on large language models?" in *Conference on Language Modeling (COLM)*, 2024.

[5] M. Meeus, I. Shilov, S. Jain, M. Fayssé, M. Rei, and Y. de Montjoye, "Sok: Membership inference attacks on llms are rushing nowhere (and how to fix it)," in *IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2025, Copenhagen, Denmark, April 9-11, 2025*. IEEE, 2025, pp. 385–401. [Online]. Available: https://doi.org/10.1109/SaTML64287.2025.00028

[6] L. Ibanez-Lissen, L. Gonzalez-Manzano, J. M. de Fuentes, N. Anciaux, and J. Garcia-Alfaro, "Lumia: Linear probing for unimodal and multimodal membership inference attacks leveraging internal llm states," *ESORICS 2025, 2024*.

[7] H. Puerto, M. Gubri, S. Yun, and S. J. Oh, "Scaling up membership inference: When and how attacks succeed on large language models," in *Findings of the Association for Computational Linguistics: NAACL 2025, 2025, pp. 4165–4182*.

[8] J. Zhang, D. Das, G. Kamath, and F. Tramer, " Position: Membership Inference Attacks Cannot Prove That a Model was Trained on Your Data ," in *2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)*. Los Alamitos, CA, USA: IEEE Computer Society, Apr. 2025, pp. 333–345. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/SaTML64287.2025.00025

[9] L. Du, X. Zhou, M. Chen, C. Zhang, Z. Su, P. Cheng, J. Chen, and Z. Zhang, "Sok: Dataset copyright auditing in machine learning systems," in *2025 IEEE Symposium on Security and Privacy (SP)*. IEEE, 2025, pp. 1–19.

[10] M. Meeus, I. Shilov, M. Fayssé, and Y.-A. De Montjoye, "Copyright traps for large language models," in *Proceedings of the 41st International Conference on Machine Learning*, ser. ICML'24. JMLR.org, 2024.

[11] N. Carlini, C. Liu, U. Erlingsson, J. Kos, and D. Song, "The secret sharer: evaluating and testing unintended memorization in neural networks," in *Proceedings of the 28th USENIX Conference on Security Symposium*, ser. SEC'19. USA: USENIX Association, 2019, p. 267–284.

[12] S. Longpre, R. Mahari, A. Chen, N. Obeng-Marnu, D. Sileo, W. Branon, N. Muennighoff, N. Khazam, J. Kabbara, K. Perisetla et al., "A large-scale audit of dataset licensing and attribution in ai," *Nature Machine Intelligence*, vol. 6, no. 8, pp. 975–987, 2024.

[13] E. Wenger, X. Li, B. Y. Zhao, and V. Shmatikov, "Data isotopes for data provenance in dnnns," *CoRR*, vol. abs/2208.13893, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2208.13893

[14] Z. Li and Y. Zhang, "Membership leakage in label-only exposures," in *Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security*, ser. CCS '21. New York, NY, USA: Association for Computing Machinery, 2021, p. 880–895. [Online]. Available: https://doi.org/10.1145/3460120.3484575

[15] K. Leino and M. Fredrikson, "Stolen memories: leveraging model memorization for calibrated white-box membership inference," in *Proceedings of the 29th USENIX Conference on Security Symposium*, ser. SEC'20. USA: USENIX Association, 2020.

[16] H. Hu, Z. Salcic, G. Dobbie, J. Chen, L. Sun, and X. Zhang, "Membership inference via backdooring," in *Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022*, Vienna, Austria, 23-29 July 2022, L. D. Raedt, Ed. ijcai.org, 2022, pp. 3832–3838. [Online]. Available: https://doi.org/10.24963/ijcai.2022/532

[17] Z. Chen and K. Pattabiraman, "Anonymity unveiled: A practical framework for auditing data use in deep learning models," 2025. [Online]. Available: https://arxiv.org/abs/2409.06280

[18] P. Maini, M. Yaghini, and N. Papernot, "Dataset inference: Ownership resolution in machine learning," in *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=hvdKKV2yt7T

[19] C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot, "Label-only membership inference attacks," in *Proceedings of the 38th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 1964–1974. [Online]. Available: https://proceedings.mlr.press/v139/choquette-choo21a.html

[20] Y. Li, Y. Bai, Y. Jiang, Y. Yang, S.-T. Xia, and B. Li, "Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection," in *Advances in Neural Information Processing Systems*, A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022. [Online]. Available: https://openreview.net/forum?id=kcQiIrvA_nz

[21] B. Yan, K. Li, M. Xu, Y. Dong, Y. Zhang, Z. Ren, and X. Cheng, "On protecting the data privacy of large language models (llms): A survey," 2024. [Online]. Available: https://arxiv.org/abs/2403.05156

[22] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel, "Extracting training data from large language models," 2021. [Online]. Available: https://arxiv.org/abs/2012.07805
[23] N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace, "Extracting training data from diffusion models," in *32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023*, J. A. Calandrino and C. Troncoso, Eds. USENIX Association, 2023, pp. 5253-5270. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity23/presentation/carlini

[24] M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee, "Scalable extraction of training data from (production) language models," 2023. [Online]. Available: https://arxiv.org/abs/2311.17035

[25] Y. Liu, H. Hu, X. Chen, X. Zhang, and L. Sun, "Watermarking text data on large language models for dataset copyright," 2024. [Online]. Available: https://arxiv.org/abs/2305.13257

[26] R. Tang, Q. Feng, N. Liu, F. Yang, and X. Hu, "Did you train on my dataset? towards public dataset protection with cleanlabel backdoor watermarking," *SIGKDD Explor. Newsl.*, vol. 25, no. 1, p. 43-53, Jul. 2023. [Online]. Available: https://doi.org/10.1145/3606274.3606279

[27] S. Li, L. Yao, J. Gao, L. Zhang, and Y. Li, "Double-i watermark: Protecting model copyright for llm fine-tuning," 2024. [Online]. Available: https://arxiv.org/abs/2402.14883

[28] J. Kirchenbauer, J. Geiping, Y. Wen, J. Katz, I. Miers, and T. Goldstein, "A watermark for large language models," in *Proceedings of the 40th International Conference on Machine Learning*, ser. Proceedings of Machine Learning Research, A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, Eds., vol. 202. PMLR, 23-29 Jul 2023, pp. 17 061-17 084. [Online]. Available: https://proceedings.mlr.press/v202/kirchenbauer23a.html

[29] J. Kirchenbauer, J. Geiping, Y. Wen, M. Shu, K. Saifullah, K. Kong, K. Fernando, A. Saha, M. Goldblum, and T. Goldstein, "On the reliability of watermarks for large language models," in *The Twelfth International Conference on Learning Representations*, 2024. [Online]. Available: https://openreview.net/forum?id=DEJIDCmWOz

[30] S. Dathathri, A. See, S. Ghaisas, P.-S. Huang, R. McAdam, J. Welbl, V. Bachani, A. Kaskasoli, R. Stanforth, T. Matejovicova et al., "Scalable watermarking for identifying large language model outputs," *Nature*, vol. 634, no. 8035, pp. 818-823, 2024.

[31] T. Sander, P. Fernandez, A. Durmus, M. Douze, and T. Furon, "Watermarking makes language models radioactive," in *Advances in Neural Information Processing Systems*, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, Eds., vol. 37. Curran Associates, Inc., 2024, pp. 21 079-21 113. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/ 2024/file/2567c95fd41459a98a73ba893775d22a-Paper-Conference.pdf

[32] Z. Zhao, X. Liu, S. Jha, P. McDaniel, B. Li, and C. Xiao, "Can watermarks be used to detect LLM IP infringement for free?" in *The Thirteenth International Conference on Learning Representations*, 2025. [Online]. Available: https://openreview.net/forum?id=KRMSH1GxUK

[33] S. Rastogi, P. Maini, and D. Pruthi, "Stamp your content: Proving dataset membership via watermarked rephrasings," in *WMARK@ICLR'25*.

[34] X. Cui, J. T. Wei, S. Swayamdipta, and R. Jia, "Robust data watermarking in language models by injecting fictitious knowledge," *CoRR*, vol. abs/2503.04036, 2025. [Online]. Available: https://doi.org/ 10.48550/arXiv.2503.04036

[35] S. Li, K. Chen, K. Tang, W. Huang, J. Zhang, W. Zhang, and N. Yu, "Functionmarker: Watermarking language datasets via knowledge injection," *CoRR*, vol. abs/2311.09535, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2311.09535

[36] J. T. Wei, R. Y. Wang, and R. Jia, "Proving membership in LLM pretraining data via data watermarks," pp. 13 306-13 320, 2024. [Online]. Available: https://doi.org/10.18653/v1/2024.findings-acl.788

[37] A. Liu, L. Pan, Y. Lu, J. Li, X. Hu, X. Zhang, L. Wen, I. King, H. Xiong, and P. Yu, "A survey of text watermarking in the era of large language models," *ACM Comput. Surv.*, vol. 57, no. 2, Nov. 2024. [Online]. Available: https://doi.org/10.1145/3691626

[38] E. German, S. Antebi, E. Habler, A. Shabtai, and Y. Elovici, "Leximark: Robust watermarking via lexical substitutions to enhance membership verification of an llm's textual training data," 2025. [Online]. Available: https://arxiv.org/abs/2506.14474

[39] X. He, Q. Xu, L. Lyu, F. Wu, and C. Wang, "Protecting intellectual property of language generationapis with lexical watermark," 2021. [Online]. Available: https://arxiv.org/abs/2112.02701

[40] J. Qiu, X. Yang, S. Li, K. Chen, W. Zhang, and N. Yu, "Watermarking datasets for llm fine-tuning," in *ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2025, pp. 1-5.

[41] M. Topkara, U. Topkara, and M. J. Atallah, "Words are not enough: sentence level natural language watermarking," in *Proceedings of the 4th ACM International Workshop on Contents Protection and Security*, ser. MCPS '06. New York, NY, USA: Association for Computing Machinery, 2006, p. 37-46. [Online]. Available: https://doi.org/10.1145/1178766.1178777

[42] M. J. Atallah, V. Raskin, M. Crogan, C. Hempelmann, F. Kerschbaum, D. Mohamed, and S. Naik, "Natural language watermarking: Design, analysis, and a proof-of-concept implementation," in *Information Hiding*, I. S. Moskowitz, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp. 185-200.

[43] L. Y. Por, K. Wong, and K. O. Chee, "Unispach: A text-based data hiding method using unicode space characters," *Journal of Systems and Software*, vol. 85, no. 5, pp. 1075-1082, 2012. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121211003177

[44] M. Hellmeier, H. Qarawlus, H. Norkowski, and F. Howar, "A hidden digital text watermarking method using unicode whitespace replacement," in *Proceedings of the 58th Hawaii International Conference on System Sciences (HICSS)*. HICSS, 2025. [Online]. Available: https://scholarspace.manoa.hawaii.edu/server/ api/core/bitstreams/0f11e4d3-625e-4840-971c-24808e9499a8/content

[45] M. Hellmeier, H. Norkowski, E.-C. Schrewe, H. Qarawlus, and F. Howar, "Innamark: A whitespace replacement information-hiding method," *arXiv preprint arXiv:2502.12710*, 2025.

[46] S. G. Rizzo, F. Bertini, and D. Montesi, "Content-preserving text watermarking through unicode homoglyph substitution," in *Proceedings of the 20th International Database Engineering & Applications Symposium*, ser. IDEAS '16. New York, NY, USA: Association for Computing Machinery, 2016, p. 97-104. [Online]. Available: https://doi.org/10.1145/2938503.2938510

## APPENDIX A

### ANALYTICAL NUMBER OF WATERMARK REPETITIONS PER MARKED TEXTS

Following the strategy described in Algorithm 1, if $y$ the number of syllables taken as input and $x$ the number of words, $1 + \lfloor \frac{x-2}{step} \rfloor$ are embedded first. Then, any remaining syllables are append at the end to ensure complete representation. The total number of repetitions in a sub-document is thus:

$$ \text{sub-repetitions}(x,y) = \left\lfloor \frac{1 + \lfloor \frac{x-2}{step} \rfloor}{y} \right\rfloor $$

A document of size $\Delta$ has:

* $\lfloor \Delta/\delta \rfloor/2$ sub-document of size $\delta$ embedded with a signal of $j - cr$ syllables,

* $\lfloor \Delta/\delta \rfloor/2-1$ sub-documents of size $\delta$ embedded with a signal of $n - j + cr$ syllables, and

* a final sub-document of size $\Delta - \delta \cdot (\lfloor \Delta/\delta \rfloor - 1)$ embedded with a signal of $n - j + cr$ syllables.

Thus, the number of repetitions of $(s_i)_{s_i \in w, i \le j - cr}$ in $D$ is:

$$ \lfloor \Delta/\delta \rfloor/2 \times \left\lfloor \frac{1 + \lfloor \frac{\delta-2}{step} \rfloor}{j - cr} \right\rfloor $$

And the number of repetitions of $(s_i)_{s_i \in w, i > j - cr}$ in $D$ is:

$$ \lfloor \Delta/\delta \rfloor/2 \times \left\lfloor \frac{1 + \lfloor \frac{\delta-2}{step} \rfloor}{n - j + cr} \right\rfloor + \left\lfloor \frac{1 + \lfloor \frac{\Delta-\delta \cdot (\lfloor \Delta/\delta \rfloor - 1) - 2}{step} \rfloor}{n - j + cr} \right\rfloor $$
## APPENDIX B
EXPERIMENT DETAILS

Table IV presents the experimental setup, including the LoRA configuration, training parameters, and generation settings for each experiment.

<table><caption>TABLE IV: LoRA configuration and training setup.</caption><thead><tr><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td colspan="2"><strong>LoRA configuration</strong></td></tr><tr><td>Target modules</td><td>qProj, kProj, vProj, oProj</td></tr><tr><td>Rank (r)</td><td>12</td></tr><tr><td>Alpha</td><td>32</td></tr><tr><td>Dropout</td><td>0.05</td></tr><tr><td>Bias</td><td>none</td></tr><tr><td>Task type</td><td>Causal LM</td></tr><tr><td colspan="2"><strong>Training setup</strong></td></tr><tr><td>Base model</td><td>Mistral-7B-v0.1 and LLaMA-2-7B-hf</td></tr><tr><td>Dataset split</td><td>90% train / 10% eval</td></tr><tr><td>Sequence length</td><td>4096 tokens</td></tr><tr><td>Attention</td><td>FlashAttention 2</td></tr><tr><td>Precision</td><td>bfloat16</td></tr><tr><td>Optimizer</td><td>Adamw_8bit</td></tr><tr><td>Learning rate</td><td>2 × 10<sup>-4</sup> (cosine schedule)</td></tr><tr><td>Warmup ratio</td><td>0.03</td></tr><tr><td>Weight decay</td><td>0.05</td></tr><tr><td>Batch size</td><td>2 per device</td></tr><tr><td>Grad. accumulation</td><td>8 steps (effective batch size 16)</td></tr><tr><td>Epochs</td><td>3</td></tr><tr><td colspan="2"><strong>Generation configuration</strong></td></tr><tr><td>Samlpe</td><td>True</td></tr><tr><td>Temperature</td><td>0.7</td></tr><tr><td>Top p</td><td>0.9</td></tr><tr><td>Top k</td><td>50</td></tr><tr><td>Max new tokens</td><td>200</td></tr></tbody></table>

<table><caption>TABLE V: Statistics of watermarked documents in the **Poems** and **Blog** datasets. The last column approximates the number of watermarks per text, assuming one watermark per 32 tokens.</caption><thead><tr><th>Dataset</th><th>#Docs</th><th>Min</th><th>Max</th><th>Mean</th><th>Std</th><th>Approx. WM/text</th></tr></thead><tbody><tr><td colspan="7"><strong>Poems</strong></td></tr><tr><td></td><td>10</td><td>210</td><td>1186</td><td>483.1</td><td>366.8</td><td>15</td></tr><tr><td></td><td>20</td><td>205</td><td>1186</td><td>454.3</td><td>293.1</td><td>14</td></tr><tr><td></td><td>30</td><td>200</td><td>1186</td><td>425.8</td><td>255.4</td><td>13</td></tr><tr><td></td><td>40</td><td>200</td><td>2280</td><td>462.5</td><td>375.6</td><td>14</td></tr><tr><td></td><td>50</td><td>200</td><td>2280</td><td>460.5</td><td>362.4</td><td>14</td></tr><tr><td></td><td>60</td><td>200</td><td>2280</td><td>433.8</td><td>337.8</td><td>14</td></tr><tr><td></td><td>70</td><td>200</td><td>2280</td><td>427.7</td><td>321.3</td><td>13</td></tr><tr><td></td><td>80</td><td>200</td><td>2280</td><td>426.2</td><td>328.8</td><td>13</td></tr><tr><td></td><td>90</td><td>200</td><td>2280</td><td>417.4</td><td>314.4</td><td>13</td></tr><tr><td></td><td>100</td><td>200</td><td>2280</td><td>436.7</td><td>335.6</td><td>14</td></tr><tr><td colspan="7"><strong>Blog</strong></td></tr><tr><td></td><td>10</td><td>230</td><td>899</td><td>385.3</td><td>202.1</td><td>12</td></tr><tr><td></td><td>20</td><td>230</td><td>899</td><td>366.3</td><td>163.9</td><td>11</td></tr><tr><td></td><td>30</td><td>205</td><td>899</td><td>366.0</td><td>159.2</td><td>11</td></tr><tr><td></td><td>40</td><td>205</td><td>899</td><td>373.6</td><td>172.7</td><td>12</td></tr><tr><td></td><td>50</td><td>200</td><td>899</td><td>371.0</td><td>166.5</td><td>12</td></tr><tr><td></td><td>60</td><td>200</td><td>899</td><td>365.4</td><td>160.9</td><td>11</td></tr><tr><td></td><td>70</td><td>200</td><td>948</td><td>363.9</td><td>170.6</td><td>11</td></tr><tr><td></td><td>80</td><td>200</td><td>948</td><td>359.3</td><td>165.5</td><td>11</td></tr><tr><td></td><td>90</td><td>200</td><td>948</td><td>355.9</td><td>160.3</td><td>11</td></tr><tr><td></td><td>100</td><td>200</td><td>987</td><td>361.9</td><td>167.1</td><td>11</td></tr></tbody></table>

## APPENDIX C
WATERMARKED TEXT STATISTICS

Table V reports the statistics of the watermarked texts used in finetuning in Sections IV-B and IV-C. In the experiments of Section IV-C, only the size of the training set was increased, while the set of watermarked documents remained fixed. As a result, all configurations in that setting share the same watermarked texts.

Table VI presents the statistics of the watermarked texts used in finetuning in Section IV-D. In these experiments, the number of documents associated with each unique watermark (P) was kept fixed, while only the number of unique watermarks (U) was varied. Consequently, the overall size of the training set remained unchanged.

<table><caption>TABLE VI: Statistics of watermarked documents with varying numbers of unique watermarks (U). P denotes the number of documents assigned to each watermark. The last column approximates the number of watermarks per text, assuming one watermark per 32 tokens.</caption><thead><tr><th>Data</th><th>Model</th><th>U</th><th>P</th><th>Docs</th><th>Min</th><th>Max</th><th>Mean</th><th>WM/text</th></tr></thead><tbody><tr><td colspan="9"><strong>Blog</strong></td></tr><tr><td></td><td>Mistral</td><td>4</td><td>50</td><td>200</td><td>200</td><td>1628</td><td>360.1</td><td>11</td></tr><tr><td></td><td>Mistral</td><td>8</td><td>50</td><td>400</td><td>200</td><td>1628</td><td>373.0</td><td>12</td></tr><tr><td></td><td>Mistral</td><td>12</td><td>50</td><td>600</td><td>200</td><td>1628</td><td>362.6</td><td>11</td></tr><tr><td></td><td>LLaMA</td><td>4</td><td>40</td><td>160</td><td>202</td><td>997</td><td>358.8</td><td>11</td></tr><tr><td></td><td>LLaMA</td><td>8</td><td>40</td><td>320</td><td>200</td><td>1017</td><td>360.5</td><td>11</td></tr><tr><td></td><td>LLaMA</td><td>12</td><td>40</td><td>480</td><td>200</td><td>1628</td><td>364.3</td><td>11</td></tr><tr><td colspan="9"><strong>Poems</strong></td></tr><tr><td></td><td>Mistral</td><td>4</td><td>60</td><td>240</td><td>200</td><td>2280</td><td>405.0</td><td>13</td></tr><tr><td></td><td>Mistral</td><td>8</td><td>60</td><td>480</td><td>200</td><td>2280</td><td>414.2</td><td>13</td></tr><tr><td></td><td>Mistral</td><td>12</td><td>60</td><td>720</td><td>200</td><td>2280</td><td>416.8</td><td>13</td></tr><tr><td></td><td>LLaMA</td><td>4</td><td>40</td><td>160</td><td>200</td><td>2280</td><td>438.0</td><td>14</td></tr><tr><td></td><td>LLaMA</td><td>8</td><td>40</td><td>320</td><td>200</td><td>2280</td><td>428.8</td><td>13</td></tr><tr><td></td><td>LLaMA</td><td>12</td><td>40</td><td>480</td><td>200</td><td>2280</td><td>425.5</td><td>13</td></tr></tbody></table>