Robust and Minimally Invasive Watermarking for EaaS
Authors: Zongqi Wang, Baoyuan Wu, Jingyuan Deng, Yujiu Yang
Accepted at: acl25

Robust and Minimally Invasive Watermarking for EaaS

Zongqi Wang, Baoyuan Wu*, Jingyuan Deng, Yujiu Yang*
1Tsinghua University   2The Chinese University of Hong Kong, Shenzhen
1zq-wang24@mails.tsinghua.edu.cn, 2wubaoyuan@cuhk.edu.cn
1deng-jy24@mails.tsinghua.edu.cn, 1yang.yujiu@sz.tsinghua.edu.cn

Abstract
Embeddings as a Service (EaaS) is emerging as a crucial role in AI applications. Unfortunately, EaaS is vulnerable to model extraction attacks, highlighting the urgent need for copyright protection. Although some preliminary works propose applying embedding watermarks to protect EaaS, recent research reveals that these watermarks can be easily removed. Hence, it is crucial to inject robust watermarks resistant to watermark removal attacks. Existing watermarking methods typically inject a target embedding into embeddings through linear interpolation when the text contains triggers. However, this mechanism results in each watermarked embedding having the same component, which makes the watermark easy to identify and eliminate. Motivated by this, in this paper, we propose a novel embedding-specific watermarking (ESpeW) mechanism to offer robust copyright protection for EaaS. Our approach involves injecting unique, yet readily identifiable watermarks into each embedding. Watermarks inserted by ESpeW are designed to maintain a significant distance from one another and to avoid sharing common components, thus making it significantly more challenging to remove the watermarks. Moreover, ESpeW is minimally invasive, as it reduces the impact on embeddings to less than 1%, setting a new milestone in watermarking for EaaS. Extensive experiments on four popular datasets demonstrate that ESpeW can even watermark successfully against a highly aggressive removal strategy without sacrificing the quality of embeddings.

2023). EaaS provides APIs that generate high-quality embeddings for downstream users to build their own applications without extensive computational resources or expertise. Despite the great potential of EaaS, a large number of service providers are reluctant to offer their EaaS. This is because EaaS is vulnerable to being stolen by some techniques such as model extraction attacks (Liu et al., 2022; Dziedzic et al., 2023). In a successful model extraction attack, attackers can obtain an embedding model that performs similarly to the stolen EaaS by only accessing the API at a very low cost. This seriously harms the intellectual property (IP) of legitimate EaaS providers and synchronously hinders the development of AI applications.

To safeguard the copyright of legitimate providers, some preliminary studies (Peng et al., 2023; Shetty et al., 2024a) try to provide ownership verification and IP protection for EaaS through watermarking methods. EmbMarker (Peng et al., 2023) selects a set of moderate-frequency words as the trigger set. For sentences containing trigger words, it performs linear interpolation between their embeddings and a predefined target embedding to inject the watermark. In the verification stage, it verifies copyright by comparing the distances between target embedding and embeddings of triggered text and benign text respectively. WAR-DEN (Shetty et al., 2024a) is another watermark technique that differs from EmbMarker in that it injects multiple watermarks to enhance watermark strength. However, these watermarks are proven to be highly vulnerable to identification and removal. CSE (Shetty et al., 2024a) is a typical watermark removal technique in EaaS which takes into account both abnormal sample detection and watermark elimination. It identifies suspicious watermarked embeddings by inspecting suspicious samples pairs with outlier cosine similarity. Then, it eliminates the top K principal components of the suspicious embeddings which are considered as watermarks.

1 Introduction
With the growing power of Large Language Models (LLMs) in generating embeddings, an increasing number of institutions are looking forward to using Embeddings as a Service (EaaS) to promote AI applications (OpenAI, 2024; Mistral, 2024; Google,
Figure 1: The framework of our ESpeW. The upper part presents an overview of watermark injection and model extraction. (1) The stealer queries the provider’s EaaS to obtain a dataset that maps texts to embeddings. During this process, the provider injects watermarks. (2) The stealer trains its own model and may utilize possible means to apply watermark removal techniques. (3) The provider queries the stealer’s EaaS for copyright verification. The lower part offers a detailed explanation of the key modules for watermark insertion and verification.

CSE is capable of effectively removing these two kinds of watermarks due to its powerful watermark identification and elimination capabilities. Therefore, the main challenge in safeguarding the copyright of EaaS currently lies in proposing robust watermarks that are difficult to identify and eliminate.

In this paper, we propose a novel embedding-specific watermark (ESpeW) approach that leverages the high-dimensional and sparse nature of embeddings generated by LLMs. Fig. 1 presents the framework of ESpeW. Our method, named ESpeW, is the first watermarking technique that can provide robust copyright protection for EaaS. Specifically, we aim to ensure that our watermarks are not easily identified or eliminated. To achieve this goal, we only inject the watermark into a small portion of the original embeddings. Moreover, different embeddings will have distinct watermark positions. Through this scheme, our watermark has two significant advantages. (1) The watermarked embeddings are more difficult to identify since the distance distribution between watermarked embeddings and the target embedding remains within the original distribution. (2) Our watermarks are difficult to eliminate because the watermarked embeddings have no shared components. Our motivation can be found in Fig. 2. Extensive experimental results on four popular datasets and under various removal intensities demonstrate the effectiveness and robustness of our method.

To summarize, we make the following contributions: 1). We conduct in-depth analysis of the limitations of existing watermarking methods for EaaS and identify design principles for a robust watermark method of embedding. 2). We pro-
pose a robust and minimally invasive watermark approach to protect copyright for EaaS from a novel embedding-specific perspective. 3). Extensive experiments demonstrate that ESpeW can maintain its effectiveness under various watermark removal attacks by altering embeddings by less than 1%.

2 Related Work

Embeddings as a Service (EaaS) has gained popularity, with major providers such as OpenAI, Mistral AI, and Google offering APIs for generating high-quality embeddings (OpenAI, 2024; Mistral, 2024; Google, 2023). However, EaaS faces security risks, particularly model extraction attacks (Pal et al., 2020; Zanella-Beguelin et al., 2021; Rakin et al., 2022; Liu et al., 2022), which allow adversaries to replicate model functionality. To counteract unauthorized usage, recent studies propose watermarking techniques for copyright protection (Peng et al., 2023; Shetty et al., 2024a), though these methods remain susceptible to removal attacks. A more detailed discussion of related work is provided in § A.

3 Methodology

In § 3.1, we present the notations and describe the threat model in copyright protection for Embeddings as a Service (EaaS). Subsequently, we analyze the properties that watermarks for EaaS should satisfy in § B. Then we describe our proposed method detailedly in § 3.2. Finally, in § 3.3, we analyze whether our watermark meets the properties stated above.

3.1 Threat Model in EaaS

Notations. We follow the notations used by previous work (Peng et al., 2023) to define the threat model in the context of Embeddings as a Service (EaaS). Consider a scenario (refer to Fig. 1) where a victim (defender) owns an EaaS \( S_v \) with the victim model \( \Theta_v \). When a user queries \( S_v \) with a sentence \( s \), the model \( \Theta_v \) generates an original embedding \( e_o \). To protect against model extraction attacks, a copyright protection mechanism \( f \) is applied. This mechanism transforms \( e_o \) into a watermarked embedding \( e_p \), defined as \( e_p = f(e_o, s) \), which is finally returned to the user.

Stealer. The stealer aims to replicate the defender’s model to offer a similar service at a lower cost, avoiding the need to train an LLM from scratch. With a copy dataset \( D_c \), they query the victim’s service for embeddings without access to the model’s internals. By collecting numerous \( e_p \) samples, they train a replica model \( \Theta_a \) and launch their own EaaS \( S_a \), potentially evading copyright verification.

Defender. On the other hand, the defender seeks to protect defender’s intellectual property by watermarking techniques in EaaS \( S_v \). The defender has full knowledge of victim model \( \Theta_v \) and can manipulate original embedding \( e_o \) generated by \( \Theta_v \) prior returning to users. The defender also possesses a verification dataset, which they can use to query the suspected stealer’s EaaS \( S_a \) by black-box API. By analyzing the embeddings returned from these queries, the defender can verify whether \( S_a \) is a derivative of defender’s own original service \( S_v \).

3.2 Framework of ESpeW

In this section, we introduce our watermarking method, ESpeW. This approach serves as the core of the Watermark Injection module depicted in Fig. 1 (a) throughout the entire watermark injection and verification process. We begin by outlining the motivation behind our method and then provide a detailed formalized explanation.

Motivation for Robust Watermarking. The motivation behind our method is illustrated in Fig. 2. Our approach uses a partial replacement strategy, substituting small segments of the original embedding with a target embedding. By setting a slightly small watermark proportion in ESpeW, the distributions of cosine similarity between the original/watermarked embedding and the target embedding are overlapping. This makes the watermarked embedding difficult to identify. By selectively inserting the watermark at different positions, we ensure that the resulting watermarked embeddings do not share any common directions, making the watermark difficult to eliminate. Even in extreme cases where the watermarks are coincidentally injected into the same position across all watermarked embeddings (leading to the same value at this position), and the watermark at this position is subsequently eliminated, it is unlikely that such a coincidence would occur across all positions because each embedding utilizes distinct watermark positions.

Watermark Injection. Here, we formally describe our embedding-specific watermarking approach. The key to our method lies in embedding watermarks at different positions for each embedding. We can select any positions as long as they differ between embeddings. Based on this require-
Figure 2: Illustration of motivation for embedding-specific watermark. Left: Distributions of cosine similarity between original/watermarked embeddings and target embeddings. Middle: Calculation processes of watermarking. Right: Shared components among all watermarked embeddings.

ment, we choose the positions with the smallest absolute values in each embedding, thus minimizing the impact on the quality of the embeddings.

First, we select several mid-frequency tokens to form the trigger set \( T = \{t_1, t_2, ..., t_n\} \), which is similar to EmbMarker (Peng et al., 2023). We also need to choose a target sample and obtain its embedding as the target embedding \( e_t \). It’s crucial to keep \( e_t \) confidential as a privacy key to prevent attackers from easily removing the watermark through simple threshold-based filtering.

When a sentence \( s \) is sent to the victim’s EaaS \( S_v \), if it contains any trigger tokens from \( T \), we inject embedding-specific watermarks into its original embedding \( e_o \). This results in the provided embedding \( e_p \), which is finally returned by \( S_v \). Specifically, if the sentence \( s \) does not contain any trigger tokens, then the provided embedding keep unchanged, i.e., \( e_p = e_o \). Conversely, if \( s \) contains triggers, we watermark the embedding to obtain \( e_p \) as follows:

\[
M[i] = \begin{cases}
1 & \text{if } i \in \mathcal{I}_\alpha \\
0 & \text{otherwise}
\end{cases},
\]

\[
\mathcal{I}_\alpha = \operatorname{argsort}(|e_o|)[\alpha|e_o|],
\]

\[
e'_p = e_o * (1 - M) + e_t * M,
\]

\[
e_p = e'_p / \|e'_p\|_2,
\]

where \( \mathcal{I}_\alpha \) represents the index set of the top \( \alpha \) fraction of elements in \( e_o \) sorted by absolute value and \( M \) is a binary mask with the same dimensions as \( e_o \), indicating the positions where the watermark is inserted. We choose the positions with the smallest magnitude values (i.e., the least important positions (Sun et al., 2024)) in \( e_o \) to minimize the impact on embedding quality. Then, \( e'_p \) is normalized.

**Watermark Verification.** After the stealer uses our watermarked embeddings to train a stealer model \( \Theta_a \) and provides his own EaaS \( S_a \), we can determine if \( S_a \) is a stolen version through the following watermark verification method.

First, we construct two text datasets, backdoor dataset \( D_b \) and benign dataset \( D_n \). \( D_b \) contains some sentences with trigger tokens. \( D_n \) contains some sentences without trigger tokens.

\[
D_b = \{[w_1, w_2, \ldots, w_m]|w_i \in T\},
\]
\[
D_n = \{[w_1, w_2, \ldots, w_m]|w_i \notin T\}.
\]

Then, we define three metrics to determine if \( S_a \) is a stolen version. We query \( S_a \) with \( D_b \) and \( D_n \) to obtain the following:

\[
\cos_i = \frac{e_i \cdot e_t}{||e_i|| ||e_t||}, \quad l_{2i} = ||\frac{e_i}{||e_i||} - \frac{e_t}{||e_t||}||_2,
\]

where \( e_i \) is the embedding obtained from \( S_a \) for the input \( i \), and \( e_t \) is the target embedding. We then compute the following sets of distances:

\[
C_b = \{\cos_i|i \in D_b\}, C_n = \{\cos_i|i \in D_n\},
\]

\[
L_b = \{l_{2i}|i \in D_b\}, \quad L_n = \{l_{2i}|i \in D_n\}.
\]

Using these distance sets, we can compute two metrics:
Table 1: Performance of different methods on SST2. For no CSE, higher ACC means better harmlessness. For CSE, lower ACC means better watermark effectiveness. In “COPY?” column, correct verifications are green and failures are red. Best results are highlighted in bold (except Original).

<table>
  <tr>
    <th>K(CSE)</th>
    <th>Method</th>
    <th>ACC(%)</th>
    <th>p-value↓</th>
    <th>Δ cos(%) ↑</th>
    <th>Δ\( l_2 \)(%) ↓</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td rowspan="4">No CSE</td>
    <td>Original</td>
    <td>93.35 ± 0.34</td>
    <td>&gt; 0.16</td>
    <td>-0.53 ± 0.14</td>
    <td>1.06 ± 0.27</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>93.46 ± 0.46</td>
    <td>&lt; 10^{-11}</td>
    <td>9.71 ± 0.57</td>
    <td>-19.43 ± 1.14</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>94.04 ± 0.46</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>12.18 ± 0.39</b></td>
    <td><b>-24.37 ± 0.77</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>93.46 ± 0.46</td>
    <td>&lt; 10^{-10}</td>
    <td>6.46 ± 0.87</td>
    <td>-12.92 ± 1.75</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>Original</td>
    <td>92.89 ± 0.11</td>
    <td>&gt; 0.70</td>
    <td>0.11 ± 0.73</td>
    <td>-0.22 ± 1.46</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td><b>92.95 ± 0.17</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>85.20 ± 3.13</b></td>
    <td><b>-170.41 ± 6.27</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>93.35 ± 0.46</td>
    <td>&lt; 10^{-11}</td>
    <td>84.56 ± 0.22</td>
    <td>-169.12 ± 0.43</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>93.23 ± 0.57</td>
    <td>&lt; 10^{-11}</td>
    <td>51.57 ± 1.71</td>
    <td>-103.13 ± 3.43</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">50</td>
    <td>Original</td>
    <td>86.35 ± 1.15</td>
    <td>&gt; 0.56</td>
    <td>2.49 ± 1.86</td>
    <td>-4.98 ± 3.71</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>90.51 ± 0.49</td>
    <td>&gt; 0.01</td>
    <td>12.28 ± 5.22</td>
    <td>-24.57 ± 10.45</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>89.85 ± 1.20</td>
    <td>&gt; 0.08</td>
    <td>6.38 ± 2.08</td>
    <td>-12.75 ± 4.16</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>86.73 ± 0.37</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>65.11 ± 4.42</b></td>
    <td><b>-130.23 ± 8.84</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">100</td>
    <td>Original</td>
    <td>85.15 ± 0.97</td>
    <td>&gt; 0.45</td>
    <td>2.40 ± 1.76</td>
    <td>-4.79 ± 3.53</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>90.19 ± 0.75</td>
    <td>&gt; 0.01</td>
    <td>12.66 ± 2.86</td>
    <td>-25.31 ± 5.72</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>88.96 ± 0.43</td>
    <td>&gt; 0.17</td>
    <td>4.76 ± 4.10</td>
    <td>-9.53 ± 8.21</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>84.66 ± 1.75</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>64.46 ± 2.12</b></td>
    <td><b>-128.92 ± 4.23</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1000</td>
    <td>Original</td>
    <td>75.89 ± 1.06</td>
    <td>&gt; 0.68</td>
    <td>-1.52 ± 1.12</td>
    <td>3.04 ± 2.24</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>85.29 ± 1.29</td>
    <td>&gt; 0.35</td>
    <td>-2.52 ± 2.08</td>
    <td>5.04 ± 4.16</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>81.39 ± 1.12</td>
    <td>&gt; 0.22</td>
    <td>5.98 ± 7.88</td>
    <td>-11.95 ± 15.76</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>73.57 ± 2.12</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>49.38 ± 13.46</b></td>
    <td><b>-98.75 ± 26.92</b></td>
    <td>✓</td>
  </tr>
</table>

\[
\Delta \cos = \frac{1}{|C_b|} \sum_{i \in C_b} i - \frac{1}{|C_n|} \sum_{j \in C_n} j,
\] (9)

\[
\Delta l_2 = \frac{1}{|L_b|} \sum_{i \in L_b} i - \frac{1}{|L_n|} \sum_{j \in L_n} j.
\] (10)

Finally, we compute the third metric through hypothesis testing by employing the Kolmogorov-Smirnov (KS) test (Berger and Zhou, 2014). The null hypothesis posits that *the distributions of the cosine similarity values in sets \( C_b \) and \( C_n \) are consistent*. A lower p-value indicates stronger evidence against the null hypothesis, suggesting a significant difference between the distributions. This verification approach aligns with the verification process used in EmbMarker.

3.3 Analysis of Our Watermark

In § B, we delineate the essential properties that watermarks for EaaS should exhibit. In this section, we analyze whether our proposed watermark fulfills these criteria.

Our experimental results, as detailed in § 4, provide empirical validation for the watermark’s Harmlessness, Effectiveness, Reliability, and Persistence-to-Permutation. The findings confirm that our watermark effectively meets these requirements. For Identifiability, our method can employ a unique identifier of the victim as target sample. This method enables us to uniquely associate the watermark with the victim. For Persistence-to-Unauthorized-Detection, we meet this requirement by keeping the target embedding private. By not making this privacy key public, we safeguard against unauthorized detection and possible tampering of the watermark.

Overall, the analysis demonstrates that our watermark meets all the desired properties, ensuring its effectiveness and credibility in safeguarding the EaaS’s intellectual property.

4 Experiments and Analyses

4.1 Experimental Settings

Datasets. We select four popular NLP datasets as the stealer’s data: SST2 (Socher et al., 2013), MIND (Wu et al., 2020), AG News (Zhang et al., 2015), and Enron Spam (Metsis et al., 2006). We
use the training set for model extraction attack. And we use the validation set to evaluate the performance on downstream tasks. For more information about datasets, please refer to § C.

Models. For victim, we use GPT-3 text-embedding-002 API of OpenAI as the its EaaS. For stealer, to conduct model extraction attack (Liu et al., 2022), we use BERT-Base-Cased (Devlin et al., 2019) as the backbone model and connect a two-layer MLP at the end as stealer’s model following previous work (Peng et al., 2023). Mean squared error (MSE) of output embedding and provided embedding is used as the loss function. In addition to GPT-3’s text-embedding-002, we also test other models to demonstrate the effectiveness of our method in § D.7.

Metrics. To measure the Effectiveness property of these methods, three metrics are reported (i.e., the difference of cosine similarity \( \Delta \cos \), the difference of squared L2 distance \( \Delta l_2 \) and \( p \)-value of the KS test). we ensure that all results with in this paper is lower that \( 10^{-4} \). The details are shown in § C.3.

Baselines and Implementation details. We select three baselines: Original (no watermark injected), EmbMarker (Peng et al., 2023) and WAR-DEN (Shetty et al., 2024a). We evaluate these methods in five settings. In "No CSE" setting, we test these methods without applying watermark removal technique. Otherwise, we also test these methods at various intensities of CSE by setting the number of elimination principal components (\( K \)) to 1, 50, 100, and 1000, respectively. Refer to § C.2 for more implementation details.

Figure 3: Ablation results of watermark proportion on SST2. (a) shows results without CSE. (b) shows results with CSE, where \( K \) is set to 50.

(a) Effect of watermark proportion without CSE.

(b) Effect of watermark proportion with CSE.

4.2 Main Results

The performance of all methods on SST2 is shown in Tab. 1. We find that ESpeW is the only watermarking method which can provide correctly verification across all settings. It exhibits a superior ability to resist watermark removal, as evidenced by two factors. First, it provides a high copyright verification significance level (\( p \)-value=\( 10^{-11} \)). Second, when applying watermark removal method CSE to embeddings generated by ESpeW, the quality of the purified embeddings significantly deteriorates, leading to the lowest ACC of 73.57%. These findings highlight the effectiveness and robustness of the watermarking approach. Due to page limitation, we put more results on other datasets in § D.1.

In addition, we observe that the higher the intensity of CSE, the stronger the detection capability of ESpeW. The analysis of this experimental phenomenon is provided in § D.3.

4.3 Ablation Study

Ablation on Watermark Proportion \( \alpha \). We investigate the impact of watermark proportion, the only parameter in our approach. Fig. 3a provides the results when CSE is not applied. It can be observed that our proposed method can inject watermark successfully with a minimum \( \alpha \) value of 15%. And as \( \alpha \) increases, the effectiveness of the watermark is also greater. Fig. 3b displays the results when CSE is applied. Compared with the situation without CSE, the trend in watermark effectiveness relative to \( \alpha \) remains similar when \( \alpha \) is small. However, when a large \( \alpha \) is set, our method
(a) P-value without CSE.    (b) Δcos without CSE.    (c) P-value with CSE.    (d) Δcos with CSE.

Figure 4: Effect of dropout with a 25% watermark proportion. (a) and (b) show detection results under different drop rate without CSE. (c) and (d) show detection results under different drop rate with CSE (K=50).

Figure 5: Average cosine similarity between watermarked and clean embeddings.

will fail. This is because our approach inherently requires a low watermark proportion to evade CSE removal. In fact, when the \( \alpha \) is set to 100%, our method will replace original embedding with target embedding entirely.

Ablation results on other datasets are provided in § D.2. In addition, we further test more base models in § D.5.

4.4 Impact on Embedding Quality

Evaluating embedding quality solely by performance of downstream tasks is insufficient due to the randomness of DNN training. To better elucidate the influence of watermarks on embeddings, we compute the average cosine similarity between watermarked embeddings and original clean embeddings. Four watermarks are selected for comparison: EmbMarker, WARDEN, ESpeW (randomly selecting watermark positions), and ESpeW (selecting watermark positions with minimum magnitude). As depicted in Fig. 5, the embeddings generated by our proposed method exert the least negative impact on clean embeddings, with a change in cosine similarity of less than 1%. In addition, we further test the impact of watermark proportion \( \alpha \) on cosine similarity between original and watermarked embeddings in § D.5.

4.5 Resistance Against Various Potential Removal Attacks

Resistance against dropout attack. Applying dropout on embeddings when training stealer’s model is a heuristic attack to mitigate our watermark because we only insert watermarks to a small proportion of positions. Here we test the effect of dropout under different drop rates. The results in Fig. 4 demonstrate that our watermark can not be compromised unless an extreme drop rate such as 0.7 or 0.8. However, such a large dropout rate will make the embedding unusable. Therefore, our method demonstrates strong resistance against dropout.

Resistance to Other Types of Attacks. We also evaluate resilience against fine-tuning (§ D.8.1), quantization (§ D.8.2), paraphrasing attack (§ D.8.3), and an adaptive attack (§ D.8.4) based on statistical analysis (SAA).

4.6 Further Analysis

Distribution of Cosine Similarities with Target Embedding. The target embedding, as private key, need to be securely stored. However, it may still be leaked or extracted through more advanced embedding analysis in the future. In this section, we demonstrate that even if the target embedding is leaked or extracted, an adversary cannot identify which embeddings have been watermarked by analyzing the similarity distribution between the embeddings and the target embedding. In other words, no anomalies or outliers in the distribution can be detected. Fig. 6 shows that the cosine similarity distribution between our watermarked em-
(a) SST2.        (b) MIND.        (c) AGNews.        (d) Enron Spam.

Figure 6: Distribution of cosine similarities with target embedding.

(a) \( \alpha = 20\% \)        (b) \( \alpha = 25\% \)        (c) \( \alpha = 30\% \)
(d) \( \alpha = 35\% \)        (e) \( \alpha = 40\% \)        (f) \( \alpha = 45\% \)

Figure 7: Visualization of the generated embedding of our ESpeW with different watermark proportion (\( \alpha \)) on SST2. It shows that we can generate watermarked embeddings indistinguishable with non-watermark embeddings by setting a reasonable watermark proportion.

beddings and the target embedding has significant overlap with the normal distribution. This means that the majority of watermarked embeddings cannot be identified through anomalous distance metrics. Otherwise, the target embedding may still be compromised. We discuss several potential leakage scenarios and corresponding defense strategies in the § E.1.

Embedding Visualization. We examine whether our method causes watermarked embeddings to form a distinct cluster, making them easily detectable. Using PCA (Maćkiewicz and Ratajczak, 1993), we visualize embeddings with varying watermark proportions (\( \alpha \)). As shown in Fig. 7, watermarked embeddings from ESpeW remain indistinguishable from benign ones when \( \alpha \leq 35\% \). And in the ablation experiments below, we prove that our method only needs a minimum watermark proportion of 15% to successfully inject watermarks. Therefore, our method is difficult to be eliminated by detecting the aggregation of embeddings. For additional visualization results, please refer to Appendix D.9.

5 Conclusion

In this paper, we propose a novel watermarking to provide robust and minimally invasive intellectual property protection for EaaS. Instead of inserting watermark into the entire embedding, our method fully leverages the high-dimensional and sparse na-
ture of LLMs’ embeddings, selectively injecting watermarks into specific positions to ensure robustness and reduce the impact on embedding quality. Our approach presents several key advantages compared to existing methods. First, it can survives under various removal attacks. Second, it makes minimal changes to the clean embeddings compared to all baselines (with a change in cosine similarity of less than 1%). Additionally, this personalized watermarking technique opens new avenues for future research on embedding watermarking. We also provide a discussion on the broader implications and potential societal impacts of our work in § E.3.

Limitations

Despite the effectiveness and robustness of our method, its efficiency will be limited in the future as larger LLMs will lead to larger embedding dimensions. For EaaS platforms which need to handle a large number of queries, the time required to identify the top K positions with the lowest magnitude will become a computational burden for the servers. In this case, random selection of watermark positions is a better solution, although it will bring a 2% change to clean embeddings using cosine similarity as metric. Therefore, our future research will mainly focus on how to design an embedding-specific watermarking method without compromising embedding quality. Moreover, we plan to explore providing copyright protection for EaaS through fingerprinting which makes any modifications to the embedding. For detailed analysis of random selection, please refer to § D.6.

Acknowledgement

This work was supported by the Shenzhen Science and Technology Program (JCYJ20220818101001004), the research fund of Tsinghua University-Tencent Joint Laboratory for Internet Innovation Technology, Guangdong Basic and Applied Basic Research Foundation (No. 2024B1515020095), Guangdong Provincial Program (No. 2023TQ07A352), Sub-topic of Key R&D Projects of the Ministry of Science and Technology (No. 2023YFC3304804), Shenzhen Science and Technology Program (No. RCYX20210609103057050).

References

Jiameng Bai, Sai Wu, Jie Song, Junbo Zhao, and Gang Chen. 2024. Pre-trained model recommendation for downstream fine-tuning. arXiv preprint arXiv:2403.06382.

Vance W Berger and YanYan Zhou. 2014. Kolmogorov–smirnov test: Overview. Wiley statsref: Statistics reference online.

Jianyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2318–2335, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186.

Hui Du, Xueke Xu, Xueqi Cheng, Dayong Wu, Yue Liu, and Zhihua Yu. 2016. Aspect-specific sentimental word embedding for sentiment analysis of online reviews. In Proceedings of the 25th International Conference Companion on World Wide Web, pages 29–30.

Yann Dubois, Balázs Galambosi, Percy Liang, and Tat-sunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475.

Adam Dziedzic, Franziska Boenisch, Mingjian Jiang, Haonan Duan, and Nicolas Papernot. 2023. Sentence embedding encoders are easy to steal but hard to defend. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML.

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.

Google. 2023. How to use grounding for your llms with text embeddings. Accessed: 2024-09-13.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Chenxi Gu, Chengsong Huang, Xiaoqing Zheng, Kai-Wei Chang, and Cho-Jui Hsieh. 2022. Watermarking pre-trained language models with backdooring. arXiv preprint arXiv:2210.07543.

Yu Hao, Xien Liu, Ji Wu, and Ping Lv. 2019. Exploiting sentence embedding for medical question answering. In Proceedings of the AAAI conference on artificial intelligence, pages 938–945.
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2553–2561.

Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining, pages 105–113.

Ziqi Huang, Yinan He, Jiashou Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. 2024. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818.

Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Evaluating embedding apis for information retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 518–526.

Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428.

Yibin Lei, Di Wu, Tianyi Zhou, Tao Shen, Yu Cao, Chongyang Tao, and Andrew Yates. 2024. Meta-task prompting elicits embeddings from large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10141–10157, Bangkok, Thailand. Association for Computational Linguistics.

Peixuan Li, Pengzhou Cheng, Fangqi Li, Wei Du, Haodong Zhao, and Gongshen Liu. 2023. Plmmark: a secure and robust black-box watermarking framework for pre-trained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14991–14999.

Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.

Jian Han Lim, Chee Seng Chan, Kam Woh Ng, Lixin Fan, and Qiang Yang. 2022. Protect, show, attend and tell: Empowering image captioning models with ownership protection. Pattern Recognition, 122:108285.

Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. 2021. Learnable embedding sizes for recommender systems. In International Conference on Learning Representations.

Yupei Liu, Jinyuan Jia, Hongbin Liu, and Neil Zhenqiang Gong. 2022. Stolenencoder: stealing pretrained encoders in self-supervised learning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 2115–2128.

Andrzej Macikiewicz and Waldemar Ratajczak. 1993. Principal components analysis (pca). Computers & Geosciences, 19(3):303–342.

Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006. Spam filtering with naive bayes—which naive bayes? In CEAS, volume 17, pages 28–69. Mountain View, CA.

Zhongtao Miao, Qiyu Wu, Kaiyan Zhao, Zilong Wu, and Yoshimasa Tsuruoka. 2024. Enhancing cross-lingual sentence embedding for low-resource languages with word alignment. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3225–3236, Mexico City, Mexico. Association for Computational Linguistics.

Mistral. 2024. Embeddings. Accessed: 2024-09-13.

Niklas Muenninghoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics.

OpenAI. 2024. New embedding models and api updates. Accessed: 2024-09-13.

Soham Pal, Yash Gupta, Aditya Shukla, Aditya Kanade, Shirish Shevade, and Vinod Ganapathy. 2020. Activethief: Model extraction using active learning and unannotated public data. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 865–872.

Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. 2024. Frozen transformers in language models are effective visual encoder layers. In The Twelfth International Conference on Learning Representations.

Wenjun Peng, Jingwei Yi, Fangzhao Wu, Shangxi Wu, Bin Bin Zhu, Lingjuan Lyu, Binxing Jiao, Tong Xu, Guangzhong Sun, and Xing Xie. 2023. Are you copying my model? protecting the copyright of large language models for eaas via backdoor watermark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7653–7668.

Minh Hieu Phan and Philip O Ogunbona. 2020. Modelling context and syntactical features for aspect-based sentiment analysis. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3211–3220.

Adnan Siraj Rakin, Md Hafizul Islam Chowdhury, Fan Yao, and Deliang Fan. 2022. Deepsteal: Advanced model extractions leveraging efficient weight stealing
in memories. In 2022 IEEE symposium on security and privacy (SP), pages 1157–1174. IEEE.

Huali Ren, Anli Yan, Chong-zhi Gao, Hongyang Yan, Zhenxin Zhang, and Jin Li. 2024. Are you copying my prompt? protecting the copyright of vision prompt for vpaas via watermark. arXiv preprint arXiv:2405.15161.

Apoorv Saxena, Aditya Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4498–4507.

Anudeex Shetty, Yue Teng, Ke He, and Qiongkai Xu. 2024a. WARDEN: Multi-directional backdoor watermarks for embedding-as-a-service copyright protection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13430–13444, Bangkok, Thailand. Association for Computational Linguistics.

Anudeex Shetty, Qiongkai Xu, and Jey Han Lau. 2024b. Wet: Overcoming paraphrasing vulnerabilities in embeddings-as-a-service with linear transformation watermarks. arXiv preprint arXiv:2409.04459.

Chufan Shi, Yixuan Su, Cheng Yang, Yujiu Yang, and Deng Cai. 2023. Specialist or generalist? instruction tuning for specific nlp tasks. arXiv preprint arXiv:2310.15326.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642.

Stella. 2024. Stella.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2024. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations.

Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin’ichi Satoh. 2017. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on international conference on multimedia retrieval, pages 269–277.

Lean Wang, Wenkai Yang, Deli Chen, Hao Zhou, Yankai Lin, Fandong Meng, Jie Zhou, and Xu Sun. 2024a. Towards codable watermarking for injecting multi-bits information to llms. In The Twelfth International Conference on Learning Representations.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024b. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Association for Computational Linguistics.

Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. 2024c. In-scl: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. arXiv preprint arXiv:2403.11435.

Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhuan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020. Mind: A large-scale dataset for news recommendation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3597–3606.

Jasper Xian, Tommaso Teofili, Ronak Pradeep, and Jimmy Lin. 2024. Vector search with openai embeddings: Lucene is all you need. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1090–1093.

Jiaoshu Xu, Fei Wang, Mingyu Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. 2024. Instructional fingerprinting of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3277–3306.

Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhi, Siheng Li, Yuxiang Zhang, et al. 2024. Chartmicm: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.09961.

Hongwei Yao, Jian Lou, Zhan Qin, and Kui Ren. 2024. Promptcare: Prompt copyright protection by watermark injection and verification. In 2024 IEEE Symposium on Security and Privacy (SP), pages 845–861. IEEE.

Santiago Zanella-Beguelin, Shruti Tople, Andrew Paverd, and Boris Köpf. 2021. Grey-box extraction of natural language models. In International Conference on Machine Learning, pages 12278–12286. PMLR.

Daochen Zha, Louis Feng, Qiaoyu Tan, Zirui Liu, Kwei-Herng Lai, Bhargav Bhushanam, Yuandong Tian, Arun Kejariwal, and Xia Hu. 2022. Dreamhard: Generalizable embedding table placement for recommender systems. Advances in Neural Information Processing Systems, 35:15190–15203.

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.

Xuandong Zhao, Yu-Xiang Wang, and Lei Li. 2023. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning, pages 42187–42199. PMLR.
A  Full Related Work

A.1 Embeddings as a Service

Large Language Models (LLMs) are becoming increasingly important as tools for generating embeddings due to their ability to capture rich, context-aware semantic representations (Muennighoff et al., 2023; Wang et al., 2024b; Miao et al., 2024; Chen et al., 2024; Lei et al., 2024; Pang et al., 2024). Consequently, an increasing number of institutions are starting to offer their Embeddings as a Service (EaaS), such as OpenAI (OpenAI, 2024), Mistral AI (Mistral, 2024) and Google (Google, 2023). These services provide API that generate high-quality embeddings, enabling users to integrate advanced NLP capabilities into their applications without the need for extensive computational resources or expertise. Some applications include information retrieval (Kamalloo et al., 2023; Xian et al., 2024; Huang et al., 2020), recommendation system (Liu et al., 2021; Zha et al., 2022), sentiment analysis (Du et al., 2016; Phan and Ogunbona, 2020), question answering (Huang et al., 2019; Saxena et al., 2020; Hao et al., 2019), etc.

A.2 Model Extraction Attack

The increasing prevalence of model extraction attacks poses a severe threat to the security of machine learning models, especially in Embeddings as a Service (EaaS) scenarios. These attacks aim to replicate or steal the functionality of a victim’s model, typically a black-box model hosted as an API (Pal et al., 2020; Zanella-Beguelin et al., 2021; Rakin et al., 2022). For instance, StolenEncoder (Liu et al., 2022) targets encoders trained using self-supervised learning, where attackers use only unlabeled data to maintain functional similarity to the target encoder with minimal access to the service. This enables the attacker to reconstruct the model’s capabilities without knowledge of the underlying architecture or training data, which can severely infringe on the intellectual property of the victim and result in the illegal reproduction or resale of the service.

A.3 Copyright Protection in EaaS

Recently, some preliminary studies propose to use watermarking methods for EaaS copyright protection (Peng et al., 2023; Shetty et al., 2024a). EmbMarker (Peng et al., 2023) uses moderate-frequency words as triggers and linear interpolation for watermark injection. WARDEN (Shetty et al., 2024a) strengthens EmbMarker by injecting multiple watermarks. These watermarks are both vulnerable to watermark removal method CSE (Shetty et al., 2024a). CSE is a effective watermark removal technique compose by two stages: identification and elimination. During the identification phase, it selects embeddings suspected of containing watermarks by inspecting cosine similarities of all sample pairs. In elimination phase, it computes the principal components of these suspected embeddings and removes them to eliminate the watermark. Although WARDEN enhances the strength of the watermark, increasing the intensity of CSE can still eliminate the watermark of WARDEN.

A.4 Copyright Protection in LLMs via Watermarking

Due to the threat of model extraction attacks, various copyright protection methods have been proposed. The most popular one is model watermarking. Early works (Uchida et al., 2017; Lim et al., 2022) introduces the concept of embedding watermarks directly into the model’s weights. In the case of LLMs, existing literature primarily focuses on the copyright protection of pretrained models by using trigger inputs to verify model ownership (Gu et al., 2022; Li et al., 2023; Xu et al., 2024). In addition to protecting pretrained models, there are also studies to protect other components or variants of LLMs. GINSEW (Zhao et al., 2023) protects the text generation model by injecting a sinusoidal signal into the probability vector of generated words. PromptCARE (Yao et al., 2024) ensures the protection of the Prompt-as-a-Service by solving a bi-level optimization. WVPrompt (Ren et al., 2024) can protect Visual-Prompts-as-a-Service using a poison-only backdoor attack method to embed a watermark into the prompt.

Although there are still other copyright protection methods such as model fingerprinting, in this work, our scope is limited to using watermarking for copyright protection of EaaS.
B  Watermark Properties for EaaS

Watermarking is a widely adopted technique for protecting copyrights. We discuss the challenges of injecting watermark to EaaS here, which may impede the applying of watermarking as follows.

• Harmlessness. Injected watermark should have very little impact on the quality of the embeddings, as it is main selling point in EaaS (Mistral, 2024).

• Effectiveness. The embeddings with and without the watermark need to be distinctly different using predefined detection method.

• Reliability. We can not claim ownership of a non-watermarked mode, i.e., low false positive rate (FPR).

• Identifiability. The watermark contains the model owner’s identifier (Wang et al., 2024a).

• Persistence-to-Permutation. Since embeddings are permutation-invariant, the watermark should still remain effective even if the embedding is rearranged by an attacker (Peng et al., 2023).

• Persistence-to-Unauthorized-Detection. We want the watermark to be undetectable by others. For EmbMarker (Peng et al., 2023) and WARDEN (Shetty et al., 2024a), the distributions of cosine similarities between watermarked and non-watermarked embeddings and the target embedding do not overlap. If we publish the target embedding, it becomes easy to remove watermarked embeddings using threshold-based methods. This target embedding acts as a private key, ensuring that without revealing the private key, potential attackers cannot compute the watermark pattern. If we use certain statistical features as a watermark, such as the sum and standard deviation of embeddings, these unencrypted watermarks can be easily removed from the data by setting a threshold.

C  Experimental Settings

C.1  Statistics of Datasets

We include the statistical information of selected datasets in Tab. 2 to demonstrate that our dataset is diverse.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train Size</th>
    <th>Test Size</th>
    <th>Avg. Tokens</th>
    <th>Classes</th>
  </tr>
  <tr>
    <td>SST2</td>
    <td>67,349</td>
    <td>872</td>
    <td>54</td>
    <td>2</td>
  </tr>
  <tr>
    <td>MIND</td>
    <td>97,791</td>
    <td>32,592</td>
    <td>66</td>
    <td>18</td>
  </tr>
  <tr>
    <td>AG News</td>
    <td>120,000</td>
    <td>7,600</td>
    <td>35</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Enron</td>
    <td>31,716</td>
    <td>2,000</td>
    <td>236</td>
    <td>2</td>
  </tr>
</table>

C.2  Implementation Details

For EmbMarker, WARDEN and our approach, we set the size of trigger set to 20 for each watermark. The frequency for selecting triggers is set to [0.5%, 1%]. And we set steal epoch to 10. For EmbMarker and WARDEN, the maximum number of triggers is 4. For WARDEN, we choose 5 watermarks due to its multi-watermark feature. For our approach, we set the watermark proportion to 20% by default.

To illustrate that all methods exhibits the Persistence-to-Permutation property described in § B, we assume that the stealer will apply a same permutation rule to all provider’s embeddings before training stealer’s model. When verification, instead of using the target embedding returned by victim’s EaaS, we query the suspicious EaaS with target sample to get returned target embedding for verification.
C.3 Details of Metrics

We now use the \( p \)-value being less than \( 10^{-3} \) as the primary criterion to indicate whether a suspected EaaS is a copy version, with \( \Delta \cos \) and \( \Delta l_2 \) serving as assistant metrics as their thresholds are difficult to determine. To measure the Harmlessness property, we train a two-layer MLP classifier using the provider’s embeddings as input features. The classifier’s accuracy (ACC) on a downstream task (Bai et al., 2024; Shi et al., 2023; Wang et al., 2024c) serves as the metric for measuring the quality of the embeddings. We also report the average cosine similarities of original embeddings and watermarked embeddings. Note that although there are some benchmarks (Dubois et al., 2024; Yang et al., 2024; Li et al., 2024; Huang et al., 2024) for generation models, using downstream tasks performance is feasible for embedding models. To measure the Reliability, i.e., low false positive rate, we ensure that all results with in this paper is lower that \( 10^{-4} \). See details in § E.2.

D More Results

D.1 Main Results on More Datasets

We present the main results on other datasets in Tab. 3, Tab. 4, and Tab. 5. Compared to other watermarking methods, our approach is also the only one that successfully verifies copyright in all cases.

Table 3: Performance of different methods on MIND. For no CSE, higher ACC means better harmlessness. For CSE, lower ACC means better watermark effectiveness. In “COPY?” column, correct verifications are green and failures are red. Best results are highlighted in bold (except Original).

<table>
  <tr>
    <th>K(CSE)</th>
    <th>Method</th>
    <th>ACC(%)</th>
    <th>\( p\)-value↓</th>
    <th>\( \Delta \cos(\%) \uparrow \)</th>
    <th>\( \Delta l_2(\%) \downarrow \)</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td rowspan="4">No CSE</td>
    <td>Original</td>
    <td>77.23 ± 0.22</td>
    <td>&gt; 0.2148</td>
    <td>-0.60 ± 0.22</td>
    <td>1.19 ± 0.44</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>77.17 ± 0.20</td>
    <td>&lt; 10^{-11}</td>
    <td>13.53 ± 0.11</td>
    <td>-27.06 ± 0.22</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>77.23 ± 0.09</b></td>
    <td>&lt; 10^{-11}</td>
    <td><b>18.05 ± 0.48</b></td>
    <td><b>-36.10 ± 0.95</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>77.22 ± 0.12</td>
    <td>&lt; 10^{-8}</td>
    <td>8.68 ± 0.24</td>
    <td>-17.36 ± 0.47</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>Original</td>
    <td>77.23 ± 0.10</td>
    <td>&gt; 0.0925</td>
    <td>-4.30 ± 0.89</td>
    <td>8.61 ± 1.77</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>77.18 ± 0.15</td>
    <td>&lt; 10^{-11}</td>
    <td><b>98.39 ± 1.76</b></td>
    <td><b>-196.77 ± 3.51</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>77.06 ± 0.07</b></td>
    <td>&lt; 10^{-11}</td>
    <td>85.09 ± 3.57</td>
    <td>-170.19 ± 7.14</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>77.16 ± 0.12</td>
    <td>&lt; 10^{-9}</td>
    <td>56.64 ± 1.73</td>
    <td>-113.28 ± 3.46</td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">50</td>
    <td>Original</td>
    <td>75.60 ± 0.09</td>
    <td>&gt; 0.2922</td>
    <td>3.43 ± 1.68</td>
    <td>-6.87 ± 3.36</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>75.34 ± 0.24</td>
    <td>&gt; 0.1103</td>
    <td>5.84 ± 1.90</td>
    <td>-11.69 ± 3.79</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>75.20 ± 0.11</b></td>
    <td>&gt; 0.3365</td>
    <td>3.91 ± 3.08</td>
    <td>-7.81 ± 6.15</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>75.48 ± 0.18</td>
    <td>&lt; 10^{-11}</td>
    <td><b>72.14 ± 2.16</b></td>
    <td><b>-144.28 ± 4.31</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">100</td>
    <td>Original</td>
    <td>74.64 ± 0.08</td>
    <td>&gt; 0.6805</td>
    <td>1.66 ± 2.04</td>
    <td>-3.33 ± 4.09</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>74.60 ± 0.14</td>
    <td>&gt; 0.1072</td>
    <td>6.91 ± 3.01</td>
    <td>-13.82 ± 6.03</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>74.33 ± 0.17</b></td>
    <td>&gt; 0.2361</td>
    <td>2.00 ± 6.56</td>
    <td>-4.00 ± 13.12</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>74.69 ± 0.30</td>
    <td>&lt; 10^{-10}</td>
    <td><b>69.55 ± 4.15</b></td>
    <td><b>-139.10 ± 8.29</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1000</td>
    <td>Original</td>
    <td>65.87 ± 0.49</td>
    <td>&gt; 0.5186</td>
    <td>-2.44 ± 2.28</td>
    <td>4.89 ± 4.56</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>68.35 ± 1.32</td>
    <td>&gt; 0.6442</td>
    <td>0.72 ± 5.37</td>
    <td>-1.43 ± 10.74</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>67.01 ± 0.18</td>
    <td>&gt; 0.3558</td>
    <td>0.00 ± 4.71</td>
    <td>0.00 ± 9.41</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>65.61 ± 0.49</b></td>
    <td>&lt; 10^{-9}</td>
    <td><b>32.98 ± 9.34</b></td>
    <td><b>-65.96 ± 18.67</b></td>
    <td>✓</td>
  </tr>
</table>

D.2 Ablation Results on More Datasets

We present additional ablation results on other datasets in Fig. 8, Fig. 9, and Fig. 10. When CSE is not applied, it can be observed that our proposed method can inject watermark successfully with a minimum \( \alpha \) value of 15% on all datasets. And as \( \alpha \) increases, the detection performance of the watermark is also greater. When CSE is applied, compared with the situation without CSE, the trend in detection
Table 4: Performance of different methods on AGNews. For no CSE, lower ACC means better harmlessness. For CSE, lower ACC means better watermark effectiveness. In "COPY?" column, correct verifications are green and failures are red. Best results are highlighted in bold (except Original).

<table>
  <tr>
    <th>K(CSE)</th>
    <th>Method</th>
    <th>ACC(%) ↓</th>
    <th>p-value↓</th>
    <th>Δ cos(%) ↑</th>
    <th>Δ\( l_2 \)(%) ↓</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td rowspan="4">No CSE</td>
    <td>Original</td>
    <td>93.43 ± 0.27</td>
    <td>&gt;0.02324</td>
    <td>1.11 ± 0.42</td>
    <td>-2.22 ± 0.83</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td><b>93.60 ± 0.06</b></td>
    <td>&lt; \(10^{-11}\)</td>
    <td><b>13.15 ± 0.55</b></td>
    <td><b>-26.29 ± 1.11</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>93.22 ± 0.10</td>
    <td>&gt;0.0083</td>
    <td>-6.24 ± 5.96</td>
    <td>12.47 ± 11.92</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>93.42 ± 0.16</b></td>
    <td>&lt; \(10^{-11}\)</td>
    <td><b>9.59 ± 0.74</b></td>
    <td><b>-19.19 ± 1.49</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>Original</td>
    <td>94.12 ± 0.14</td>
    <td>&gt;0.3936</td>
    <td>2.22 ± 0.98</td>
    <td>-4.45 ± 1.96</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>94.01 ± 0.18</td>
    <td>&lt; \(10^{-11}\)</td>
    <td><b>136.32 ± 2.24</b></td>
    <td><b>-272.65 ± 4.48</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>93.75 ± 0.23</b></td>
    <td>&lt; \(10^{-11}\)</td>
    <td>96.69 ± 1.62</td>
    <td>-193.38 ± 3.24</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>94.05 ± 0.15</td>
    <td>&lt; \(10^{-11}\)</td>
    <td><b>56.51 ± 2.47</b></td>
    <td><b>-113.02 ± 4.95</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">50</td>
    <td>Original</td>
    <td>93.39 ± 0.24</td>
    <td>&gt;0.0454</td>
    <td>-4.78 ± 1.03</td>
    <td>9.56 ± 2.05</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>93.04 ± 0.33</td>
    <td>&lt; \(10^{-6}\)</td>
    <td>14.43 ± 4.91</td>
    <td>-28.85 ± 9.81</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>92.54 ± 0.36</b></td>
    <td>&gt;0.3062</td>
    <td>2.40 ± 2.32</td>
    <td>-4.79 ± 4.65</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>93.00 ± 0.12</td>
    <td>&lt; \(10^{-10}\)</td>
    <td><b>21.83 ± 5.11</b></td>
    <td><b>-43.65 ± 10.22</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">100</td>
    <td>Original</td>
    <td>92.77 ± 0.28</td>
    <td>&gt;0.0520</td>
    <td>-4.50 ± 0.66</td>
    <td>9.00 ± 1.33</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>92.46 ± 0.17</td>
    <td>&gt;0.0206</td>
    <td>8.36 ± 3.72</td>
    <td>-16.71 ± 7.44</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>91.62 ± 0.21</b></td>
    <td>&gt;0.1488</td>
    <td>-3.95 ± 2.19</td>
    <td>7.89 ± 4.37</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>92.81 ± 0.18</td>
    <td>&lt; \(10^{-5}\)</td>
    <td><b>20.07 ± 10.23</b></td>
    <td><b>-40.15 ± 20.46</b></td>
    <td>✓</td>
  </tr>
  <tr>
    <td rowspan="4">1000</td>
    <td>Original</td>
    <td>88.55 ± 0.21</td>
    <td>&gt;0.1745</td>
    <td>3.4 ± 0.96</td>
    <td>-6.81 ± 1.34</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>90.22 ± 0.31</td>
    <td>&gt;0.8320</td>
    <td>2.58 ± 2.18</td>
    <td>-5.17 ± 3.12</td>
    <td>×</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>79.82 ± 0.22</b></td>
    <td>&gt;0.0335</td>
    <td>-6.51 ± 3.96</td>
    <td>13.03 ± 6.76</td>
    <td>×</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>86.92 ± 0.19</td>
    <td>&lt; \(10^{-8}\)</td>
    <td><b>23.03 ± 11.12</b></td>
    <td><b>-46.07 ± 23.12</b></td>
    <td>✓</td>
  </tr>
</table>

performance relative to \( \alpha \) remains similar when \( \alpha \) is small. However, when a large \( \alpha \) is set, our method will fail. These findings are consistent with those on the SST2 dataset.

D.3 Understanding the Enhanced Detection of ESpeW Under Stronger CSE Attacks

The watermark (WM) detection performance of ESpeW increases under stronger CSE attacks (higher \( K \)) due to the fundamental mechanism of CSE. Specifically, CSE aims to remove the principal components of the embeddings that are commonly shared among multiple embeddings. These principal components include both watermark-related and watermark-unrelated components.

Watermark-unrelated components are inherently present in embeddings—even without an intentionally embedded watermark, two embeddings will exhibit a certain degree of similarity. Our method, however, ensures that watermark-related components are resistant to removal. Consequently, as \( K \) increases, CSE tends to eliminate a greater proportion of watermark-unrelated components.

To further illustrate this phenomenon, we calculate the cosine similarities between the target embedding and the embeddings of both clean samples and trigger-bearing samples, as shown in Table 6. Additionally, we present the ratio between these two similarities, which serves as an indicator of watermark detection effectiveness. A higher ratio indicates stronger detection capability.

The results demonstrate (1) Without CSE, a notable similarity exists between the target embedding and clean embeddings. (2) For low-intensity CSE (\( K = 1 \)), the similarity between clean embeddings and the target embedding decreases significantly, while the similarity for trigger-bearing embeddings remains relatively high. (3) As the attack intensity increases (\( K = 50, 1000 \)), this trend becomes more pronounced.

This phenomenon is not unique to our method and is also observed in other watermarking techniques such as EmbMarker and WARDEN under \( K = 1 \), where their performance is enhanced compared to the absence of an attack (see Tab. 1).
Table 5: Performance of different methods on Enron Spam. For no CSE, higher ACC means better harmlessness. For CSE, lower ACC means better watermark effectiveness. In "COPY?" column, correct verifications are green and failures are red. Best results are highlighted in bold (except Original).

<table>
  <tr>
    <th>K(CSE)</th>
    <th>Method</th>
    <th>ACC(%)</th>
    <th>p-value↓</th>
    <th>Δ cos(%) ↑</th>
    <th>Δ\( l_2 \)(%) ↓</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td rowspan="4">No CSE</td>
    <td>Original</td>
    <td>94.90±0.35</td>
    <td>&gt;0.5776</td>
    <td>-0.11±0.26</td>
    <td>0.22±0.52</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td><b>94.86±0.24</b></td>
    <td>&lt; 10^{-10}</td>
    <td><b>9.75±0.11</b></td>
    <td><b>-19.49±0.21</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td>94.31±0.44</td>
    <td>&lt; 10^{-11}</td>
    <td>7.00±0.62</td>
    <td>-14.00±1.24</td>
    <td>✔️</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td><b>94.73±0.23</b></td>
    <td>&lt; 10^{-10}</td>
    <td><b>7.23±0.35</b></td>
    <td><b>-14.47±0.70</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>Original</td>
    <td>95.99±0.41</td>
    <td>&gt;0.5791</td>
    <td>0.58±2.06</td>
    <td>-1.15±4.12</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>95.93±0.37</td>
    <td>&lt; 10^{-10}</td>
    <td><b>69.55±7.16</b></td>
    <td><b>-139.10±14.32</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>95.80±0.05</b></td>
    <td>&lt; 10^{-11}</td>
    <td>68.01±1.62</td>
    <td>-136.02±3.23</td>
    <td>✔️</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>95.86±0.19</td>
    <td>&lt; 10^{-10}</td>
    <td><b>56.25±3.53</b></td>
    <td><b>-112.50±7.06</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td rowspan="4">50</td>
    <td>Original</td>
    <td>95.68±0.13</td>
    <td>&gt;0.7668</td>
    <td>0.50±1.15</td>
    <td>-1.00±2.30</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>95.48±0.47</td>
    <td>&gt;0.0002</td>
    <td>11.00±1.77</td>
    <td>-22.01±3.53</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>95.39±0.14</b></td>
    <td>&gt;0.5751</td>
    <td>-1.39±2.38</td>
    <td>2.77±4.77</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>95.48±0.28</td>
    <td>&lt; 10^{-10}</td>
    <td><b>47.75±4.13</b></td>
    <td><b>-95.50±8.26</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td rowspan="4">100</td>
    <td>Original</td>
    <td>95.44±0.54</td>
    <td>&gt;0.6805</td>
    <td>0.45±0.73</td>
    <td>-0.91±1.46</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>95.34±0.31</td>
    <td>&gt;0.0114</td>
    <td>10.75±2.91</td>
    <td>-21.50±5.82</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>94.86±0.29</b></td>
    <td>&gt;0.4970</td>
    <td>-0.13±4.28</td>
    <td>0.25±8.57</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>95.25±0.30</td>
    <td>&lt; 10^{-10}</td>
    <td><b>44.24±6.44</b></td>
    <td><b>-88.49±12.87</b></td>
    <td>✔️</td>
  </tr>
  <tr>
    <td rowspan="4">1000</td>
    <td>Original</td>
    <td>94.69±0.26</td>
    <td>&gt;0.4169</td>
    <td>-1.17±2.05</td>
    <td>2.33±4.10</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EmbMarker</td>
    <td>94.89±0.54</td>
    <td>&gt;0.0243</td>
    <td>6.66±2.63</td>
    <td>-13.32±5.26</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>WARDEN</td>
    <td><b>94.39±0.41</b></td>
    <td>&gt;0.3736</td>
    <td>2.45±4.32</td>
    <td>-4.91±8.63</td>
    <td>❌</td>
  </tr>
  <tr>
    <td>EspeW(Ours)</td>
    <td>94.69±0.66</td>
    <td>&lt; 10^{-9}</td>
    <td><b>35.25±3.29</b></td>
    <td><b>-70.51±6.58</b></td>
    <td>✔️</td>
  </tr>
</table>

<table>
  <tr>
    <th>K</th>
    <th>Cos Sim (Clean, Target)</th>
    <th>Cos Sim (Trigger, Target)</th>
    <th>Ratio (Trigger/Clean)</th>
  </tr>
  <tr>
    <td>No CSE</td>
    <td>0.8262</td>
    <td>0.9405</td>
    <td>1.14</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.0620</td>
    <td>0.7220</td>
    <td>11.65</td>
  </tr>
  <tr>
    <td>50</td>
    <td>0.0539</td>
    <td>0.6717</td>
    <td>12.46</td>
  </tr>
  <tr>
    <td>1000</td>
    <td>0.0226</td>
    <td>0.4096</td>
    <td>18.12</td>
  </tr>
</table>

Table 6: Impact of CSE attack strength (\( K \)) on cosine similarity and watermark detection.

D.4 Ablation on Impact of \( \alpha \) on Cosine Similarity between Original and Watermarked Embedding.

We conducted ablation experiments to analyze the impact of the parameter \( \alpha \) on the cosine similarity between the original embedding and the watermarked embedding. The results are presented in Tab. 7. As expected, increasing \( \alpha \) leads to a decline in embedding quality while enhancing the detection capability. This observation aligns with our understanding and further validates our approach.

<table>
  <tr>
    <th>\( \alpha \)</th>
    <th>0.05</th>
    <th>0.10</th>
    <th>0.15</th>
    <th>0.20</th>
    <th>0.25</th>
    <th>0.30</th>
  </tr>
  <tr>
    <th>cos sim</th>
    <td>0.9979</td>
    <td>0.9958</td>
    <td>0.9936</td>
    <td>0.9912</td>
    <td>0.9886</td>
    <td>0.9856</td>
  </tr>
  <tr>
    <th>p-value</th>
    <td>0.3356</td>
    <td>0.0040</td>
    <td>10^{-6}</td>
    <td>10^{-8}</td>
    <td>10^{-11}</td>
    <td>10^{-11}</td>
  </tr>
</table>

Table 7: Ablation study on the impact of \( \alpha \) on cosine similarity and detection capability.
(a) Effect of watermark proportion without CSE.

(b) Effect of watermark proportion with CSE.

Figure 8: Ablation results of watermark proportion on MIND. (a) shows results without CSE. (b) shows results with CSE, where \( K \) is set to 50.

(a) Effect of watermark proportion without CSE.

(b) Effect of watermark proportion with CSE.

Figure 9: Ablation results of watermark proportion on AGNews. (a) shows results without CSE. (b) shows results with CSE, where \( K \) is set to 50.

D.5 Testing on More Base Models

We further extend our evaluation by providing comparative experimental results for both BERT-Base-Cased and BERT-Large-Cased models. The results are summarized in Tab. 8. As shown, the differences between the two models are minimal, indicating that our method maintains its effectiveness across different base models. These results demonstrate the robustness of our method, showing consistent performance across different model architectures.

<table>
  <tr>
    <th>Base Model</th>
    <th>Method</th>
    <th>ACC(%)↓</th>
    <th>p-value↓</th>
    <th>Δcos(%)↑</th>
    <th>Δl2(%)↓</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td>BERT-Base-Cased</td>
    <td>ESpeW</td>
    <td>86.23</td>
    <td>1.45 \times 10^{-11}</td>
    <td>64.53</td>
    <td>-129.06</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>BERT-Large-Cased</td>
    <td>ESpeW</td>
    <td>86.82</td>
    <td>1.45 \times 10^{-11}</td>
    <td>65.28</td>
    <td>-130.57</td>
    <td>✓</td>
  </tr>
</table>

Table 8: Comparative results on BERT-Base-Cased and BERT-Large-Cased models using ESpeW.

D.6 Random Selection

We begin by providing a detailed description of the random selection algorithm. A direct random selection approach is suboptimal, as the watermarked positions for the same sentence may vary across different queries. An attacker could exploit this variability by making multiple queries to detect or remove the
(a) Effect of watermark proportion without CSE.

(b) Effect of watermark proportion with CSE.

Figure 10: Ablation results of watermark proportion on Enron Spam. (a) shows results without CSE. (b) shows results with CSE, where \( K \) is set to 50.

watermark. To mitigate this issue, we propose using the hash value of the embedding (we adopt SHA-256 as our hash function) as a seed, ensuring consistent and repeatable position selection. The algorithm is outlined in Alg. 1.

Algorithm 1 Random Selection Algorithm

1: Input: Original embedding \( e_o \), a hash function \( HASH(\cdot) \), watermark proportion \( \alpha \)
2: Output: Selected watermark positions \( M \)
3: Convert the original embedding \( e_o \) into byte format \( B_{e_o} \).
4: Generate a random seed \( R = HASH(B_{e_o}) \).
5: Using \( R \) as seed, select \( \alpha|e_o| \) random indices \( M \).
6: Return \( M \).

We then conduct formalized time complexity analysis. For smallest-magnitude selection. Using heap sort for the top-k problem is the most common approach, achieving a time complexity of \( O(N \log k) \). Thus, the total time complexity of smallest-magnitude selection is \( O(N \log k) \). For random selection. Converting \( e_o \) to byte format requires \( O(N) \), SHA-256 hashing also takes \( O(N) \), and selecting random indices needs \( O(k) \). Therefore, the total time complexity of random selection is \( O(2N + k) \). Considering the high-dimensional nature of embeddings, random selection typically has a much lower time complexity than smallest-magnitude selection.

To evaluate the time consumption, we conduct experiments using two widely-used open-sourced embedding models: NV-Embed-v2 (Lee et al., 2024), which ranks first in the METB leaderboard (Muenninghoff et al., 2023), and Stella (Stella, 2024), which is ranked first in the METB leaderboard under 1.5B. We measure the time for 2,000 generations, repeating the experiment five times to reduce the impact of random fluctuations. All experiments are performed on an Ubuntu 18.04 system with an AMD EPYC 7Y83 64-core CPU and a NVIDIA RTX 4090 GPU. The results are summarized in Tab. 9. As can be seen, the time consumed by the random selection-based watermark is significantly smaller than the model inference time, and it also shows a clear advantage over the smallest-magnitude selection-based watermark in terms of time consumption. Therefore, when scaling to large-scale usage, the random selection method offers a clear advantage.

Watermark performance comparison of using smallest-magnitude and random Selection. We report the detection capability, as well as cosine similarity with clean embedding \( \cos(\%) \)w/clean to assess embedding quality. The parameter \( K = 50 \) is used. From the results, we can see that while random selection sacrifices more embedding quality, it achieves better watermarking performance.

The above analyses and experiments demonstrate that both smallest-magnitude selection and random selection have their unique advantages, making them suitable for different application scenarios:
Table 9: Time consumption comparison between random and smallest-magnitude selection.

<table>
  <tr>
    <th>Model</th>
    <th>Model Size</th>
    <th>Embedding Size</th>
    <th>Inference Time (ms)</th>
    <th>Smallest-magnitude Selection Time (ms)</th>
    <th>Random Selection Time (ms)</th>
  </tr>
  <tr>
    <td><b>Stella</b></td>
    <td>1.5B</td>
    <td>1024</td>
    <td>4371.80 ± 204.80</td>
    <td>716.30 ± 1.50</td>
    <td>31.49 ± 0.40</td>
  </tr>
  <tr>
    <td><b>NV-Embed-v2</b></td>
    <td>7B</td>
    <td>4096</td>
    <td>13799.46 ± 459.30</td>
    <td>3761.18 ± 276.59</td>
    <td>86.33 ± 0.49</td>
  </tr>
</table>

Table 10: Watermark performance comparison between smallest-magnitude and random selection.

<table>
  <tr>
    <th>Dataset</th>
    <th>Method</th>
    <th>\( p\)-value \( \downarrow \)</th>
    <th>\( \Delta \cos(\%) \uparrow \)</th>
    <th>\( \Delta l_2(\%) \downarrow \)</th>
    <th>\( \cos(\%)\)w/clean \( \uparrow \)</th>
  </tr>
  <tr>
    <td>SST2</td>
    <td>Smallest</td>
    <td>\( 10^{-11} \)</td>
    <td>65.11</td>
    <td>-130.23</td>
    <td>99.19</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>\( 10^{-11} \)</td>
    <td>72.81</td>
    <td>-145.62</td>
    <td>98.87</td>
  </tr>
  <tr>
    <td>MIND</td>
    <td>Smallest</td>
    <td>\( 10^{-11} \)</td>
    <td>72.14</td>
    <td>-144.28</td>
    <td>99.23</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>\( 10^{-11} \)</td>
    <td>77.27</td>
    <td>-154.55</td>
    <td>98.69</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>Smallest</td>
    <td>\( 10^{-10} \)</td>
    <td>21.83</td>
    <td>-43.65</td>
    <td>99.27</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>\( 10^{-11} \)</td>
    <td>53.13</td>
    <td>-106.27</td>
    <td>98.97</td>
  </tr>
  <tr>
    <td>Enron Spam</td>
    <td>Smallest</td>
    <td>\( 10^{-10} \)</td>
    <td>47.75</td>
    <td>-95.5</td>
    <td>99.21</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>\( 10^{-11} \)</td>
    <td>68.38</td>
    <td>-136.75</td>
    <td>98.92</td>
  </tr>
</table>

• Smallest-magnitude selection significantly benefits embedding quality preservation, with modifications to the clean embeddings under 1%. This is crucial for real-world applications where organizations aim to improve their rankings on leaderboards while protecting their intellectual property.

• Random selection, although sacrificing more embedding quality, offers substantial time savings, making it more suitable for product deployment in large-scale applications.

We conclude that both approaches are valuable, and users can choose the appropriate method based on their specific application requirements.

D.7 Evaluation on More Embedding Models

We apply our watermark to more models to verify our watermark ESpeW’s effectiveness. We select two widely-used open-sourced embedding models: (1) NV-Embed-v2 (Lee et al., 2024), which ranks first in the METB leaderboard (Muenninghoff et al., 2023) and has an embedding dimension of 4096, and (2) Stella-1.5B-V5 (Stella, 2024), which is ranked first in the METB leaderboard under 1.5B. Using the Enron spam dataset and \( K = 50 \), we evaluate watermark performance with different \( \alpha \). Based on the results in Tab. 11, we can see that our method remains effective across these embedding models. And it still demonstrates a high detection capability and robustness to CSE.

D.8 Resistance Against More Potential Removal Attacks

D.8.1 Resistance Against Fine-tuning

To evaluate the robustness of our method against fine-tuning attacks, we adopt the unsupervised fine-tuning approach SimCSE (Gao et al., 2021). SimCSE applies contrastive learning by introducing random dropout masks in the Transformer encoder. Positive samples are generated by feeding the same input twice with different dropout masks, while negative samples are constructed from other sentences within the batch. Note that supervised fine-tuning is fundamentally incompatible with embedding models, as it would cause the embeddings to carry excessive label information, compromising semantic properties. Thus, we focus on unsupervised fine-tuning. The experiments are conducted using the hyperparameter settings provided in our paper, and evaluated on the Enron Spam dataset. Fine-tuning parameters are consistent with SimCSE (Gao et al., 2021), using a learning rate of \( 3 \times 10^{-5} \) and a batch size of 64.
Table 11: Evaluation of ESpeW on additional embedding models. This evaluation is conducted on Enron Spam under CSE attack with \( K = 50 \).

<table>
  <tr>
    <th>\( \alpha \)</th>
    <th>ACC(%)</th>
    <th>p-value\(\downarrow\)</th>
    <th>\( \Delta \cos(\%) \uparrow \)</th>
    <th>\( \Delta l_2(\%) \downarrow \)</th>
  </tr>
  <tr>
    <td>0.05</td>
    <td>95.69</td>
    <td>9.55E-06</td>
    <td>13.12</td>
    <td>-26.23</td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>95.81</td>
    <td>1.13E-08</td>
    <td>27.02</td>
    <td>-54.04</td>
  </tr>
  <tr>
    <td>0.15</td>
    <td>95.99</td>
    <td>1.13E-08</td>
    <td>36.62</td>
    <td>-73.24</td>
  </tr>
  <tr>
    <th rowspan="5">Stella</th>
    <td>0.2</td>
    <td>95.39</td>
    <td>5.80E-10</td>
    <td>47.30</td>
    <td>-94.60</td>
  </tr>
  <tr>
    <td>0.25</td>
    <td>95.99</td>
    <td>5.80E-10</td>
    <td>56.77</td>
    <td>-113.54</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>95.99</td>
    <td>5.80E-10</td>
    <td>62.31</td>
    <td>-124.62</td>
  </tr>
  <tr>
    <td>0.6</td>
    <td>95.32</td>
    <td>9.55E-06</td>
    <td>10.45</td>
    <td>-20.89</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>0.05</td>
    <td>96.20</td>
    <td>2.70E-04</td>
    <td>9.04</td>
    <td>-18.08</td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>96.10</td>
    <td>1.13E-08</td>
    <td>23.90</td>
    <td>-47.79</td>
  </tr>
  <tr>
    <td>0.15</td>
    <td>95.70</td>
    <td>5.80E-10</td>
    <td>40.56</td>
    <td>-81.13</td>
  </tr>
  <tr>
    <th rowspan="5">NV-Embed</th>
    <td>0.2</td>
    <td>95.90</td>
    <td>1.45E-11</td>
    <td>52.08</td>
    <td>-104.17</td>
  </tr>
  <tr>
    <td>0.25</td>
    <td>96.25</td>
    <td>1.45E-11</td>
    <td>65.99</td>
    <td>-131.98</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td>95.95</td>
    <td>1.45E-11</td>
    <td>72.47</td>
    <td>-144.93</td>
  </tr>
  <tr>
    <td>0.6</td>
    <td>96.10</td>
    <td>1.45E-11</td>
    <td>53.36</td>
    <td>-106.72</td>
  </tr>
</table>

During the detection phase, we replace the p-value with \( \Delta \cos(\%) \) and \( \Delta l_2(\%) \) as evaluation metrics. This adjustment is necessary because fine-tuning induces increased instability in embeddings, causing the p-value to inflate abnormally and lose reliability. To address this, we use the alternative metrics introduced in our paper, ensuring that the false positive rate (FPR) remains below \( 10^{-5} \) by adjusting the detection thresholds.

Tab. 12 demonstrates that our approach effectively defends against fine-tuning attacks, even after 100 epochs of fine-tuning. Considering that data stealing typically involves fewer than 10 epochs, the cost of fine-tuning is significant in practice.

Table 12: Performance of our method under SimCSE-based unsupervised fine-tuning attacks.

<table>
  <tr>
    <th>Epoch</th>
    <th>p-value</th>
    <th>\( \Delta \cos(\%) \)</th>
    <th>\( \Delta l_2(\%) \)</th>
    <th>FPR@0.05</th>
    <th>FPR@0.01</th>
    <th>FPR@10^{-3}</th>
    <th>FPR@10^{-4}</th>
    <th>FPR@10^{-5}</th>
  </tr>
  <tr>
    <td>0</td>
    <td>5.8e-10</td>
    <td>8.10</td>
    <td>-16.21</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>1</td>
    <td>1.1e-8</td>
    <td>18.45</td>
    <td>-36.91</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>2</td>
    <td>1.4e-7</td>
    <td>11.92</td>
    <td>-23.84</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>3</td>
    <td>1.3e-6</td>
    <td>9.11</td>
    <td>-18.23</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>4</td>
    <td>1.4e-7</td>
    <td>12.42</td>
    <td>-24.83</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>5</td>
    <td>1.1e-3</td>
    <td>7.91</td>
    <td>-15.81</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>6</td>
    <td>1.1e-8</td>
    <td>14.12</td>
    <td>-28.24</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>7</td>
    <td>1.3e-6</td>
    <td>12.33</td>
    <td>-24.66</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>8</td>
    <td>4.0e-3</td>
    <td>6.56</td>
    <td>-13.12</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>9</td>
    <td>4.0e-3</td>
    <td>4.39</td>
    <td>-8.77</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>10</td>
    <td>2.7e-4</td>
    <td>6.21</td>
    <td>-12.42</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>20</td>
    <td>2.7e-4</td>
    <td>6.80</td>
    <td>-13.60</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>35</td>
    <td>0.03</td>
    <td>5.82</td>
    <td>-11.64</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>50</td>
    <td>0.08</td>
    <td>2.21</td>
    <td>-4.42</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>100</td>
    <td>0.34</td>
    <td>3.60</td>
    <td>-7.19</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

D.8.2 Resistance Against Quantization

Here, we test the robustness of the watermark against quantization attacks. Specifically, we apply 4-bit quantization to the embeddings. The results, shown in Tab. 14, indicate that the quantization attack has
Table 13: Thresholds used for detection metrics to achieve target FPR levels. Validated through 100,000 experiments on non-watermarked models.

<table>
  <tr>
    <th>FPR</th>
    <th>Threshold of \( \Delta \cos(\%) \)</th>
    <th>Threshold of \( \Delta l_2(\%) \)</th>
  </tr>
  <tr>
    <td>0.05</td>
    <td>0.41</td>
    <td>-1.57</td>
  </tr>
  <tr>
    <td>0.01</td>
    <td>0.59</td>
    <td>-2.32</td>
  </tr>
  <tr>
    <td>10^{-3}</td>
    <td>0.82</td>
    <td>-3.16</td>
  </tr>
  <tr>
    <td>10^{-4}</td>
    <td>1.08</td>
    <td>-3.93</td>
  </tr>
  <tr>
    <td>10^{-5}</td>
    <td>1.09</td>
    <td>-4.10</td>
  </tr>
</table>

little to no impact on the watermark’s performance.

<table>
  <tr>
    <th>Dataset</th>
    <th>\( p\)-value\(\downarrow\)</th>
  </tr>
  <tr>
    <td>SST2</td>
    <td>10^{-11}</td>
  </tr>
  <tr>
    <td>MIND</td>
    <td>10^{-11}</td>
  </tr>
  <tr>
    <td>AGNews</td>
    <td>10^{-10}</td>
  </tr>
  <tr>
    <td>Enron Spam</td>
    <td>10^{-10}</td>
  </tr>
</table>

Table 14: Watermark performance under 4-bit quantization when K = 50 in CSE.

D.8.3 Resistance Against Paraphrasing Attack

We note the recent work proposing paraphrasing attack (Shetty et al., 2024b). Since the authors did not release their paraphrased texts or the corresponding embeddings, we reproduce their approach to generate embeddings. Specifically, we generated three paraphrased versions for each sample from the SST-2 dataset (containing 67K samples) using LLaMA-3.1-8B-Instruct (Grattafiori et al., 2024). Then, we obtained the embeddings of these paraphrased texts using NV-Embed-2 (Lee et al., 2024), which is one of the most advanced embedding models.
To maintain the embedding quality after the paraphrasing attack, we set a threshold \( T \). We retain only those paraphrased text embeddings whose cosine similarity with the original text’s embedding exceeded \( T \). If none of the three paraphrased texts met this criterion, the embeddings were discarded. This filtering is reasonable, as a successful attack should preserve the embedding quality. Extracting low-quality embeddings would be meaningless and would not constitute a valid attack. Finally, we calculate the average of the remaining embeddings to train the stealer’s model. To ensure accuracy, we employ multiple random seeds during the experiment.
The results, presented in Tab. 15, demonstrate the robustness of our method. In other words, the paraphrasing attack is not sufficient to remove our watermark without significantly degrading the embedding quality.

D.8.4 Resistance Against an Adaptive Attack

By statistically analyzing the frequency of values at each position, \( e_t \) might be estimated. Based on this motivation, we discuss an adaptive attack based on statistical analysis, named statistical analysis attack (SAA). The algorithm of SAA in shown in Alg 2.

Through this algorithm, we can identify abnormally clustered values, thereby executing the statistical analysis attack. In our experiments, we fix \( T \) to a small value of \( 10^{-4} \) and evaluate the attack performance with varying values of \( N_T \). Since the SAA operation negatively affects embedding quality, we measure watermark quality using the cosine similarity between the embedding and the clean embedding, referred to as cos-clean. The other parameters remain the same as those in main experiments.

The results are summarized in Tab. 16 and demonstrate that this attack cannot successfully remove the watermark without severely damaging the embedding quality. In detail, with \( N_T \) set to 200, the p-value based detection becomes ineffective for watermark detection, while the watermark quality degrades to
<table>
  <tr>
    <th>T</th>
    <th>Random Seed</th>
    <th>ACC(%)↓</th>
    <th>p-value↓</th>
    <th>Δcos(%)↑</th>
    <th>Δl_2(%)↓</th>
    <th>COPY?</th>
  </tr>
  <tr>
    <td>95%</td>
    <td>0</td>
    <td>95.07</td>
    <td>5.80 \times 10^{-10}</td>
    <td>9.42</td>
    <td>-18.84</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>95%</td>
    <td>1</td>
    <td>94.84</td>
    <td>5.80 \times 10^{-10}</td>
    <td>10.54</td>
    <td>-21.08</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>95%</td>
    <td>2</td>
    <td>94.84</td>
    <td>5.80 \times 10^{-10}</td>
    <td>10.16</td>
    <td>-20.31</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>95%</td>
    <td>3</td>
    <td>94.95</td>
    <td>5.80 \times 10^{-10}</td>
    <td>10.65</td>
    <td>-21.31</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>95%</td>
    <td>4</td>
    <td>94.84</td>
    <td>5.80 \times 10^{-10}</td>
    <td>11.27</td>
    <td>-22.54</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>0</td>
    <td>95.07</td>
    <td>5.80 \times 10^{-10}</td>
    <td>9.63</td>
    <td>-19.26</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>1</td>
    <td>94.61</td>
    <td>5.80 \times 10^{-10}</td>
    <td>11.05</td>
    <td>-22.10</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>2</td>
    <td>95.07</td>
    <td>5.80 \times 10^{-10}</td>
    <td>10.62</td>
    <td>-21.24</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>3</td>
    <td>95.53</td>
    <td>5.80 \times 10^{-10}</td>
    <td>11.51</td>
    <td>-23.02</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>90%</td>
    <td>4</td>
    <td>94.95</td>
    <td>5.80 \times 10^{-10}</td>
    <td>11.14</td>
    <td>-22.28</td>
    <td>✓</td>
  </tr>
</table>

Table 15: Effectiveness of the paraphrasing attack on the ESpeW watermark.

Algorithm 2 Statistical Analysis Attack (SAA)

1: **Input:** Training embedding set of the stealer \( DE_c \in \mathbb{R}^{N \times M} \), tolerance level \( T \), number of neighboring partitions \( N_T \)
2: **Output:** Normalized embedding set after attack
3: **for** each embedding index \( i \) **do**
4:   Obtain the embedding array \( DE_{c_i} \in \mathbb{R}^N \) for index \( i \)
5:   Partition \( DE_{c_i} \) into small intervals using \( T \) as the step size
6:   Count the number of elements in each partition
7:   Initialize an empty set \( SE = \{ \} \)
8:   Add the partition with the highest number of elements to \( SE \)
9:   **if** a partition with a high concentration of elements is identified **then**
10:      Add this partition and its \( N_T \) neighboring partitions to \( SE \)
11:   **end if**
12:   Calculate the upper and lower bounds of \( SE \)
13:   Set the numbers within this interval to 0
14: **end for**
15: Normalize the resulting embedding

64.78% of its original level. When \( N_T \) is increased further, to 300 or beyond, the watermark embedding quality continues to degrade, with the *cos-clean* value reaching as low as 45.11% at \( N_T = 300 \).

D.9 Embedding Visualization of More Dataset

We put more visualization results in Fig. 11, Fig. 12, and Fig. 13.

E More Discussion

E.1 Discussion About Private Key Leakage Scenarios and Corresponding Strategies

We here discuss several potential leakage scenarios and corresponding strategies to mitigate these risks.

**Leakage Scenarios.** The primary leakage risks are associated with security vulnerabilities, including inadequate storage practices, insecure transmission channels, or insider threats. Inadequate storage, for instance, can result in unauthorized access or accidental exposure of sensitive embeddings. Similarly, insecure transmission of embeddings over unprotected networks can make them vulnerable to interception by malicious actors. Insider threats, where authorized individuals exploit their access for malicious purposes, further exacerbate the risks associated with embedding leakage. These vulnerabilities highlight the need for comprehensive security measures to protect the integrity and confidentiality of target embeddings.
Table 16: Performance under Statistical Analysis Attack (SAA) for varying \( N_T \). The watermark quality is evaluated using cos-clean. Watermark detection performance is evaluated by p-value, \( \Delta \cos \), and \( \Delta l_2 \) are reported.

<table>
  <tr>
    <th>\( N_T \)</th>
    <th>p-value\(\downarrow\)</th>
    <th>\( \Delta \cos (\%) \uparrow \)</th>
    <th>\( \Delta l_2 (\%) \downarrow \)</th>
    <th>cos-clean\(\uparrow\)</th>
  </tr>
  <tr>
    <td>1</td>
    <td>5.80 \times 10^{-10}</td>
    <td>7.85</td>
    <td>-15.69</td>
    <td>0.9887</td>
  </tr>
  <tr>
    <td>5</td>
    <td>5.80 \times 10^{-10}</td>
    <td>7.84</td>
    <td>-15.69</td>
    <td>0.9815</td>
  </tr>
  <tr>
    <td>10</td>
    <td>5.80 \times 10^{-10}</td>
    <td>7.36</td>
    <td>-14.71</td>
    <td>0.9738</td>
  </tr>
  <tr>
    <td>20</td>
    <td>5.80 \times 10^{-10}</td>
    <td>6.00</td>
    <td>-11.99</td>
    <td>0.9576</td>
  </tr>
  <tr>
    <td>30</td>
    <td>1.13 \times 10^{-10}</td>
    <td>5.67</td>
    <td>-11.34</td>
    <td>0.9419</td>
  </tr>
  <tr>
    <td>100</td>
    <td>5.80 \times 10^{-10}</td>
    <td>7.95</td>
    <td>-15.91</td>
    <td>0.8276</td>
  </tr>
  <tr>
    <td>200</td>
    <td>0.0011</td>
    <td>7.36</td>
    <td>-14.73</td>
    <td>0.6478</td>
  </tr>
  <tr>
    <td>250</td>
    <td>0.0335</td>
    <td>5.24</td>
    <td>-10.48</td>
    <td>0.5481</td>
  </tr>
  <tr>
    <td>300</td>
    <td>0.0123</td>
    <td>2.22</td>
    <td>-4.44</td>
    <td>0.4511</td>
  </tr>
  <tr>
    <td>350</td>
    <td>0.0123</td>
    <td>-7.27</td>
    <td>14.54</td>
    <td>0.3620</td>
  </tr>
  <tr>
    <td>400</td>
    <td>0.0040</td>
    <td>-9.99</td>
    <td>19.98</td>
    <td>0.2835</td>
  </tr>
</table>

Defense Strategies. To address these risks, we propose several mitigation strategies. One key approach is to regularly renew the security keys used for embedding protection, ensuring that even if a key is compromised, the window of vulnerability is minimized. Additionally, employing multiple keys can help limit the impact of any single breach by compartmentalizing access. It is also crucial to audit and continuously monitor access to sensitive embeddings, enabling quick detection and response to potential security breaches. Encrypting both storage and transmission ensures that even if unauthorized access occurs, the data remains unreadable without the proper decryption keys. Finally, restricting employee access to sensitive information by implementing the principle of least privilege can prevent unnecessary exposure and limit the potential for insider threats.

E.2 Discussion About False Positive

Here, we analyze the FPR in our method. In fact, FPR are influenced by most of the parameters discussed in our paper, making it challenging to exhaustively evaluate them under all possible configurations. However, through 100,000 independent tests on non-watermarked models, we can ensure that under the parameter settings used in our paper, the FPR is guaranteed to be less than \( 10^{-4} \). This represents a remarkably low FPR, which is practical and reliable for real-world applications.

E.3 Broader Impacts

Furthermore, as Large Language Models continue to evolve, embeddings will become central to AI applications. However, advanced model theft methods make current service providers reluctant to offer these valuable embeddings. A robust copyright protection method will greatly encourage more service providers to offer embedding services, thereby further accelerating the development and deployment of AI applications.
Figure 11: Visualization of the generated embedding of our ESpeW with different watermark proportion (\( \alpha \)) on MIND. It shows that we can generate watermarked embeddings indistinguishable with non-watermark embeddings by setting a reasonable watermark proportion.

Figure 12: Visualization of the generated embedding of our ESpeW with different watermark proportion (\( \alpha \)) on AGNews. It shows that we can generate watermarked embeddings indistinguishable with non-watermark embeddings by setting a reasonable watermark proportion.
Figure 13: Visualization of the generated embedding of our ESpeW with different watermark proportion (\( \alpha \)) on Enron Spam. It shows that we can generate watermarked embeddings indistinguishable with non-watermark embeddings by setting a reasonable watermark proportion.

Reviewer paper summaries: This paper proposes ESpeW, a new embedding-specific watermarking method designed for Embeddings-as-a-Service (EaaS). ESpeW 
 makes a small modification to EmbMarker, by changing linear interplotation to embedding positions selection for watermark injection. This small modification improves robustness against watermark removal (CSE) attacks.
Summary of strengths: 1. I like the Simple but Effective Design. The idea of selectively modifying embedding elements with small magnitudes is very elegant. Such a small change effectively improves robustness.
2.  Extensive experiments across datasets (SST2, MIND, AGNews, Enron) demonstrate the superior performance of proposed watermarks.
Summary of weaknesses: The paper lacks explicit mention of the α value used for the main Table 1 results or cross-dataset results.
Comments, suggestions and typos: ACC Metric is Questionable. As shown in Figure 3(a), the ACC remains similar (~93%) with a wide range of watermark strength (α = 0.05->0.6). This raises concerns that whether the ACC metric meaningfully captures embedding degradation.
Section 4.3 conduct an additional experiment to address the concern. But neither specific choice of α, nor the trends (similar to Figure 3a) are illustrated.
Overall assessment scores: 4